<chapter id="replication">
   <title>Clustered Caches</title>

   <para>The <literal>TreeCache</literal> can be configured to be either local (standalone) or clustered. If
   in a cluster, the cache can be configured to replicate changes, or to
   invalidate changes.  A detailed discussion on this follows.</para>

   <section>
     <title>Local Cache</title>

     <para>Local caches don't join a cluster and don't communicate with other
     nodes in a cluster. Therefore their elements don't need to be
     serializable - however, we recommend making them serializable, enabling a
     user to change the cache mode at any time.</para>
   </section>

   <section>
     <title>Clustered Cache - Using Replication</title>

     <para>Replicated caches replicate all changes to the
     other <literal>TreeCache</literal> instances in the cluster. Replication can either happen
     after each modification (no transactions), or at the end of a
     transaction (commit time).</para>

     <para>Replication can be synchronous or asynchronous . Use of either one
     of the options is application dependent. Synchronous replication blocks
     the caller (e.g. on a put()) until the modifications have been
     replicated successfully to all nodes in a cluster. Asynchronous
     replication performs replication in the background (the put() returns
     immediately). <literal>TreeCache</literal> also offers a replication queue, where
     modifications are replicated periodically (i.e. interval-based), or when
     the queue size exceeds a number of elements, or a combination
     thereof.</para>

     <para>Asynchronous replication is faster (no caller blocking), because
     synchronous replication requires acknowledgments from all nodes in a
     cluster that they received and applied the modification successfully
     (round-trip time). However, when a synchronous replication returns
     successfully, the caller knows for sure that all modifications have been
     applied at all nodes, whereas this may or may not be the case with
     asynchronous replication. With asynchronous replication, errors are
     simply written to a log.  Even when using transactions, a transaction may succeed but replication may not succeed on all <literal>TreeCache</literal> instances.</para>

      <section>
         <title>Replicated Caches and Transactions</title>
         <para>
            When using transactions, replication only occurs at the transaction boundary - i.e., when a transaction commits.
            This results in minimising replication traffic since a single modification os broadcast rather than a series of
            individual modifications, and can be a lot more efficient than not using transactions.  Another effect of this
            is that if a transaction were to roll back, nothing is broadcast across a cluster.
         </para>
         <para>
            Depending on whether you are running your cluster in asynchronous or synchronous mode, JBoss Cache will use
            either a single phase or <ulink url="http://en.wikipedia.org/wiki/Two-phase_commit_protocol">two phase commit</ulink> protocol, respectively.
         </para>
         <section>
            <title>One Phase Commits</title>
            <para>
               Used when your cache mode is REPL_ASYNC.  All modifications are replicated in a single call, which instructs
               remote caches to apply the changes to their local in-memory state and commit locally.  Remote
               errors/rollbacks are never fed back to the originator of the transaction since the communication is asynchronous.
            </para>
         </section>
         <section>
            <title>Two Phase Commits</title>
            <para>
               Used when your cache mode is REPL_SYNC.  Upon committing your transaction, JBoss Cache broadcasts a prepare call,
               which carries all modifications relevant to the transaction.  Remote caches then acquire local locks on their
               im-memory state and apply the modifications.  Once all remote caches respond to the prepare call, the
               originator of the transaction broadcasts a commit.  This instructs all remote caches to commit their data.
               If any of the caches fail to respond to the prepare phase, the originator broadcasts a rollback.
            </para>
            <para>
               Note that although the prepare phase is synchronous, the commit and rollback phases are asynchronous.  This
               is because <ulink url="http://java.sun.com/products/jta/">Sun's JTA specification</ulink> does not specify how transactional resources should deal with failures
               at this stage of a transaction; and other resources participating in the transaction may have indeterminate
               state anyway.  As such, we do away with the overhead of synchronous communication for this phase of the
               transaction.  That said, they can be forced to be synchronous using the <literal>SyncCommitPhase</literal>
               and <literal>SyncRollbackPhase</literal> configuration options.
            </para>
         </section>

      </section>

       <section>
           <title>Buddy Replication</title>
           <para>
                Buddy Replication allows you to suppress replicating your data to all instances in a cluster.  Instead,
               each instance picks one or more 'buddies' in the cluster, and only replicates to these specific buddies.
               This greatly helps scalability as there is no longer a memory and network traffic impact every time another
               instance is added to a cluster.
           </para>
           <para>
               One of the most common use cases of Buddy Replication is when a replicated cache is used by a servlet
               container to store HTTP session data.  One of the pre-requisites to buddy replication working well and being
               a real benefit is the use of <emphasis>session affinity</emphasis>, also known as <emphasis>sticky sessions</emphasis>
               in HTTP session replication speak.  What this means is that if certain data is frequently accessed, it is
               desirable that this is always accessed on one instance rather than in a round-robin fashion as this helps
               the cache cluster optimise how it chooses buddies, where it stores data, and minimises replication traffic.
            </para>
            <para>
               If this is not possible, Buddy Replication may prove to be more of an overhead than a benefit.
            </para>
            <section>
                <title>Selecting Buddies</title>
                <para>
                    Buddy Replication uses an instance of a <literal>org.jboss.cache.buddyreplication.BuddyLocator</literal>
                    which contains the logic used to select buddies in a network.  JBoss Cache currently ships with a single
                    implementation, <literal>org.jboss.cache.buddyreplication.NextMemberBuddyLocator</literal>, which is used
                    as a default if no implementation is provided.  The <literal>NextMemberBuddyLocator</literal> selects the
                    next member in the cluster, as the name suggests, and guarantees an even spread of buddies for each instance.
                </para>
                <para>
                    The <literal>NextMemberBuddyLocator</literal> takes in 2 parameters, both optional.
                    <itemizedlist>
                        <listitem><literal>numBuddies</literal> - specifies how many buddies each instance should pick to back its data onto.  This defaults to 1.</listitem>
                        <listitem><literal>ignoreColocatedBuddies</literal> - means that
                        each instance will <emphasis>try</emphasis> to select a buddy on a different physical host.  If not able to do so though,
                        it will fall back to colocated instances.  This defaults to <literal>true</literal>.</listitem>
                    </itemizedlist>
                </para>
            </section>
            <section>
                <title>BuddyPools</title>
                <para>
                    Also known as replication groups, a buddy pool is an optional construct where each instance in a cluster
                    may be configured with a buddy pool name.  Think of this as an 'exclusive club membership' where when
                    selecting buddies, <literal>BuddyLocator</literal>s would try and select buddies sharing the same
                    buddy pool name.  This allows system administrators a degree of flexibility and control over how buddies
                    are selected.  For example, a sysadmin may put two instances on two separate physical servers that
                    may be on two separate physical racks in the same buddy pool.  So rather than picking an
                    instance on a different host on the same rack, <literal>BuddyLocator</literal>s would rather pick
                    the instance in the same buddy pool, on a separate rack which may add a degree of redundancy.
                </para>
            </section>
            <section>
                <title>Failover</title>
                <para>
                    In the unfortunate event of an instance crashing, it is assumed that the client connecting to the
                    cache (directly or indirectly, via some other service such as HTTP session replication) is able to
                    redirect the request to any other random cache instance in the cluster.  This is where a concept of
                    Data Gravitation comes in.
                </para>
                <para>
                    Data Gravitation is a concept where if a request is made on a cache in the cluster and the cache does
                    not contain this information, it then asks other instances in the cluster for the data.  If even this
                    fails, it would (optionally) ask other instances to check in the backup data they store for other caches.
                    This means that even if a cache containing your session dies, other instances will still be able
                    to access this data by asking the cluster to search through their backups for this data.
                </para>
                <para>
                    Once located, this data is then transferred to the instance which requested it and is added to this
                    instance's data tree.  It is then (optionally) removed from all other instances (and backups) so
                    that if session affinity is used, the affinity should now be to this new cache instance which has
                    just <emphasis>taken ownership</emphasis> of this data.
                </para>
                <para>
                    Data Gravitation is implemented as an interceptor.  The following (all optional) configuration properties pertain to data gravitation.
                    <itemizedlist>	
                        <listitem><literal>dataGravitationRemoveOnFind</literal> - forces all remote caches that own the data or hold backups for the data to remove that data,
                            thereby making the requesting cache the new data owner. If set to <literal>false</literal> an evict is broadcast instead of a remove, so any state
                            persisted in cache loaders will remain. This is useful if you have a shared cache loader configured.  Defaults to <literal>true</literal>.</listitem>
                        <listitem><literal>dataGravitationSearchBackupTrees</literal> - Asks remote instances to search through their backups as well as main data trees.  Defaults to <literal>true</literal>.  The resulting effect is that if this is <literal>true</literal> then backup nodes can respond to data gravitation requests in addition to data owners.</listitem>
                        <listitem><literal>autoDataGravitation</literal> - Whether data gravitation occurs for every cache miss.  My default this is set to <literal>false</literal> to prevent unnecessary network calls.  Most use cases will know when it may need to gravitate data and will pass in an <literal>Option</literal> to enable data gravitation on a per-invocation basis.  If <literal>autoDataGravitation</literal> is <literal>true</literal> this <literal>Option</literal> is unnecessary.</listitem>
                    </itemizedlist>
                </para>
            </section>
            <section>
                <title>Implementation</title>
                <para>
                <figure>
                <title>Class diagram of the classes involved in buddy replication and how they are related to each other</title>

                  <mediaobject>
                    <imageobject>
                      <imagedata fileref="images/PackageOverview-BuddyReplication.png" />
                    </imageobject>
                  </mediaobject>
                </figure>
                </para>

            </section>
            <section>
                <title>Configuration</title>
                <para>
                    <programlisting><![CDATA[
                    <!-- Buddy Replication config -->
                    <attribute name="BuddyReplicationConfig">
                        <config>
                            
							<!-- Enables buddy replication.  This is the ONLY mandatory configuration element here. -->
                            <buddyReplicationEnabled>true</buddyReplicationEnabled>
                            
							<!-- These are the default values anyway -->
                            <buddyLocatorClass>org.jboss.cache.buddyreplication.NextMemberBuddyLocator</buddyLocatorClass>
                            
							<!-- numBuddies is the number of backup nodes each node maintains.  ignoreColocatedBuddies means that
                                each node will *try* to select a buddy on a different physical host.  If not able to do so though,
                                it will fall back to colocated nodes. -->
                            <buddyLocatorProperties>
                                numBuddies = 1
                                ignoreColocatedBuddies = true
                            </buddyLocatorProperties>
                            
							<!-- A way to specify a preferred replication group.  If specified, we try and pick a buddy why shares
                            the same pool name (falling back to other buddies if not available).  This allows the sysdmin to hint at
                            backup buddies are picked, so for example, nodes may be hinted topick buddies on a different physical rack
                            or power supply for added fault tolerance.  -->
                            <buddyPoolName>myBuddyPoolReplicationGroup</buddyPoolName>
                            
							<!-- Communication timeout for inter-buddy group organisation messages (such as assigning to and removing
                            from groups, defaults to 1000. -->
                            <buddyCommunicationTimeout>2000</buddyCommunicationTimeout>
							
							<!-- Whether data is removed from old owners when gravitated to a new owner.  Defaults to true.  -->
							<dataGravitationRemoveOnFind>true</dataGravitationRemoveOnFind>	
							
							<!-- Whether backup nodes can respond to data gravitation requests, or only the data owner is supposed to respond.  
								defaults to true. -->
							<dataGravitationSearchBackupTrees>true</dataGravitationSearchBackupTrees>	
							
							<!-- Whether all cache misses result in a data gravitation request.  Defaults to false, requiring callers to 
								enable data gravitation on a per-invocation basis using the Options API.  -->
						    <autoDataGravitation>false</autoDataGravitation>

                        </config>
                    </attribute>
                    ]]>
                    </programlisting>
                </para>
            </section>
       </section>
   </section>

   <section>
     <title>Clustered Cache - Using Invalidation</title>

     <para>If a cache is configured for invalidation rather than replication,
     every time data is changed in a cache other caches in the cluster
     receive a message informing them that their data is now stale and should
     be evicted from memory. Invalidation, when used with a shared cache
     loader (see chapter on Cache Loaders) would cause remote caches to refer
     to the shared cache loader to retrieve modified data. The benefit of
     this is twofold: network traffic is minimised as invalidation messages
     are very small compared to replicating updated data, and also that other
     caches in the cluster look up modified data in a lazy manner, only when
     needed.</para>

     <para>Invalidation messages are sent after each modification (no
     transactions), or at the end of a transaction, upon successful commit.
     This is usually more efficient as invalidation messages can be optimised
     for the transaction as a whole rather than on a per-modification
     basis.</para>

     <para>Invalidation too can be synchronous or asynchronous, and just as
     in the case of replication, synchronous invalidation blocks
     until all caches in the cluster receive invalidation messages and have
     evicted stale data while asynchronous invalidation works in a
     'fire-and-forget' mode, where invalidation messages are broadcast but
     doesn't block and wait for responses.</para>
   </section>
 </chapter>