
// $Id: CacheLoader.txt 2073 2006-06-19 12:33:28Z  $


Use cases for CacheLoader
=========================

Defs:
- Member: a process in a cluster
- Persistent state: state managed by CacheLoader, e.g. a DB, or a set of files
- State, or transient state: state that is kept in the cache. Persistent state is always larger than transient state
- Cache: TreeCache
- Coordinator: oldest (first started) member in a cluster
- 2PC: two phase commit


Shared DB (all members connect to the same DB)
==============================================

1. First member in cluster starts:
- Check for other members. If no response, become coordinator
- Create a CacheLoader and call loader.create() and loader.start(). The loader may for example create a connection
  to a database here

2. Second member in cluster starts
- Check for other members.
- If found, ask the oldest member (coordinator) for the transient state (if enabled).
  Otherwise just join the group and don't do state transfer. No state transfer means that the new cache starts cold.
- Don't create a CacheLoader (only the coordinator has a CacheLoader)

3. CRUD operation (put(), remove(), clear() etc)
- If *not* coordinator: don't do anything
- If transactional:
  - At commit time (before returning from beforeCompletion()):
    - Run the cache internal 2PC protocol:
      - In prepare:
        - Lock all nodes
        - Add modifications to list for given transaction
        - Call CacheLoader.prepare(). CacheLoader stores modifications in memory (or writes to secondary storage
          if one_phase is true (asynchronous replication))
      - In commit:
        - Execute all modifications. Note that the CRUDs (e.g. _run()) cannot call CacheLoader.store() again
          (check for TX !)
        - Call CacheLoader.commit()
      - In rollback:
        - Call CacheLoader.rollback()
- If not transactional:
  - Before each CRUD operation (_put(), _remove()) returns, call CacheLoader's store() or remove() operation, e.g.
    _put() would call loader.store()

4. Read operations (get())
- If the value is in the cache, it is returned
- Otherwise, loader.load() is called, and the return value is added to the cache

5. Coordinator leaves or crashes
- The next member in line creates a CacheLoader (essentially the same as #1)

Requirements for Shared DB:
- Get rid of MethodCalls and move to Modifications
- No undo list (compensating transactions): just acquire locks and add modifications to list on prepare(),
  and apply them on commit(), or remove locks on rollback()





LocalDBs (each member has its own local DB)
===========================================

1. First member in cluster starts:
- Check for other members. If no response, become coordinator
- Create a CacheLoader and call loader.create() and loader.start(). The loader may for example create a connection
  to a database here

2. Second member in cluster starts
- Check for other members.
- If found, ask the oldest member (coordinator) for the transient *and* persistent state (state transfer
  has to be enabled). This requires that the state (byte[]) is divided into transient and persistent state.
- The coordinator calls loader.loadEntireState() to get the persistent state and getState() to get the transient
  state. Note that a lock for the entire tree has to be acquired and held for the duration of the fetching of
  both states, in order to prevent another transaction from modifying data while state transfer is in progress.
- The new member calls loader.storeEntireState() and setState() to set the persistent and transient states.

3. CRUD operation (put(), remove(), clear() etc)
- Every member's cache loader performs these operations
- If transactional:
  - At commit time (before returning from beforeCompletion()):
    - Run the cache internal 2PC protocol:
      - In prepare:
        - Lock all nodes
        - Add modifications to list for given transaction
        - Call CacheLoader.prepare(). CacheLoader stores modifications in memory (or writes to secondary storage
          if one_phase is true (asynchronous replication))
      - In commit:
        - Execute all modifications. Note that the CRUDs (e.g. _run()) cannot call CacheLoader.store() again
          (check for TX !)
        - Call CacheLoader.commit()
      - In rollback:
        - Call CacheLoader.rollback()
- If not transactional:
  - Before each CRUD operation (_put(), _remove()) returns, call CacheLoader's store() or remove() operation, e.g.
    _put() would call loader.store()

4. Read operations (get())
- If the value is in the cache, it is returned
- Otherwise, loader.load() is called, and the return value is added to the cache



Design Notes
============


Lazy loading
------------

- When a node is not found in memory, we load it via the CacheLoader (if provided). We do *not* yet load the
  attributes of the node, we only load them when they are accessed, e.g. looking up one value, all values,
  printing the attributes, or adding an attribute

- We use the _UNINITIALIZED key to mark attributes of a node that have yet to be loaded. When uninitialized
  attributes are accessed, we load them via CacheLoader, and then remove the _UNINITIALIZED key

- When we evict attributes from the cache, we essentially remove all attributes and add the _UNINITIALIZED key
  to the attributes, so that the CacheLoader know that the attributes have to be reloaded next time they are
  accessed

- Similarly, the getChildrenNames() method asks the CacheLoader for the names of the children, and creates a node
  for each child found. The attributes are loaded lazily (as described above)



Shared vs unshared store
------------------------
(CacheLoader always set, put() is used as example of CRUD operation)

NON-TRANSACTIONAL

put(): store in CacheLoader, then call local _put(), then
       replicate (calls _put() in all other nodes (excluding self))

-put(): if(shared): no-op, else store in CacheLoader


TRANSACTIONAL

put(): no-op wrt CacheLoader
_put(): no-op wrt CacheLoader
beforeCompletion(): call prepare() in CacheLoader
afterCompletion(): call commit() or rollback() in CacheLoader
prepare(): if(shared): no-op, else call prepare() in CacheLoader
commit(): if(shared): no-op, else call commit() in CacheLoader
rollback(): if(shared): no-op, else call rollback() in CacheLoader


Common format for state transfer
--------------------------------

The goal for a common state transfer format be is to have different cache loaders communicate with each other.
This allows us to for example have a cache loader running Sleepycat on one node, and another cache loader running a file
system based cache loader on another node.
To be able to exchange data correctly between different cache loaders, we need to define a common format for the data,
as it is sent across the wire.

For loadEntireState(), just read all the information from the filesystem, create a NodeData (FQN plus Map) for each node
and serialize it to the output stream. So the byte buffer is a simple sequence of serialized NodeData instances.
One can omit non-leaf nodes who have no attributes. Preferrably traverse the tree in pre-order (node first, children second).

For storeEntireState() you get a sequence of NodeData. Use a MarshalledInputStream and read one NodeData at a time.
Put this information into the database, erasing prev information. Note that since loadEntireState() may omit
certain non-leaf nodes, you have to create parent nodes, e.g. if you have /a/b in your db, and get a NodeData for
a/b/c/d, you have to create /a/b/c.

The format is a byte buffer consisting of a sequence of serialized org.jboss.cache.loader.NodeData, e.g. for the
following tree:

/a
  /b (k1,v1;k2,v2)
    /c
  /b2
     /b3

, the resulting byte buffer is:

<NodeData /a/b><NodeData /a/b/c><NodeData /a/b2/b3>

Note that NodeData for nodes /a and /a/b2 has been omitted since it will be created as result of their children
nodes being created.

To correctly handle classloading within JBoss, we have to use our own customized subclasses of
ObjectInputStream (org.jboss.invocation.MarshalledValueInputStream) and
ObjectOutputStream (org.jboss.invocation.MarshalledValueOutputStream).

The code below shows how storeEntireState() is implemented in FileCacheLoader:

   public void storeEntireState(byte[] state) throws Exception {
      ByteArrayInputStream in_stream=new ByteArrayInputStream(state);
      MarshalledValueInputStream in=new MarshalledValueInputStream(in_stream);
      NodeData nd;

      // remove entire existing state
      this.remove(Fqn.fromString("/"));

      // store new state
      try {
         while(true) {
            nd=(NodeData)in.readObject();
            if(nd.attrs != null)
               this.put(nd.fqn, nd.attrs, true); // creates a node with 0 or more attributes
            else
               this.put(nd.fqn);  // creates a node with null attributes
         }
      }
      catch(EOFException eof_ex) {
         ;
      }
   }


