# List of Protocols

For each protocol define:

- Properties provided

- Required services

- Provided services

- Behavior

## Transport

### UDP

### TCP

### TUNNEL

### JMS

### LOOPBACK

## Initinial membership discovery

### PING

### TCPPING

### TCPGOSSIP

### MPING

## Fragmentation

### FRAG and FRAG2

## Reliable message transmission

### NAKACK

### pbcast.NAKACK

### SMACK

### UNICAST

## Ordering (FIFO covered by NAKACK)

### CAUSAL

### Total Order (sequencer: TOTAL, token-passing: TOTAL_TOKEN

## Group Membership

Group membership takes care of joining new members, handling leave requests by existing members, and handling SUSPECT messages for crashed members, as emitted by failure detection protocols. The algorithm for joining a new member is essentially:

                      - loop
                          - find initial members (discovery)
                          - if no responses:
                              - become singleton group and break out of the loop
                          - else:
                              - determine the coordinator (oldest member) from the responses
                              - send JOIN request to coordinator
                              - wait for JOIN response
                              - if JOIN response received:
                                   - install view and break out of the loop
                              - else
                                   - sleep for 5 seconds and continue the loop

## pbcast.GMS

Properties

Name | Description
--- | ---
join_timeout | Number of milliseconds to wait for a JOIN response from the coordinator, until we send a new JOIN request. Default=5000
join_retry_timeou | Number of ms to wait before sending a new JOIN request
leave_timeout | Number of ms to wait until a LEAVE response has been received from the coordinator. Once this time has elapsed, we leave anyway.
shun |
merge_leader |
print_local_addr | Whether or not to print to stdout the local address of a newly started member. Default is "true". Example:
                                    -------------------------------------------------------
                                    GMS: address is 192.168.5.2:4682
                                    -------------------------------------------------------
merge_timeout |
digest_timeout |
view_ack_collection_timeout |
resume_task_timeout |
disable_initial_coord |
handle_concurrent_startup | Default: "true". Handles concurrent starting of N initial members. Setting it to false is only used for unit tests, where the correctness of the subsequent merge is tested, we don't recommend setting it to false.
num_prev_mbrs |
use_flush |
flush_timeout |
view_bundling | Whether to enable view bundling (default is true). View bundling means that multiple view-affecting requests to GNS, such as JOIN, LEAVE or SUSPECT, are bundled for a number of milliseconds in order to avoid having to generate one view per request. This is especially interesting if we have many members joining or leaving a group at the same time.
max_bundling_time | Max number of milliseconds to wait for subsequent JOIN/LEAVE/SUSPECT requests (default is 50). Therefore, when 5 JOIN or LEAVE requests are received within 50ms, only 1 view will be generated

## Disabling the initial coordinator

Consider the following situation: a new member wants to join a group. The prodedure to do so is:

- Multicast an (unreliable) discovery request (ping)

- Wait for n responses or m milliseconds (whichever is first)

- Every member responds with the address of the coordinator

- If the initial responses are > 0: determine the coordinator and start the JOIN protocolg

- If the initial response are 0: become coordinator, assuming that no one else is out there

However, the problem is that the initial mcast discovery request might get lost, e.g. when multiple members start at the same time, the outgoing network buffer might overflow, and the mcast packet might get dropped. Nobody receives it and thus the sender will not receive any responses, resulting in an initial membership of 0. This could result in multiple coordinators, and multiple subgroups forming. How can we overcome this problem ? There are 3 solutions:

1. Increase the timeout, or number of responses received. This will only help if the reason of the empty membership was a slow host. If the mcast packet was dropped, this solution won't help

2. Add the MERGE(2) protocol. This doesn't actually prevent multiple initial cordinators, but rectifies the problem by merging different subgroups back into one. Note that this involves state merging which needs to be done by the application.

3. (new) Prevent members from becoming coordinator on initial startup. This solution is applicable when we know which member is going to be the initial coordinator of a fresh group. We don't care about afterwards, then coordinatorship can migrate to another member. In this case, we configure the member that is always supposed to be started first with disable_initial_coord=false (the default) and all other members with disable_initial_coord=true.This works as described below.

When the initial membership is received, and is null, and the property disable_initial_coord is true, then we just continue in the loop and retry receving the initial membership (until it is non-null). If the property is false, we are allowed to become coordinator, and will do so. Note that - if a member is started as first member of a group - but its property is set to true, then it will loop until another member whose disable_initial_coord property is set to false, is started.

## Failure Detection

The task of failure detection is to probe members of a group and see whether they are alive. When a member is suspected (= deemed dead), then a SUSPECT message is sent to all nodes of the cluster. It is not the task of the failure detection layer to exclude a crashed member (this is done by the group membership protocol, GMS), but simply to notify everyone that a node in the cluster is suspected of having crashed.

### FD

Failure detection based on heartbeat messages. If reply is not received without timeout ms, max_tries times, a member is declared suspected, and will be excluded by GMS

Each member send a message containing a "FD" - HEARTBEAT header to its neighbor to the right (identified by the ping_dest address). The heartbeats are sent by the inner class Monitor. When the neighbor receives the HEARTBEAT, it replies with a message containing a "FD" - HEARTBEAT_ACK header. The first member watches for "FD" - HEARTBEAT_ACK replies from its neigbor. For each received reply, it resets the last_ack timestamp (sets it to current time) and num_tries counter (sets it to 0). The same Monitor instance that sends heartbeats whatches the difference between current time and last_ack. If this difference grows over timeout, the Monitor cycles several more times (until max_tries) is reached) and then sends a SUSPECT message for the neighbor's address. The SUSPECT message is sent down the stack, is addressed to all members, and is as a regular message with a FdHeader.SUSPECT header.

Properties

Name | Description
--- | ---
timeout | Max number of ms to wait for a response, e.g timeout="2500"
max_tries | Max number of missed responses until a member is declare suspected., e.g. `max_tries="5"`
shun | Once a member is excluded from the group, and then rejoins (e.g. because it didn't crash, but was just slow, or a router that had crashed came back), it will be excluded (shunned) and then has to rejoin. JGroups allows to configure itself such that shunning leads to automatic rejoins and state transfer (default in JBoss). , e.g. `shun="true"`

### FD_SIMPLE

### FD_PING

FD_PING uses a script or command that is run with 1 argument (the host to be pinged) and needs to return 0 (success) or 1 (failure). The default command is /sbin/ping (ping.exe on Windows), but this is user configurable and can be replaced with any user-provided script or executable.

Properties

Name | Description
--- | ---
cmd | The command to be executed, e.g. "/sbin/ping" (or "ping" if found on path)
verbose | Whether or not to show the output of the command. Valid: "true" or "false"

### FD_ICMP

Uses InetAddress.isReachable() to determine whether a host is up or not. Note that this is only available in JDK 5, so reflection is used to determine whether InetAddress provides such a method. If not, an exception will be thrown at protocol initialization time.

The problem with InetAddress.isReachable() is that it may or may not use ICMP in its implementation ! For example, an implementation might try to establish a TCP connection to port 9 (echo service), and - if the echo service is not running - the host would be suspected, although a real ICMP packet would not have suspected the host ! Please check your JDK/OS combo before running this protocol.

Properties

Name | Description
--- | ---
bind_addr | The network interface to be used for sending ICMP packets, e.g. `bind_addr="192.16.8.0.2"`

### FD_SOCK

Failure detection protocol based on a ring of TCP sockets created between group members. Each member in a group connects to its neighbor (last member connects to first) thus forming a ring. Member B is suspected when its neighbor A detects abnormally closed TCP socket (presumably due to a node B crash). However, if a member B is about to leave gracefully, it lets its neighbor A know, so that it does not become suspected.

If you are using a multi NIC machine note that JGroups versions prior to 2.2.8 have FD_SOCK implementation that does not assume this possibility. Therefore JVM can possibly select NIC unreachable to its neighbor and setup FD_SOCK server socket on it. Neighbor would be unable to connect to that server socket thus resulting in immediate suspecting of a member. Suspected member is kicked out of the group, tries to rejoin, and thus goes into join/leave loop. JGroups version 2.2.8 introduces srv_sock_bind_addr property so you can specify network interface where FD_SOCK TCP server socket should be bound. This network interface is most likely the same interface used for other JGroups traffic. JGroups versions 2.2.9 and newer consult bind.address system property or you can specify network interface directly as FD_SOCK bind_addr property.

Properties

Name | Description
--- | ---
bind_addr | The network interface to be used for sending ICMP packets, e.g. `bind_addr="192.16.8.0.2"`

### VERIFY_SUSPECT

## Probabilistic Broadcast

### PBCAST

## Encryption and AUTH

## State Transfer

### pbcast.STATE_TRANSFER

### pbcast.STREAMING_STATE_TRANSFER

#### Overview

In order to transfer application state to a joining member of a group pbcast.STATE_TRANSFER has to load entire state into memory and send it to a joining member. Major limitation of this approach is that the state transfer that is very large (>1Gb) would likely result in OutOfMemoryException. In order to alleviate this problem a new state transfer methodology, based on a streaming state transfer, was introduced in JGroups 2.4

Streaming state transfer supports both partial and full state transfer.

Streaming state transfer provides an InputStream to a state reader and an OutputStream to a state writer. OutputStream and InputStream abstractions enable state transfer in byte chunks thus resulting in smaller memory requirements. For example, if application state consists a huge DOM tree, whose aggregate size is 2GB (and which has partly been passivated to disk), then the state provider (ie. the coordinator) can simply iterate over the DOM tree (activating the parts which have been passivated out to disk), and write to the OutputStream as it traverses the tree. The state receiver will simply read from the InputStream and reconstruct the tree on its side, possibly again passivating parts to disk.

Rather than having to provide a 2GB byte[] buffer, streaming state transfer transfers the state in chunks of N bytes where N is user configurable.

#### API

Streaming state transfer, just as regular byte based state transfer, can be used in both pull and push mode. Similarly to the current getState and setState methods of org.jgroups.MessageListener, application interested in streaming state transfer in a push mode would implement streaming getState method(s) by sending/writing state through a provided OutputStream reference and setState method(s) by receiving/reading state through a provided InputStream reference. In order to use streaming state transfer in a push mode, existing ExtendedMessageListener has been expanded to include additional four methods:

    public interface ExtendedMessageListener
    {

    /*non-streaming callback methods ommitted for clarity*/

        /**
     * Allows an application to write a state through a provided OutputStream.
     * An application is obligated to always close the given OutputStream reference.
     *
     * @param ostream the OutputStream
     * @see OutputStream#close()
     */
    public void getState(OutputStream ostream);

    /**
     * Allows an application to write a partial state through a provided OutputStream.
     * An application is obligated to always close the given OutputStream reference.
     *
     * @param state_id id of the partial state requested
     * @param ostream the OutputStream
     *
     * @see OutputStream#close()
     */
    public void getState(String state_id, OutputStream ostream);

    /**
     * Allows an application to read a state through a provided InputStream.
     * An application is obligated to always close the given InputStream reference.
     *
     * @param istream the InputStream
     * @see InputStream#close()
     */
    public void setState(InputStream istream);

    /**
     * Allows an application to read a partial state through a provided InputStream.
     * An application is obligated to always close the given InputStream reference.
     *
     * @param state_id id of the partial state requested
     * @param istream the InputStream
     *
     * @see InputStream#close()
     */
    public void setState(String state_id, InputStream istream);

    }

For a pull mode (when application uses channel.receive() to fetch events) two new event classes will be introduced:

- StreamingGetStateEvent

- StreamingSetStateEvent

These two events/classes are very similar to existing GetStateEvent and SetStateEvent but introduce a new field; StreamingGetStateEvent has an OutputStream and StreamingSetStateEvent has an InputStream.

The following code snippet demonstrates how to pull events from a channel, processing StreamingGetStateEvent and sending hypothetical state through a provided OutputStream reference. Handling of StreamingSetStateEvent is analogous to this example:

       ...
       Object obj=channel.receive(0);
       if(obj instanceof StreamingGetStateEvent) {
         StreamingGetStateEvent evt=(StreamingGetStateEvent)obj;
         OutputStream oos = null;
         try {
           oos = new ObjectOutputStream(evt.getArg());
           oos.writeObject(state);
           oos.flush();
         } catch (Exception e) {}
         finally{
           try {
             oos.close();
           } catch (IOException e) {
               System.err.println(e);
           }
        }
      }
      ...

API that initiates state transfer on a JChannel level has the following methods:

    public boolean getState(Address target,long timeout)throws ChannelNotConnectedException,ChannelClosedException;
    public boolean getState(Address target,String state_id,long timeout)throws ChannelNotConnectedException,ChannelClosedException;

Introduction of STREAMING_STATE_TRANSFER does not change the current API.

#### Configuration

State transfer type choice is static, implicit and mutually exclusive. JChannel cannot use both STREAMING_STATE_TRANSFER and STATE_TRANSFER in one JChannel configuration.

STREAMING_STATE_TRANSFER allows the following confguration parameters:

Properties

Name | Description
--- | ---
bind_addr | The network interface to be used for receiving of state requests, e.g. `bind_addr="192.16.8.0.2"`
start_port | Port on the bind_addr network interface to be used for receiving of state requests, e.g. `start_port="4444"`
max_pool | Maximum number of threads in the pool providing state requests, default=5, e.g. `max_pool="10"`
pool_thread_keep_alive | Pool thread keep alive in msec, default=30000, e.g. `pool_thread_keep_alive="60000"`
use_reading_thread | Use separate thread for reading state, default=false, e.g. `use_reading_thread="true"`
socket_buffer_size | Chunk size used for state transfer, default=8192 e.g. `socket_buffer_size="32768"`

#### Other considerations

Threading model used for state writing in a member providing state and state reading in a member receiving a state is tunable. For state provider thread pool is used to spawn threads providing state. Thus member providing state, in a push mode, will be able to concurrently serve N state requests where N is max_threads configuration parameter of the thread pool. If there are no further state transfer requests pool threads will be automatically reaped after configurable "pool_thread_keep_alive" timeout expires. For a channel operating in the push mode state reader channel can read state by piggybacking on jgroups protocol stack thread or optionally use a separate thread. State reader should use a separate thread if state reading is expensive (eg. large state, serialization) thus potentially affecting liveness of jgroups protocol thread. Since most state transfers are very short (<2-3 sec) by default we do not use a separate thread.

## Merge

### MERGE, MERGE2

## Debugging

### PERF

### SIZE

### TRACE

### PRINTOBJS

## Misc

### COMPRESS

### pbcast.STABLE

### pbcast.FLUSH

#### Introduction and overview

Flushing forces group members to send all their pending messages prior to a certain event. The process of flushing acquiesces the cluster so that state transfer or a join can be done. It is also called the stop-the-world model as nobody will be able to send messages while a flush is in process. Flush is used:

- State transfer

  When a member requests state transfer it tells everyone to stop sending messages and waits for everyone's ack. Then it asks the application for its state and ships it back to the requester. After the requester has received and set the state successfully, the requester tells everyone to resume sending messages.

- View changes (e.g.a join). Before installing a new view V2, flushing would ensure that all messages *sent* in the current view V1 are indeed *delivered* in V1, rather than in V2 (in all non-faulty members). This is essentially Virtual Synchrony.

FLUSH is designed as another protocol positioned just below the channel, e.g. above STATE_TRANSFER and FC. STATE_TRANSFER and GMS protocol request flush by sending a SUSPEND event up the stack, where it is handled by the FLUSH protcol. The SUSPEND_OK ack sent back by the FLUSH protocol let's the caller know that the flush has completed. When done (e.g. view was installed or state transferred), the protocol sends up a RESUME event, which will allow everyone in the cluster to resume sending.

Channel can be notified that FLUSH phase has been started by turning channel block option on. By default it is turned off. If channel blocking is turned on FLUSH notifies application layer that channel has been blocked by sending EVENT.BLOCK event. Channel responds by sending EVENT.BLOCK_OK event down to FLUSH protocol. We recommend turning on channel block notification only if channel is used in push mode. In push mode application that uses channel can perform block logic by implementing MembershipListener.block() callback method.

Properties

Name | Description
--- | ---
timeout | Maximum time that FLUSH.down() will be blocked before being unblocked. Should be sufficient enough to allow large state transfers,default=8000 msec
block_timeout | Maximum amount of time that FLUSH will be waiting for EVENT.BLOCK_OK once Event.BLOCK has been sent to application level, default=10000 msec

## Use case: joining a group

## Use case: leaving a group

## Use case: removing a crashed member