[[Clustering_and_Domain_Setup_Walkthrough]]
= Clustering and Domain Setup Walkthrough

ifdef::env-github[:imagesdir: ../images/]
:tip-caption: :bulb:
:note-caption: :information_source:
:important-caption: :heavy_exclamation_mark:
:caution-caption: :fire:
:warning-caption: :warning:
endif::[]

In this article, I'd like to show you how to setup WildFly in domain
mode and enable clustering so we could get HA and session replication
among the nodes. It's a step to step guide so you can follow the
instructions in this article and build the sandbox by yourself.

[[preparation-scenario]]
== Preparation & Scenario

[[preparation]]
=== Preparation

We need to prepare two hosts (or virtual hosts) to do the experiment. We
will use these two hosts as following:

* Install Fedora on them (Other linux version may also fine but I'll
use Fedora in this article)

* Make sure that they are in same local network

* Make sure that they can access each other via different TCP/UDP
ports (better turn off firewall and disable SELinux during the experiment
or they will cause network problems).

[[scenario]]
=== Scenario

Here are some details on what we are going to do:

* Let's call one host as 'primary', the other one as 'secondary'.

* Both primary and secondary hosts will run WildFly, and the primary host
will run as a domain controller, the secondary host will under the domain management of primary host.

* Apache httpd will be run on the primary host, and in httpd we will enable the
mod_cluster module. The WildFly on primary and secondary hosts will form a
cluster and discovered by httpd.

image:clustering/Clustering.jpg[images/clustering/Clustering.jpg]

* We will deploy a demo project into the domain, and verify that the project
is deployed into both primary and secondary Host Controllers by the domain controller.
Thus we could see that domain management provide us a single point to manage the
deployments across multiple hosts in a single domain.

* We will access the cluster URL and verify that httpd has distributed
the request to one of the WildFly hosts. So we could see the cluster is
working properly.

* We will try to make a request on cluster, and if the request is
forwarded to the Domain Controller, we then kill the WildFly process on the primary host. After
that we will go on requesting cluster and we should see the request is
forwarded to the secondary Host Controller, but the session is not lost. Our goal is to verify
the HA is working and sessions are replicated.

* After previous step finished, we reconnect the Domain Controller by restarting
it. We should see the Domain Controller is registered back into cluster, also we
should see the secondary Host Controller sees the primary Host Controller as a domain controller again and connect to it.

image:clustering/test_scenario.jpg[images/clustering/test_scenario.jpg]

Please don't worry if you cannot digest so many details currently. Let's
move on and you will get the points step by step.

[[download-wildfly]]
== Download WildFly

First we should download WildFly from the http://wildfly.org/downloads/[WildFly website].

Then I unzipped the package to the primary host and try to make a test run:

[source,subs="verbatim,attributes"]
----
unzip wildfly-{wildflyVersion}.0.0.Final.zip
cd wildfly-{wildflyVersion}.0.0.Final/bin
./domain.sh
----

If everything ok we should see WildFly successfully startup in domain
mode:

[source,subs="verbatim,attributes"]
----
wildfly-{wildflyVersion}.0.0.Final/bin$ ./domain.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /Users/weli/Downloads/wildfly-{wildflyVersion}.0.0.Final

  JAVA: /Library/Java/Home/bin/java

  JAVA_OPTS: -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

=========================================================================
...

[Server:server-two] 14:46:12,375 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly {wildflyVersion}.0.0.Final "Kenny" started in 8860ms - Started 210 of 258 services (8{wildflyVersion} services are lazy, passive or on-demand)
----

Now exit from the primary host and let's repeat the same steps on the secondary host. Finally,
we get WildFly run on both primary and secondary hosts, then we could move on to
next step.

[[domain-configuration]]
== Domain Configuration

[[interface-config-on-primary-hc]]
=== Interface config on the Primary Host Controller

In this section we'll setup both the primary and secondary hosts for them to run in
domain mode. And we will configure the primary host to be the domain controller.

First open the host.xml in the primary host for editing:

[source,options="nowrap"]
----
vi domain/configuration/host.xml
----

The default settings for interface in this file is like:

[source,xml,options="nowrap"]
----
<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
    </interface>
    <interface name="public">
       <inet-address value="${jboss.bind.address:127.0.0.1}"/>
    </interface>
    <interface name="unsecured">
       <inet-address value="127.0.0.1" />
    </interface>
</interfaces>
----

We need to change the address to the management interface so the secondary Host Controller could
connect to the primary Host Controller. The public interface allows the application to be
accessed by non-local HTTP, and the unsecured interface allows remote
RMI access. My primary host's ip address is 10.211.55.7, so I change the
config to:

[source,xml,options="nowrap"]
----
<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:10.211.55.7}"/>
    </interface>
    <interface name="public">
       <inet-address value="${jboss.bind.address:10.211.55.7}"/>
    </interface>
    <interface name="unsecured">
       <inet-address value="10.211.55.7" />
    </interface>
</interfaces>
----

[[interface-config-on-secondary-hc]]
=== Interface config on the Secondary Host Controller

Now we will setup interfaces on the secondary host. Let's edit host.xml. Similar to
the steps on the primary host, open host.xml first:

[source,options="nowrap"]
----
vi domain/configuration/host.xml
----

The configuration we'll use on the secondary host is a little bit different, because
we need to let the secondary Host Controller connect to the primary Host Controller. First we need to set the
hostname. We change the name property from:

[source,xml,options="nowrap"]
----
<host name="primary" xmlns="urn:jboss:domain:3.0">
----

to:

[source,xml,options="nowrap"]
----
<host name="secondary" xmlns="urn:jboss:domain:3.0">
----

Then we need to modify domain-controller section so the secondary Host Controller can connect to
primary Host Controller management port:

[source,xml,options="nowrap"]
----
<domain-controller>
   <remote protocol="remote" host="10.211.55.7" port="9999" />
</domain-controller>
----

As we know, 10.211.55.7 is the ip address of the primary host.
You may use discovery options to define multiple mechanisms to connect
to the remote domain controller:

[source,xml,options="nowrap"]
----
<domain-controller>
 <remote authentication-context="hcAuthContext" >
   <discovery-options>
     <static-discovery name="primary-native" protocol="remote"  host="10.211.55.7" port="9999" />
     <static-discovery name="primary-https" protocol="https-remoting" host="10.211.55.7" port="9993"
                       authentication-context="hcAuthContext"/>
     <static-discovery name="primary-http" protocol="http-remoting" host="10.211.55.7" port="9990" />
   </discovery-options>
 </remote>
</domain-controller>
----

Finally, we also need to configure interfaces section and expose the
management ports to public address:

[source,xml,options="nowrap"]
----
<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:10.211.55.2}"/>
    </interface>
    <interface name="public">
       <inet-address value="${jboss.bind.address:10.211.55.2}"/>
    </interface>
    <interface name="unsecured">
       <inet-address value="10.211.55.2" />
    </interface>
</interfaces>
----

10.211.55.2 is the ip address of the secondary host. Refer to the domain
controller configuration above for an explanation of the management,
public, and unsecured interfaces.

[TIP]

It is easier to turn off all firewalls for testing, but in production,
you need to enable the firewall and allow access to the following ports:
9999.

[[dry-run]]
==== Dry Run

Now everything is set for the two hosts to run in domain mode. Let's
start them by running domain.sh on both hosts. If everything goes fine,
we could see from the log on the primary Host Controller:

[source,options="nowrap"]
----
[Host Controller] 21:30:52,042 INFO  [org.jboss.as.domain] (management-handler-threads - 1) JBAS010918: Registered remote secondary host "secondary"
----

That means all the configurations are correct and two hosts are run in
domain mode now as expected. Hurrah!

[[deployment]]
== Deployment

Now we can deploy a demo project into the domain. I have created a
simple project located at:

[source,options="nowrap"]
----
https://github.com/liweinan/cluster-demo
----

We can use git command to fetch a copy of the demo:

[source,options="nowrap"]
----
git clone git@github.com:liweinan/cluster-demo.git
----

In this demo project we have a very simple web application. In web.xml
we've enabled session replication by adding following entry:

[source,xml,options="nowrap"]
----
<distributable/>
----

And it contains a jsp page called put.jsp which will put current time to
a session entry called 'current.time':

[source,java,options="nowrap"]
----
<%
    session.setAttribute("current.time", new java.util.Date());
%>
----

Then we could fetch this value from get.jsp:

[source,options="nowrap"]
----
The time is <%= session.getAttribute("current.time") %>
----

It's an extremely simple project but it could help us to test the
cluster later: We will access put.jsp from cluster and see the request
are distributed to the primary host, then we disconnect the primary Host Controller and access get.jsp.
We should see the request is forwarded to secondary host but the 'current.time'
value is held by session replication. We'll cover more details on this
one later.

Let's go back to this demo project. Now we need to create a war from it.
In the project directory, run the following command to get the war:

[source,options="nowrap"]
----
mvn package
----

It will generate cluster-demo.war. Then we need to deploy the war into
domain. First we should access the http management console on
primary host (Because primary host is acting as domain controller):

[source,options="nowrap"]
----
http://10.211.55.7:9990
----

It will popup a windows ask you to input account name and password, we
can use the 'admin' account we've added just now. After logging in we
could see the 'Server Instances' window. By default there are three
servers listed, which are:

* server-one

* server-two

* server-three

We could see server-one and server-two are in running status and they
belong to main-server-group; server-three is in idle status, and it
belongs to other-server-group.

All these servers and server groups are set in domain.xml on the primary host.
What we are interested in is the 'other-server-group' in domain.xml:

[source,xml,options="nowrap"]
----
<server-group name="other-server-group" profile="ha">
   <jvm name="default">
       <heap size="64m" max-size="512m"/>
   </jvm>
   <socket-binding-group ref="ha-sockets"/>
</server-group>
----

We could see this server-group is using 'ha' profile, which then uses
'ha-sockets' socket binding group. It enables all the modules we need to
establish cluster later (including infinispan, jgroup and mod_cluster
modules). So we will deploy our demo project into a server that belongs
to 'other-server-group', so 'server-three' is our choice.

[IMPORTANT]

In newer version of WildFly, the profile 'ha' changes to 'full-ha':

[source,xml,options="nowrap"]
----
<server-group name="other-server-group" profile="full-ha">
----

Let's go back to domain controller's management console:

[source,options="nowrap"]
----
http://10.211.55.7:9990
----

Now server-three is not running, so let's click on 'server-three' and
then click the 'start' button at bottom right of the server list. Wait a
moment and server-three should start now.

Now we should also enable 'server-three' on the secondary Host Controller: From the top of menu
list on left side of the page, we could see now we are managing the primary Host Controller
currently. Click on the list, and click 'secondary', then choose
'server-three', and we are in secondary Host Controller management page now.

Then repeat the steps we've done on the primary host to start 'server-three' on
secondary host.

[TIP]

server-three on primary and secondary host are two different hosts, their names
can be different.

After server-three on both primary and secondary hosts are started, we will add our
cluster-demo.war for deployment. Click on the 'Manage Deployments' link
at the bottom of left menu list.

image:clustering/JBoss_Management.png[images/clustering/JBoss_Management.png] +
(We should ensure the server-three should be started on both primary and
secondary hosts)

After enter 'Manage Deployments' page, click 'Add Content' at top right
corner. Then we should choose our cluster-demo.war, and follow the
instruction to add it into our content repository.

Now we can see cluster-demo.war is added. Next we click 'Add to Groups'
button and add the war to 'other-server-group' and then click 'save'.

Wait a few seconds, management console will tell you that the project is
deployed into 'other-server-group'.：

image:clustering/JBoss_Management_2.png[images/clustering/JBoss_Management_2.png]

Please note we have two hosts participate in this server group, so the
project should be deployed in both primary and secondary hosts now - that's the
power of domain management.

Now let's verify this, trying to access cluster-demo from both primary
and secondary hosts, and they should all work now:

[source,options="nowrap"]
----
http://10.211.55.7:8330/cluster-demo/
----

image:clustering/http---10.211.55.7-8330-cluster-demo-.png[images/clustering/http---10.211.55.7-8330-cluster-demo-.png]

[source,options="nowrap"]
----
http://10.211.55.2:8330/cluster-demo/
----

image:clustering/http---10.211.55.2-8330-cluster-demo-.png[images/clustering/http---10.211.55.2-8330-cluster-demo-.png]

Now that we have finished the project deployment and see the usages of
domain controller, we will then head up for using these two hosts to
establish a cluster icon:smile-o[role="yellow"]

[IMPORTANT]

Why is the port number 8330 instead of 8080? Please check the settings
in host.xml on both primary and secondary hosts:

[source,xml,options="nowrap"]
----
<server name="server-three" group="other-server-group" auto-start="false">
    <!-- server-three avoids port conflicts by incrementing the ports in
         the default socket-group declared in the server-group -->
    <socket-bindings port-offset="250"/>
</server>
----

The port-offset is set to 250, so 8080 + 250 = 8330

Now we quit the WildFly process on both primary and secondary hosts. We have some
work left on host.xml configurations. Open the host.xml of primary host, and
do some modifications the servers section from:

[source,xml,options="nowrap"]
----
<server name="server-three" group="other-server-group" auto-start="false">
    <!-- server-three avoids port conflicts by incrementing the ports in
         the default socket-group declared in the server-group -->
    <socket-bindings port-offset="250"/>
</server>
----

to:

[source,xml,options="nowrap"]
----
<server name="server-three" group="other-server-group" auto-start="true">
    <!-- server-three avoids port conflicts by incrementing the ports in
         the default socket-group declared in the server-group -->
    <socket-bindings port-offset="250"/>
</server>
----

We've set auto-start to true so we don't need to enable it in management
console each time WildFly restart. Now open secondary's host.xml, and modify
the server-three section:

[source,xml,options="nowrap"]
----
<server name="server-three-secondary" group="other-server-group" auto-start="true">
    <!-- server-three avoids port conflicts by incrementing the ports in
         the default socket-group declared in the server-group -->
    <socket-bindings port-offset="250"/>
</server>
----

Besides setting auto-start to true, we've renamed the 'server-three' to
'server-three-secondary'. We need to do this because mod_cluster will fail
to register the hosts with same name in a single server group. It will
cause name conflict.

After finishing the above configuration, let's restart two as7 hosts and
go on cluster configuration.

[[cluster-configuration]]
== Cluster Configuration

We will use mod_cluster + apache httpd on primary host as our cluster
controller here. Because WildFly has been configured to support
mod_cluster out of box so it's the easiest way.

[IMPORTANT]

The WildFly domain controller and httpd are not necessary to be on
same host. But in this article I just install them all on primary host for
convenience.

First we need to ensure that httpd is installed:

[source,options="nowrap"]
----
sudo yum install httpd
----

And then we need to download newer version of mod_cluster from its
website:

[source,options="nowrap"]
----
http://www.jboss.org/mod_cluster/downloads
----

The version I downloaded is:

[source,options="nowrap"]
----
http://downloads.jboss.org/mod_cluster/1.1.3.Final/mod_cluster-1.1.3.Final-linux2-x86-so.tar.gz
----

[TIP]

Jean-Frederic has suggested to use mod_cluster 1.2.x. Because 1.1.x it
is affected by CVE-2011-4608

With mod_cluster-1.2.0 you need to add EnableMCPMReceive in the
VirtualHost.

Then we extract it into:

[source,options="nowrap"]
----
/etc/httpd/modules
----

Then we edit httpd.conf:

[source,options="nowrap"]
----
sudo vi /etc/httpd/conf/httpd.conf
----

We should add the modules:

[source,options="nowrap"]
----
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
----

Please note we should comment out:

[source,options="nowrap"]
----
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
----

This is conflicted with cluster module. And then we need to make httpd
to listen to public address so we could do the testing. Because we
installed httpd on primary host so we know the ip address of it:

[source,options="nowrap"]
----
Listen 10.211.55.7:80
----

Then we do the necessary configuration at the bottom of httpd.conf:

[source,options="nowrap"]
----
# This Listen port is for the mod_cluster-manager, where you can see the status of mod_cluster.
# Port 10001 is not a reserved port, so this prevents problems with SELinux.
Listen 10.211.55.7:10001
# This directive only applies to Red Hat Enterprise Linux. It prevents the temmporary
# files from being written to /etc/httpd/logs/ which is not an appropriate location.
MemManagerFile /var/cache/httpd

<VirtualHost 10.211.55.7:10001>

  <Directory />
    Order deny,allow
    Deny from all
    Allow from 10.211.55.
  </Directory>


  # This directive allows you to view mod_cluster status at URL http://10.211.55.4:10001/mod_cluster-manager
  <Location /mod_cluster-manager>
   SetHandler mod_cluster-manager
   Order deny,allow
   Deny from all
   Allow from 10.211.55.
  </Location>

  KeepAliveTimeout 60
  MaxKeepAliveRequests 0

  ManagerBalancerName other-server-group
  AdvertiseFrequency 5

</VirtualHost>
----

[IMPORTANT]

For more details on mod_cluster configurations please see this document:

[source,options="nowrap"]
----
http://docs.jboss.org/mod_cluster/1.1.0/html/Quick_Start_Guide.html
----

[[testing]]
== Testing

If everything goes fine we can start httpd service now:

[source,options="nowrap"]
----
service httpd start
----

Now we access the cluster:

[source,options="nowrap"]
----
http://10.211.55.7/cluster-demo/put.jsp
----

image:clustering/http---10.211.55.7-cluster-demo-put.jsp.png[images/clustering/http---10.211.55.7-cluster-demo-put.jsp.png]

We should see the request is distributed to one of the hosts (primary or
secondary) from the WildFly log. For me the request is sent to primary host:

[source,options="nowrap"]
----
[Server:server-three] 16:06:22,256 INFO  [stdout] (http-10.211.55.7-10.211.55.7-8330-4) Putting date now
----

Now I disconnect the Domain Controller by using the management interface. Select
'runtime' and the server 'primary' in the upper corners.

Select 'server-three' and kick the stop button, the active-icon should
change.

Killing the server by using system commands will have the effect that
the Host-Controller restart the instance immediately!

Then wait for a few seconds and access cluster:

[source,options="nowrap"]
----
http://10.211.55.7/cluster-demo/get.jsp
----

image:clustering/http---10.211.55.7-cluster-demo-get.jsp.png[images/clustering/http---10.211.55.7-cluster-demo-get.jsp.png]

Now the request should be served by the secondary host and we should see the log from
the secondary Host Controller:

[source,options="nowrap"]
----
[Server:server-three-secondary] 16:08:29,860 INFO  [stdout] (http-10.211.55.2-10.211.55.2-8330-1) Getting date now
----

And from the get.jsp we should see that the time we get is the same
we've put by 'put.jsp'. Thus it's proven that the session is correctly
replicated to the secondary host.

Now we restart the primary Host Controller and should see the host is registered back to
cluster.

[TIP]

It doesn't matter if you found the request is distributed to the secondary host at
first time. Then just disconnect the secondary Host Controller and do the testing, the request
should be sent to the primary host instead. The point is we should see the request
is redirect from one host to another and the session is held.

[[special-thanks]]
== Special Thanks

https://community.jboss.org/people/wdfink[Wolf-Dieter Fink] has
contributed the updated add-user.sh usages and configs in host.xml from
7.1.0.Final. +
https://community.jboss.org/people/jfclere[Jean-Frederic Clere] provided
the mod_cluster 1.2.0 usages. +
Misty Stanley-Jones has given a lot of suggestions and helps to make
this document readable.

NOTE: References in this document to JavaServer Pages (JSP) refer to the Jakarta Server Pages unless otherwise noted
