Author: Tom Jenkinson
Level: Advanced
Technologies: JTS, Crash Recovery
Summary: The jts-distributed-crash-rec
quickstart uses JTS and Byteman to demonstrate distributed crash recovery across multiple application servers.
Prerequisites: jts
Target Product: WildFly
Source: https://github.com/wildfly/quickstart/
The jts-distributed-crash-rec
quickstart demonstrates a distributed crash recovery across multiple application servers in WildFly Application Server.
Crash recovery is a key feature provided by an application server and ensures an application's data consistency, even in the presence of failure. Providing reliable crash recovery helps qualify the pedigree of an application server, distributed crash recovery even more so.
This quickstart uses the application components from the jts
quickstart. It provides a Byteman rule to inject a halt into the application server at a crucial point in the phase2 commit of the transaction. Byteman is used solely to raise an artificial fault. An IDE could also provide this simulation, although it is more complex to use an IDE for this purpose.
Apart from that, this quickstart works the same as the jts
quickstart and if the Byteman rule is left out, this quickstart is the JTS quickstart.
As an overview, the sequence of events to expect:
jts
quickstart.jts
quickstart.The application this project produces is designed to be run on WildFly Application Server 11 or later.
All you need to build this project is Java 8.0 (Java SDK 1.8) or later and Maven 3.3.1 or later. See Configure Maven for WildFly 11 to make sure you are configured correctly for testing the quickstarts.
This quickstart uses Byteman to help demonstrate crash recovery. You can find more information about Byteman here: Configure Byteman for Use with the Quickstarts
Follow the instructions here to download and configure Byteman: Download and Configure Byteman
Developers should be familiar with the concepts introduced in the following quickstarts:
IMPORTANT: This quickstart depends on the deployment of the jts
quickstart for its test. Before running this quickstart, see the jts README file for details on how to deploy it.
You can verify the deployment of the jts
quickstart by accessing the following URL: http://localhost:8080/jts-1/.
In the following instructions, replace WILDFLY_HOME
with the actual path to your WildFly installation. The installation path is described in detail here: Use of WILDFLY_HOME and JBOSS_HOME Variables.
Note: This quickstart README file uses the following replaceable values. When you encounter these values in a README file, be sure to replace them with the actual path to the correct WildFly server.
`WILDFLY_HOME` denotes the path to the original WildFly installation.
`WILDFLY_HOME_1` denotes the path to the modified WildFly server 1 configuration.
`WILDFLY_HOME_2` denotes the path to the modified WildFly server 2 configuration.
If you have not yet done so, configure the two application servers and deploy the jts
quickstart. Follow the instructions in the jts README file.
Configure Byteman to halt WildFly server 1
For Linux:
JAVA_OPTS="-javaagent:/BYTEMAN_HOME/lib/byteman.jar=script:/QUICKSTART_HOME/jts-distributed-crash-rec/byteman-scripts/failAfterPrepare.btm ${JAVA_OPTS}"
JAVA_OPTS="-Dorg.jboss.byteman.transform.all -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose=true ${JAVA_OPTS}"
For Windows:
JAVA_OPTS=%JAVA_OPTS% -javaagent:C:BYTEMAN_HOME\lib\byteman.jar=script:C:\QUICKSTART_HOME\jts-distributed-crash-rec\byteman-scripts\failAfterPrepare.btm %JAVA_OPTS%
JAVA_OPTS=%JAVA_OPTS% -Dorg.jboss.byteman.transform.all -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose=true
Start both of the WildFly servers
If you are using Linux:
Server 1: WILDFLY_HOME_1/bin/standalone.sh -c standalone-full.xml -Djboss.tx.node.id=UNIQUE_NODE_ID_1
Server 2: WILDFLY_HOME_2/bin/standalone.sh -c standalone-full.xml -Djboss.tx.node.id=UNIQUE_NODE_ID_2 -Djboss.socket.binding.port-offset=100
If you are using Windows
Server 1: WILDFLY_HOME_1\bin\standalone.bat -c standalone-full.xml -Djboss.tx.node.id=UNIQUE_NODE_ID_1
Server 2: WILDFLY_HOME_2\bin\standalone.bat -c standalone-full.xml -Djboss.tx.node.id=UNIQUE_NODE_ID_2 -Djboss.socket.binding.port-offset=100
Access the application at the following URL: http://localhost:8080/jts-1/
INFO [org.jboss.ejb.client] (default task-2) JBoss EJB Client version 2.1.4.Final-redhat-1
INFO [stdout] (default task-2) Rule.execute called for Fail 2PC after prepare_0
INFO [stdout] (default task-2) HelperManager.install for helper class org.jboss.byteman.rule.helper.Helper
INFO [stdout] (default task-2) calling activated() for helper class org.jboss.byteman.rule.helper.Helper
INFO [stdout] (default task-2) Default helper activated
INFO [stdout] (default task-2) calling installed(Fail 2PC after prepare) for helper classorg.jboss.byteman.rule.helper.Helper
INFO [stdout] (default task-2) Installed rule using default helper : Fail 2PC after prepare
INFO [stdout] (default task-2) Fail 2PC after prepare execute
INFO [stdout] (default task-2) rule.debug{Fail 2PC after prepare} : Prepare completed
INFO [stdout] (default task-2) rule.debug{Fail 2PC after prepare} : !!!killing JVM!!!
At this point, Byteman halts or crashes server 1. You should be able to view the contents of the object store for this server by typing the following in the terminal for server 1. Be sure to replace WILDFLY_HOME_1
with the path to the first server.
tree WILDFLY_HOME_1/standalone/data/tx-object-store
This should display:
WILDFLY_HOME_1/standalone/data/tx-object-store
-- ShadowNoFileLockStore
-- defaultStore
|-- CosTransactions
| -- XAResourceRecord
| -- 0_ffffc0a8013c_38e104bd_4f280cdb_1d
|-- Recovery
| -- FactoryContact
| |-- 0_ffffc0a8013c_38e104bd_4f280cdb_17
| |-- 0_ffffc0a8013c_-671009a_4f280e7e_17
| -- 0_ffffc0a8013c_6d5d82b5_4f280a16_f
|-- RecoveryCoordinator
| -- 0_ffff52e38d0c_c91_4140398c_0
-- StateManager
-- BasicAction
-- TwoPhaseCoordinator
-- ArjunaTransactionImple
-- 0_ffffc0a8013c_38e104bd_4f280cdb_19
View the contents of the object store for the second server by typing the following in the terminal for server 2. Be sure to replace WILDFLY_HOME_2
with the path to the second server.
tree WILDFLY_HOME_2/standalone/data/tx-object-store
This should display:
WILDFLY_HOME_2/standalone/data/tx-object-store
-- ShadowNoFileLockStore
-- defaultStore
|-- CosTransactions
| -- XAResourceRecord
| -- 0_ffffc0a8013c_-2eb1158b_4f280ce3_1e
|-- Recovery
| -- FactoryContact
| |-- 0_ffffc0a8013c_-2eb1158b_4f280ce3_18
| -- 0_ffffc0a8013c_4f6459f0_4f280a24_f
|-- RecoveryCoordinator
| -- 0_ffff52e38d0c_c91_4140398c_0
-- StateManager
-- BasicAction
-- TwoPhaseCoordinator
-- ArjunaTransactionImple
-- ServerTransaction
-- 0_ffffc0a8013c_-2eb1158b_4f280ce3_1a
Disable the Byteman script by restoring the backup configuration file for server 1.
Follow the steps above to restart server 1 and wait for recovery to complete.
IMPORTANT: By default, the recovery process checks the transactional state every two minutes, therefore it can take a while for recovery to happen. Also recovery for each server will take place at its own recovery interval. * You will know when recovery is complete for server 2 as you will see the following in application-server-2 console:
INFO [org.jboss.ejb.client] (RequestProcessor-10) JBoss EJB Client version 2.1.2.Final
INFO [class org.jboss.as.quickstarts.cmt.jts.mdb.HelloWorldMDB] (Thread-3 (group:ActiveMQ-client-global-threads-649946595)) Received Message: Created invoice for customer named: Tom
WARN [com.arjuna.ats.jts] (Periodic Recovery) ARJUNA022223: ExtendedResourceRecord.topLevelCommit caught exception: org.omg.CORBA.OBJECT_NOT_EXIST: ----------BEGIN server-side stack trace----------
org.omg.CORBA.OBJECT_NOT_EXIST: vmcid: SUN minor code: 1004 completed: No
at com.sun.corba.se.impl.logging.POASystemException.nullServant(POASystemException.java:2040)
at com.sun.corba.se.impl.logging.POASystemException.nullServant(POASystemException.java:2062)
at com.sun.corba.se.impl.oa.poa.POAPolicyMediatorImpl_R_AOM.internalGetServant(POAPolicyMediatorImpl_R_AOM.java:68)
at com.sun.corba.se.impl.oa.poa.POAPolicyMediatorBase.getInvocationServant(POAPolicyMediatorBase.java:121)
at com.sun.corba.se.impl.oa.poa.POAImpl.getInvocationServant(POAImpl.java:1634)
at com.sun.corba.se.impl.protocol.CorbaServerRequestDispatcherImpl.getServant(CorbaServerRequestDispatcherImpl.java:326)
at com.sun.corba.se.impl.protocol.CorbaServerRequestDispatcherImpl.getServantWithPI(CorbaServerRequestDispatcherImpl.java:360)
at com.sun.corba.se.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:202)
at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1700)
at com.sun.corba.se.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:180)
at com.sun.corba.se.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:148)
at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:475)
at com.arjuna.ArjunaOTS._ArjunaSubtranAwareResourceStub.commit(_ArjunaSubtranAwareResourceStub.java:124)
at com.arjuna.ats.internal.jts.resources.ExtendedResourceRecord.topLevelCommit(ExtendedResourceRecord.java:502)
...
tree WILDFLY_HOME_1/standalone/data/tx-object-store
You should see this output:
WILDFLY_HOME_1/standalone/data/tx-object-store
-- ShadowNoFileLockStore
-- defaultStore
|-- CosTransactions
| -- XAResourceRecord
|-- Recovery
| -- FactoryContact
| |-- 0_ffffc0a8013c_38e104bd_4f280cdb_17
| |-- 0_ffffc0a8013c_-671009a_4f280e7e_17
| -- 0_ffffc0a8013c_6d5d82b5_4f280a16_f
|-- RecoveryCoordinator
| -- 0_ffff52e38d0c_c91_4140398c_0
-- StateManager
-- BasicAction
-- TwoPhaseCoordinator
-- ArjunaTransactionImple
View the contents of the object store for the second server by typing the following in the terminal for server 2. Be sure to replace WILDFLY_HOME_2
with the path to the second server.
tree WILDFLY_HOME_2/standalone/data/tx-object-store
This should display:
WILDFLY_HOME_2/standalone/data/tx-object-store
-- ShadowNoFileLockStore
-- defaultStore
|-- CosTransactions
| -- XAResourceRecord
|-- Recovery
| -- FactoryContact
| |-- 0_ffffc0a8013c_-2eb1158b_4f280ce3_18
| -- 0_ffffc0a8013c_4f6459f0_4f280a24_f
|-- RecoveryCoordinator
| -- 0_ffff52e38d0c_c91_4140398c_0
-- StateManager
-- BasicAction
-- TwoPhaseCoordinator
-- ArjunaTransactionImple
-- ServerTransaction
After recovery is complete, access the application URL http://localhost:8080/jts-1/customers.jsf. The user you created should now appear in the list.
Do NOT forget to Disable the Byteman script by restoring the backup server configuration file. The Byteman rule must be removed to ensure that your application server will be able to commit 2PC transactions!