JBoss.orgCommunity Documentation

Chapter 1. JCR Performance Tuning Guide

1.1. Introduction
1.2. JCR Performance and Scalability
1.2.1. Cluster configuration
1.2.2. JCR Clustered Performance
1.3. Performance Tuning Guide
1.3.1. JBoss AS Tuning
1.3.2. JCR Cache Tuning
1.3.3. Clustering

1.1. Introduction

This guide will show you possible ways of improving JCR performance.

It is intended to GateIn Administrators and those who wants to use JCR features.

1.2. JCR Performance and Scalability

EC2 network: 1Gbit

Servers hardware:

7.5 GB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage (2×420 GB plus 10 GB root partition)
64-bit platform
I/O Performance: High
API name: m1.large

Note

NFS and statistics (cacti snmp) server were located on one physical server.

JBoss AS configuration

JAVA_OPTS: -Dprogram.name=run.sh -server -Xms4g -Xmx4g -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseParallelGC -Djava.net.preferIPv4Stack=true

Benchmark test using webdav (Complex read/write load test (benchmark)) with 20K same file. To obtain per-operation results we have used custom output from the testscase threads to CSV file.

Read operation:

Warm-up iterations: 100
Run iterations: 2000
Background writing threads: 25
Reading threads: 225

Table 1.1. 

Nodes counttps Responses >2sResponses >4s
15236.87%1.27%
217540.64% 0.08%
323880.49% 0.09%
427060.46% 0.1%

Read operaion with more threads:

Warm-up iterations: 100
Run iterations: 2000
Background writing threads: 50
Reading threads: 450

Table 1.2. 

Nodes counttpsResponses >2sResponses >4s
1116??
215586.1%0.6%
322423.1%0.38%
42756 2.2%0.41%

1.3. Performance Tuning Guide

You can use maxThreads parameter to increase maximum amount of threads that can be launched in AS instance. This can improve performance if you need a high level of concurrency. also you can use -XX:+UseParallelGC java directory to use paralel garbage collector.

Tip

Beware of setting maxThreads too big, this can cause OutOfMemoryError. We've got it with maxThreads=1250 on such machine:

7.5 GB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage (2×420 GB plus 10 GB root partition)
64-bit platform
I/O Performance: High
API name: m1.large
java -Xmx 4g

Cache size

JCR-cluster implementation is built using JBoss Cache as distributed, replicated cache. But there is one particularity related to remove action in it. Speed of this operation depends on the actual size of cache. As many nodes are currently in cache as much time is needed to remove one particular node (subtree) from it.

Eviction

Manipulations with eviction wakeUpInterval value doestn't affect on performance. Performance results with values from 500 up to 3000 are approximately equal.

Transaction Timeout

Using short timeout for long transactions such as Export/Import, removing huge subtree defined timeout may cause TransactionTimeoutException. [TODO] put recomended timeout value

For performance it is better to have loadbalacer, DB server and shared NFS on different computers. If in some reasons you see that one node gets more load than others you can decrease this load using load value in load balancer.

JGroups configuration

It's recommended to use "multiplexer stack" feature present in JGroups. It is set by default in eXo JCR and offers higher performance in cluster, using less network connections also. If there are two or more clusters in your network, please check that they use different ports and different cluster names.

Write performance in cluster

Exo JCR implementation uses Lucene indexing engine to provide search capabilities. But Lucene brings some limitations for write operations: it can perform indexing only in one thread. Thats why write performance in cluster is not higher than in singleton environment. Data is indexed on coordinator node, so increasing write-load on cluster may lead to ReplicationTimeout exception. It occurs because writing threads queue in the indexer and under high load timeout for replication to coordinator will be exceeded.

Taking in consideration this fact, it is recommended to exceed replTimeout value in cache configurations in case of high write-load.

Replication timeout

Some operations may take too much time. So if you get ReplicationTimeoutException try increasing replication timeout:

   <clustering mode="replication" clusterName="${jbosscache-cluster-name}">
      ...
      <sync replTimeout="60000" />
   </clustering>

value is set in miliseconds.