Infinispan server is a managed, distributed, and clusterable data grid that provides elastic scaling and high performance access to caches from multiple endpoints, such as Hot Rod and REST.
1. Getting Started with Infinispan Server
Quickly set up Infinispan server and learn the basics.
1.1. Infinispan Server Requirements
Check host system requirements for the Infinispan server.
Infinispan server requires a Java Virtual Machine and supports:
-
Java 8
-
Java 11
1.2. Downloading Server Distributions
The Infinispan server distribution is an archive of Java libraries (JAR
files), configuration files, and a data
directory.
Download the Infinispan 10.1 server from Infinispan downloads.
Use the checksum to verify the integrity of your download.
-
Run the
sha1sum
command with the server download archive as the argument, for example:$ sha1sum infinispan-server-${version}.zip
-
Compare with the
SHA-1
checksum value on the Infinispan downloads page.
Infinispan Server README describes the contents of the server distribution.
1.3. Installing Infinispan Server
Extract the Infinispan server archive to any directory on your host.
Use any extraction tool with the server archive, for example:
$ unzip infinispan-server-${version}.zip
The resulting directory is your $ISPN_HOME
.
1.4. Running Infinispan Servers
Spin up Infinispan server instances that automatically form clusters. Learn how to create cache definitions to store your data.
1.4.1. Starting Infinispan Servers
Launch Infinispan server with the startup script.
-
Open a terminal in
$ISPN_HOME
. -
Run the
server
script.- Linux
-
$ bin/server.sh
- Microsoft Windows
-
bin\server.bat
The server gives you these messages when it starts:
INFO [org.infinispan.SERVER] (main) ISPN080004: Protocol SINGLE_PORT listening on 127.0.0.1:11222 INFO [org.infinispan.SERVER] (main) ISPN080001: Infinispan Server ${version} started in 7453ms
Hello Infinispan!
-
Open
127.0.0.1:11222
in any browser to see the Infinispan server welcome message.
Infinispan Server README describes command line
arguments for the server
script.
1.4.2. Verifying Infinispan Cluster Discovery
Infinispan servers running on the same network discover each other with the MPING
protocol.
This procedure shows you how to use Infinispan server command arguments to start two instances on the same host and verify that the cluster view forms.
Start a Infinispan server.
-
Install and run a new Infinispan server instance.
-
Open a terminal in
$ISPN_HOME
. -
Copy the root directory to
server2
.$ cp -r server server2
-
-
Specify a port offset and the location of the
server2
root directory.$ bin/server.sh -o 100 -s server2
Running servers return the following messages when new servers join clusters:
INFO [org.infinispan.CLUSTER] (jgroups-11,<server_hostname>)
ISPN000094: Received new cluster view for channel cluster:
[<server_hostname>|3] (2) [<server_hostname>, <server2_hostname>]
INFO [org.infinispan.CLUSTER] (jgroups-11,<server_hostname>)
ISPN100000: Node <server2_hostname> joined the cluster
Servers return the following messages when they join clusters:
INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main)
ISPN000078: Starting JGroups channel cluster
INFO [org.infinispan.CLUSTER] (main)
ISPN000094: Received new cluster view for channel cluster:
[<server_hostname>|3] (2) [<server_hostname>, <server2_hostname>]
Infinispan Server README describes command line
arguments for the server
script.
1.4.3. Performing Operations with the Infinispan CLI
Connect to servers with the Infinispan command line interface (CLI) to access data and perform administrative functions.
Starting the Infinispan CLI
Start the Infinispan CLI as follows:
-
Open a terminal in
$ISPN_HOME
. -
Run the CLI.
$ bin/cli.sh [disconnected]>
Connecting to Infinispan Servers
Do one of the following:
-
Run the
connect
command to connect to a Infinispan server on the default port of11222
:[disconnected]> connect [hostname1@cluster//containers/default]>
-
Specify the location of a Infinispan server. For example, connect to a local server that has a port offset of 100:
[disconnected]> connect 127.0.0.1:11322 [hostname2@cluster//containers/default]>
Press the tab key to display available commands and options. Use the |
Creating Caches from Templates
Use Infinispan cache templates to add caches with recommended default settings.
-
Create a distributed, synchronous cache from a template and name it "mycache".
[//containers/default]> create cache --template=org.infinispan.DIST_SYNC mycache
Press the tab key after the
--template=
argument to list available cache templates. -
Retrieve the cache configuration.
[//containers/default]> describe caches/mycache { "distributed-cache" : { "mode" : "SYNC", "remote-timeout" : 17500, "state-transfer" : { "timeout" : 60000 }, "transaction" : { "mode" : "NONE" }, "locking" : { "concurrency-level" : 1000, "acquire-timeout" : 15000, "striping" : false } } }
Adding Cache Entries
Add data to caches with the Infinispan CLI.
-
Create a cache named "mycache".
-
Add a key/value pair to
mycache
.[//containers/default]> put --cache=mycache hello world
If the CLI is in the context of a cache, do
put k1 v1
for example:[//containers/default]> cd caches/mycache [//containers/default/caches/mycache]> put hello world
-
List keys in the cache.
[//containers/default]> ls caches/mycache hello
-
Get the value for the
hello
key.-
Navigate to the cache.
[//containers/default]> cd caches/mycache
-
Use the
get
command to retrieve the key value.[//containers/default/caches/mycache]> get hello world
-
Shutting Down Infinispan Servers
Use the CLI to gracefully shutdown running servers. This ensures that Infinispan passivates all entries to disk and persists state.
-
Use the
shutdown server
command to stop individual servers, for example:[//containers/default]> shutdown server server_hostname
-
Use the
shutdown cluster
command to stop all servers joined to the cluster, for example:[//containers/default]> shutdown cluster
Infinispan servers log the following shutdown messages:
INFO [org.infinispan.SERVER] (pool-3-thread-1) ISPN080002: Infinispan Server stopping
INFO [org.infinispan.CONTAINER] (pool-3-thread-1) ISPN000029: Passivating all entries to disk
INFO [org.infinispan.CONTAINER] (pool-3-thread-1) ISPN000030: Passivated 28 entries in 46 milliseconds
INFO [org.infinispan.CLUSTER] (pool-3-thread-1) ISPN000080: Disconnecting JGroups channel cluster
INFO [org.infinispan.CONTAINER] (pool-3-thread-1) ISPN000390: Persisted state, version=<Infinispan version> timestamp=YYYY-MM-DDTHH:MM:SS
INFO [org.infinispan.SERVER] (pool-3-thread-1) ISPN080003: Infinispan Server stopped
INFO [org.infinispan.SERVER] (Thread-0) ISPN080002: Infinispan Server stopping
INFO [org.infinispan.SERVER] (Thread-0) ISPN080003: Infinispan Server stopped
When you shutdown Infinispan clusters, the shutdown messages include:
INFO [org.infinispan.SERVER] (pool-3-thread-1) ISPN080029: Cluster shutdown
INFO [org.infinispan.CLUSTER] (pool-3-thread-1) ISPN000080: Disconnecting JGroups channel cluster
2. Configuring Infinispan Server Networking
Infinispan servers let you configure interfaces and ports to make endpoints available across your network.
By default, Infinispan servers multiplex endpoints to a single TCP/IP port and automatically detect protocols of inbound client requests.
2.1. Server Interfaces
Infinispan servers can use different strategies for binding to IP addresses.
2.1.1. Address Strategy
Uses an inet-address
strategy that maps a single public
interface to the
IPv4 loopback address (127.0.0.1
).
<interfaces>
<interface name="public">
<inet-address value="${infinispan.bind.address:127.0.0.1}"/>
</interface>
</interfaces>
You can use the CLI |
2.1.2. Loopback Strategy
Selects a loopback address.
-
IPv4 the address block
127.0.0.0/8
is reserved for loopback addresses. -
IPv6 the address block
::1
is the only loopback address.
<interfaces>
<interface name="public">
<loopback/>
</interface>
</interfaces>
2.1.3. Non-Loopback Strategy
Selects a non-loopback address.
<interfaces>
<interface name="public">
<non-loopback/>
</interface>
</interfaces>
2.1.4. Network Address Strategy
Selects networks based on IP address.
<interfaces>
<interface name="public">
<inet-address value="10.1.2.3"/>
</interface>
</interfaces>
2.1.5. Any Address Strategy
Selects the INADDR_ANY
wildcard address. As a result
Infinispan servers listen on all interfaces.
<interfaces>
<interface name="public">
<any-address/>
</interface>
</interfaces>
2.1.6. Link Local Strategy
Selects a link-local IP address.
-
IPv4 the address block
169.254.0.0/16
(169.254.0.0 – 169.254.255.255
) is reserved for link-local addressing. -
IPv6 the address block
fe80::/10
is reserved for link-local unicast addressing.
<interfaces>
<interface name="public">
<inet-address value="10.1.2.3"/>
</interface>
</interfaces>
2.1.7. Site Local Strategy
Selects a site-local (private) IP address.
-
IPv4 the address blocks
10.0.0.0/8
,172.16.0.0/12
, and192.168.0.0/16
are reserved for site-local addressing. -
IPv6 the address block
fc00::/7
is reserved for site-local unicast addressing.
<interfaces>
<interface name="public">
<inet-address value="10.1.2.3"/>
</interface>
</interfaces>
2.1.8. Match Host Strategy
Resolves the host name and selects one of the IP addresses that is assigned to any network interface.
Infinispan servers enumerate all available operating system interfaces to locate IP addresses resolved from the host name in your configuration.
<interfaces>
<interface name="public">
<match-host value="my_host_name"/>
</interface>
</interfaces>
2.1.9. Match Interface Strategy
Selects an IP address assigned to a network interface that matches a regular expression.
Infinispan servers enumerate all available operating system interfaces to locate the interface name in your configuration.
Use regular expressions with this strategy for additional flexibility. |
<interfaces>
<interface name="public">
<match-interface value="eth0"/>
</interface>
</interfaces>
2.1.10. Match Address Strategy
Similar to inet-address
but selects an IP address using a regular expression.
Infinispan servers enumerate all available operating system interfaces to locate the IP address in your configuration.
Use regular expressions with this strategy for additional flexibility. |
<interfaces>
<interface name="public">
<match-address value="132\..*"/>
</interface>
</interfaces>
2.1.11. Fallback Strategy
Interface configurations can include multiple strategies. Infinispan servers try each strategy in the declared order.
For example, with the following configuration, Infinispan servers first
attempt to match a host, then an IP address, and then fall back to the
INADDR_ANY
wildcard address:
<interfaces>
<interface name="public">
<match-host value="my_host_name"/>
<match-address value="132\..*"/>
<any-address/>
</interface>
</interfaces>
2.1.12. Changing the Default Bind Address for Infinispan Servers
You can use the server -b
switch or the infinispan.bind.address
system property to bind to a different address.
For example, bind the public
interface to 127.0.0.2
as follows:
- Linux
$ bin/server.sh -b 127.0.0.2
- Windows
bin\server.bat -b 127.0.0.2
2.2. Socket Bindings
Socket bindings map endpoint connectors to server interfaces and ports.
By default, Infinispan servers provide the following socket bindings:
<socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
<socket-binding name="default" port="${infinispan.bind.port:11222}"/>
<socket-binding name="memcached" port="11221"/>
</socket-bindings>
-
socket-bindings
declares the default interface and port offset. -
default
binds to hotrod and rest connectors to the default port11222
. -
memcached
binds the memcached connector to port11221
.The memcached endpoint is disabled by default.
To override the default interface for socket-binding
declarations, specify the interface
attribute.
For example, you add an interface
declaration named "private":
<interfaces>
...
<interface name="private">
<inet-address value="10.1.2.3"/>
</interface>
</interfaces>
You can then specify interface="private"
in a socket-binding
declaration to
bind to the private IP address, as follows:
<socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
...
<socket-binding name="private_binding" interface="private" port="1234"/>
</socket-bindings>
2.2.1. Specifying Port Offsets
Configure port offsets with Infinispan servers when running multiple instances
on the same host. The default port offset is 0
.
Use the -o
switch with the Infinispan CLI or the
infinispan.socket.binding.port-offset
system property to set port offsets.
For example, start a server instance with an offset of 100
as follows. With the default configuration, this results in the Infinispan server listening on port 11322
.
- Linux
$ bin/server.sh -o 100
- Windows
bin\server.bat -o 100
2.3. Infinispan Protocol Handling
Infinispan servers use a router connector to expose multiple protocols over
the same TCP port, 11222
. Using a single port for multiple protocols
simplifies configuration and management and increases security by reducing the
attack surface for unauthorized users.
Infinispan servers handle HTTP/1.1, HTTP/2, and Hot Rod protocol requests via
port 11222
as follows:
- HTTP/1.1 upgrade headers
-
Client requests can include the
HTTP/1.1 upgrade
header field to initiate HTTP/1.1 connections with Infinispan servers. Client applications can then send theUpgrade: protocol
header field, whereprotocol
is a Infinispan server endpoint. - Application-Layer Protocol Negotiation (ALPN)/Transport Layer Security (TLS)
-
Client applications specify Server Name Indication (SNI) mappings for Infinispan server endpoints to negotiate protocols in a secure manner.
- Automatic Hot Rod detection
-
Client requests that include Hot Rod headers automatically route to Hot Rod endpoints if the single port router configuration includes Hot Rod.
2.3.1. Configuring Clients for ALPN
Configure clients to provide ALPN messages for protocol negotiation during TLS handshakes with Infinispan servers.
-
Enable Infinispan server endpoints with encryption.
-
Provide your client application with the appropriate libraries to handle ALPN/TLS exchanges with Infinispan servers.
Infinispan uses Wildfly OpenSSL bindings for Java.
-
Configure clients with trust stores as appropriate.
ConfigurationBuilder builder = new ConfigurationBuilder()
.addServers("127.0.0.1:11222");
builder.security().ssl().enable()
.trustStoreFileName("truststore.pkcs12")
.trustStorePassword(DEFAULT_TRUSTSTORE_PASSWORD.toCharArray());
RemoteCacheManager remoteCacheManager = new RemoteCacheManager(builder.build());
RemoteCache<String, String> cache = remoteCacheManager.getCache("default"");
infinispan.client.hotrod.server_list = 127.0.0.1:11222
infinispan.client.hotrod.use_ssl = true
infinispan.client.hotrod.trust_store_file_name = truststore.pkcs12
infinispan.client.hotrod.trust_store_password = trust_store_password
3. Configuring Infinispan Server Endpoints
Infinispan servers provide listener endpoints that handle requests from remote client applications.
3.1. Infinispan Endpoints
Infinispan endpoints expose the CacheManager
interface over different
connector protocols so you can remotely access data and perform operations to
manage and maintain Infinispan clusters.
You can define multiple endpoint connectors on different socket bindings.
3.1.1. Hot Rod
Hot Rod is a binary TCP client-server protocol designed to provide faster data access and improved performance in comparison to text-based protocols.
Infinispan provides Hot Rod client libraries in Java, C++, C#, Node.js and other programming languages.
Infinispan uses topology caches to provide clients with cluster views. Topology caches contain entries that map internal JGroups transport addresses to exposed Hot Rod endpoints.
When client send requests, Infinispan servers compare the topology ID in request headers with the topology ID from the cache. Infinispan servers send new topology views if client have older topology IDs.
Cluster topology views allow Hot Rod clients to immediately detect when nodes join and leave, which enables dynamic load balancing and failover.
In distributed cache modes, the consistent hashing algorithm also makes it possible to route Hot Rod client requests directly to primary owners.
3.1.2. REST
Infinispan exposes a RESTful interface that allows HTTP clients to access data, monitor and maintain clusters, and perform administrative operations.
You can use standard HTTP load balancers to provide clients with load balancing and failover capabilities. However, HTTP load balancers maintain static cluster views and require manual updates when cluster topology changes occur.
3.1.3. Memcached
Infinispan provides an implementation of the Memcached text protocol for remote client access.
The Infinispan Memcached server supports clustering with replicated and distributed cache modes.
There are some Memcached client implementations, such as the Cache::Memcached Perl client, that can offer load balancing and failover detection capabilities with static lists of Infinispan server addresses that require manual updates when cluster topology changes occur.
3.2. Endpoint Connectors
You configure Infinispan server endpoints with connector declarations that specify socket bindings, authentication mechanisms, and encryption configuration.
The default endpoint connector configuration is as follows:
<endpoints socket-binding="default">
<hotrod-connector name="hotrod"/>
<rest-connector name="rest"/>
<memcached-connector socket-binding="memcached"/>
</endpoints>
-
endpoints
contains endpoint connector declarations and defines global configuration for endpoints such as default socket bindings, security realms, and whether clients must present valid TLS certificates. -
<hotrod-connector name="hotrod"/>
declares a Hot Rod connector. -
<rest-connector name="rest"/>
declares a Hot Rod connector. -
<memcached-connector socket-binding="memcached"/>
declares a Memcached connector that uses the memcached socket binding.
urn:infinispan:server schema provides all available endpoint configuration.
3.2.1. Hot Rod Connectors
Hot Rod connector declarations enable Hot Rod servers.
<hotrod-connector name="hotrod">
<topology-state-transfer />
<authentication>
...
</authentication>
<encryption>
...
</encryption>
</hotrod-connector>
-
name="hotrod"
logically names the Hot Rod connector. -
topology-state-transfer
tunes the state transfer operations that provide Hot Rod clients with cluster topology. -
authentication
configures SASL authentication mechanisms. -
encryption
configures TLS settings for client connections.
urn:infinispan:server schema provides all available Hot Rod connector configuration.
3.2.2. REST Connectors
REST connector declarations enable REST servers.
<rest-connector name="rest">
<authentication>
...
</authentication>
<cors-rules>
...
</cors-rules>
<encryption>
...
</encryption>
</rest-connector>
-
name="rest"
logically names the REST connector. -
authentication
configures authentication mechanisms. -
cors-rules
specifies CORS (Cross Origin Resource Sharing) rules for cross-domain requests. -
encryption
configures TLS settings for client connections.
urn:infinispan:server schema provides all available REST connector configuration.
3.2.3. Memcached Connectors
Memcached connector declarations enable Memcached servers.
Infinispan servers do not enable Memcached connectors by default. |
<memcached-connector name="memcached" socket-binding="memcached" cache="mycache" />
-
name="memcached"
logically names the Memcached connector. -
socket-binding="memcached"
declares a unique socket binding for the Memcached connector. -
cache="mycache"
names the cache that the Memcached connector exposes. The default ismemcachedCache
.Memcached connectors expose a single cache only. To expose multiple caches through the Memcached endpoint, you must declare additional connectors. Each Memcached connector must also have a unique socket binding.
urn:infinispan:server schema provides all available Memcached connector configuration.
4. Monitoring Infinispan Servers
4.1. Working with Infinispan Server Logs
Infinispan uses Apache Log4j 2 to provide configurable logging mechanisms that capture details about the environment and record cache operations for troubleshooting purposes and root cause analysis.
4.1.1. Infinispan Log Files
Infinispan writes log messages to the following directory:
$ISPN_HOME/${infinispan.server.root}/log
server.log
-
Messages in human readable format, including boot logs that relate to the server startup.
Infinispan creates this file by default when you launch servers. server.log.json
-
Messages in JSON format that let you parse and analyze Infinispan logs.
Infinispan creates this file when you enable theJSON-FILE
appender.
4.1.2. Configuring Infinispan Log Properties
You configure Infinispan logs with log4j2.xml
, which is described in the
Log4j 2 manual.
-
Open
$ISPN_HOME/${infinispan.server.root}/conf/log4j2.xml
with any text editor. -
Change logging configuration as appropriate.
-
Save and close
log4j2.xml
.
Log Levels
Log levels indicate the nature and severity of messages.
Log level | Description |
---|---|
|
Fine-grained debug messages, capturing the flow of individual requests through the application. |
|
Messages for general debugging, not related to an individual request. |
|
Messages about the overall progress of applications, including lifecycle events. |
|
Events that can lead to error or degrade performance. |
|
Error conditions that might prevent operations or activites from being successful but do not prevent applications from running. |
|
Events that could cause critical service failure and application shutdown. |
In addition to the levels of individual messages presented above, the configuration allows two more values: ALL
to include all messages, and OFF
to exclude all messages.
Infinispan Log Categories
Infinispan provides categories for INFO
, WARN
, ERROR
, FATAL
level messages that organize logs by functional area.
org.infinispan.CLUSTER
-
Messages specific to Infinispan clustering that include state transfer operations, rebalancing events, partitioning, and so on.
org.infinispan.CONFIG
-
Messages specific to Infinispan configuration.
org.infinispan.CONTAINER
-
Messages specific to the data container that include expiration and eviction operations, cache listener notifications, transactions, and so on.
org.infinispan.PERSISTENCE
-
Messages specific to cache loaders and stores.
org.infinispan.SECURITY
-
Messages specific to Infinispan security.
org.infinispan.SERVER
-
Messages specific to Infinispan servers.
org.infinispan.XSITE
-
Messages specific to cross-site replication operations.
Log Appenders
Log appenders define how Infinispan records log messages.
- CONSOLE
-
Write log messages to the host standard out (
stdout
) or standard error (stderr
) stream.
Uses theorg.apache.logging.log4j.core.appender.ConsoleAppender
class by default. - FILE
-
Write log messages to a file.
Uses theorg.apache.logging.log4j.core.appender.RollingFileAppender
class by default. - JSON-FILE
-
Write log messages to a file in JSON format.
Uses theorg.apache.logging.log4j.core.appender.RollingFileAppender
class by default.
Log Patterns
The CONSOLE
and FILE
appenders use a
PatternLayout
to format the log messages according to a pattern.
An example is the default pattern in the FILE appender:
%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c{1}] %m%throwable%n
-
%d{yyyy-MM-dd HH:mm:ss,SSS}
adds the current time and date. -
%-5p
specifies the log level, aligned to the right. -
%t
adds the name of the current thread. -
%c{1}
adds the short name of the logging category. -
%m
adds the log message. -
%throwable
adds the exception stack trace. -
%n
adds a new line.
Patterns are fully described in
the PatternLayout
documentation
.
Enabling and Configuring the JSON Log Handler
Infinispan provides a JSON log handler to write messages in JSON format.
Ensure that Infinispan is not running. You cannot dynamically enable log handlers.
When you start Infinispan, it writes each log message as a JSON map in the following file:
$ISPN_HOME/${infinispan.server.root}/log/server.log.json
4.1.3. Access Logs
Hot Rod and REST endpoints can record all inbound client requests as log entries with the following categories:
-
org.infinispan.HOTROD_ACCESS_LOG
logging category for the Hot Rod endpoint. -
org.infinispan.REST_ACCESS_LOG
logging category for the REST endpoint.
Enabling Access Logs
Access logs for Hot Rod and REST endpoints are disabled by default.
To enable either logging category, set the level to TRACE
in the
Infinispan logging configuration, as in the following example:
<Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="INFO">
<AppenderRef ref="HR-ACCESS-FILE"/>
</Logger>
Access Log Properties
The default format for access logs is as follows:
`%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] "%X{method} %m %X{protocol}" %X{status} %X{requestSize} %X{responseSize} %X{duration}%n`
The preceding format creates log entries such as the following:
127.0.0.1 - [DD/MM/YYYY:HH:MM:SS +0000] "PUT /rest/v2/caches/default/key HTTP/1.1" 404 5 77 10
Logging properties use the %X{name}
notation and let you modify the format of access logs. The following are the default logging properties:
Property | Description |
---|---|
|
Either the |
|
Principal name, if using authentication. |
|
Method used. |
|
Protocol used. |
|
An HTTP status code for the REST endpoint. |
|
Size, in bytes, of the request. |
|
Size, in bytes, of the response. |
|
Number of milliseconds that the server took to handle the request. |
Use the header name prefixed with |
4.2. Retrieving Server Health Statistics
Monitor the health of your Infinispan clusters in the following ways:
-
Programmatically with
embeddedCacheManager.getHealth()
method calls. -
JMX MBeans
-
Infinispan REST Server
4.2.1. Accessing the Health API via JMX
Retrieve Infinispan cluster health statistics via JMX.
-
Connect to Infinispan server using any JMX capable tool such as JConsole and navigate to the following object:
org.infinispan:type=CacheManager,name="default",component=CacheContainerHealth
-
Select available MBeans to retrieve cluster health statistics.
4.2.2. Accessing the Health API via REST
Get Infinispan cluster health via the REST API.
-
Invoke a
GET
request to retrieve cluster health.GET /rest/v2/cache-managers/{cacheManagerName}/health
Infinispan responds with a JSON
document such as the following:
{
"cluster_health":{
"cluster_name":"ISPN",
"health_status":"HEALTHY",
"number_of_nodes":2,
"node_names":[
"NodeA-36229",
"NodeB-28703"
]
},
"cache_health":[
{
"status":"HEALTHY",
"cache_name":"___protobuf_metadata"
},
{
"status":"HEALTHY",
"cache_name":"cache2"
},
{
"status":"HEALTHY",
"cache_name":"mycache"
},
{
"status":"HEALTHY",
"cache_name":"cache1"
}
]
}
Get cache manager status as follows:
|
See the REST v2 (version 2) API documentation for more information.
4.3. Collecting Infinispan Metrics
Infinispan servers provide monitoring data through an HTTP metrics
endpoint
that exposes OS, JVM, and application-level statistics.
-
Start at least one Infinispan server.
-
Test the metrics endpoint with any HTTP client, as in the following examples:
To get metrics in Prometheus or OpenMetrics format:
$ curl -v http://localhost:11222/metrics
To get metrics in Microprofile JSON format:
$ curl --header "Accept: application/json" http://localhost:11222/metrics
If you configure Infinispan servers with security, you should include the appropriate credentials or client certificates to access the metrics endpoint.
Infinispan responds with monitoring data.
Configure monitoring applications to collect Infinispan metrics from the
endpoint as appropriate. For example, add the following to your
prometheus.yml
file:
static_configs: - targets: ['localhost:9090', 'localhost:11222']
4.4. Monitoring Infinispan Servers Over JMX
You can monitor an Infinispan Server over JMX in two ways:
-
Use JConsole or VisualVM running locally as the same user. This will use a local jvmstat connection and requires no additional setup
-
Use JMX remoting (aka JSR-160) to connect from any host. This requires connecting through the management port (usually 9990) using a special protocol which respects the server security configuration
To setup a client for JMX remoting you need to add the $ISPN_HOME/bin/client/jboss-client.jar to your client’s classpath and use one of the following service URLs:
-
service:jmx:remote-http-jmx://host:port
for plain connections through the management interface -
service:jmx:remote-https-jmx://host:port
for TLS connections through the management interface (although this requires having the appropriate keys available) -
service:jmx:remoting-jmx://localhost:port
for connections through the remoting interface (necessary for connecting to individual servers in a domain)
The JMX subsystem registers a service with the Remoting endpoint so that remote access to JMX can be obtained over the exposed Remoting connector. This is switched on by default in standalone mode and accessible over port 9990 but in domain mode it is switched off so it needs to be enabled. In domain mode the port will be the port of the Remoting connector for the Server instance to be monitored.
<subsystem xmlns="urn:jboss:domain:jmx:1.3">
<expose-resolved-model/>
<expose-expression-model/>
<remoting-connector use-management-endpoint="false"/>
</subsystem>
4.4.1. JMX Monitoring
Management of Infinispan instances is all about exposing as much relevant statistical information that allows administrators to get a view of the state of each Infinispan instance. Taking in account that a single installation could be made up of several tens or hundreds Infinispan instances, providing clear and concise information in an efficient manner is imperative. The following sections dive into the range of management tooling that Infinispan provides.
Any management tool that supports JMX already has basic support for Infinispan. However, custom plugins could be written to adapt the JMX information for easier consumption. |
JMX
Over the years, JMX has become the de facto standard for management and administration of middleware and as a result, the Infinispan team has decided to standardize on this technology for the exposure of management and statistical information.
Understanding The Exposed MBeans
By connecting to the VM(s) where Infinispan is running with a standard JMX GUI such as JConsole or VisualVM you should find the following MBeans:
-
For CacheManager level JMX statistics, without further configuration, you should see an MBean called org.infinispan:type=CacheManager,name="DefaultCacheManager" with properties specified by the CacheManager MBean .
-
Using the cacheManagerName attribute in globalJmxStatistics XML element, or using the corresponding GlobalJmxStatisticsConfigurationBuilder.cacheManagerName(String cacheManagerName) call, you can name the cache manager in such way that the name is used as part of the JMX object name. So, if the name had been "Hibernate2LC", the JMX name for the cache manager would have been: org.infinispan:type=CacheManager,name="Hibernate2LC" . This offers a nice and clean way to manage environments where multiple cache managers are deployed, which follows JMX best practices .
-
For Cache level JMX statistics, you should see several different MBeans depending on which configuration options have been enabled. For example, if you have configured a write behind cache store, you should see an MBean exposing properties belonging to the cache store component. All Cache level MBeans follow the same format though which is the following:
org.infinispan:type=Cache,name="${name-of-cache}(${cache-mode})",manager="${name-of-cache-manager}",component=${component-name}
where: -
${name-of-cache} has been substituted by the actual cache name. If this cache represents the default cache, its name will be
___defaultCache
. -
${cache-mode} has been substituted by the cache mode of the cache. The cache mode is represented by the lower case version of the possible enumeration values shown here.
-
${name-of-cache-manager} has been substituted by the name of the cache manager to which this cache belongs. The name is derived from the cacheManagerName attribute value in
globalJmxStatistics
element. -
${component-name} has been substituted by one of the JMX component names in the JMX reference documentation .
For example, the cache store JMX component MBean for a default cache configured with synchronous distribution would have the following name: org.infinispan:type=Cache,name="___defaultcache(dist_sync)",manager="DefaultCacheManager",component=CacheStore
Please note that cache and cache manager names are quoted to protect against illegal characters being used in these user-defined names.
Enabling JMX Statistics
The MBeans mentioned in the previous section are always created and registered in the MBeanServer allowing you to manage your caches but some of their attributes do not expose meaningful values unless you take the extra step of enabling collection of statistics. Gathering and reporting statistics via JMX can be enabled at 2 different levels:
The CacheManager is the entity that governs all the cache instances that have been created from it. Enabling CacheManager statistics collections differs depending on the configuration style:
-
If configuring the CacheManager via XML, make sure you add the following XML under the
<cache-container />
element:<cache-container statistics="true"/>
-
If configuring the CacheManager programmatically, simply add the following code:
GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics().enable();
At this level, you will receive management information generated by individual cache instances. Enabling Cache statistics collections differs depending on the configuration style:
-
If configuring the Cache via XML, make sure you add the following XML under the one of the top level cache elements, such as
<local-cache />
:
<local-cache statistics="true"/>
-
If configuring the Cache programmatically, simply add the following code:
ConfigurationBuilder configurationBuilder = ...
configurationBuilder.jmxStatistics().enable();
Monitoring cluster health
It is also possible to monitor Infinispan cluster health using JMX. On CacheManager there’s an additional object called CacheContainerHealth
. It contains the following attributes:
-
cacheHealth - a list of caches and corresponding statuses (HEALTHY, DEGRADED or HEALTHY_REBALANCING)
-
clusterHealth - overall cluster health
-
clusterName - cluster name
-
freeMemoryKb - Free memory obtained from JVM runtime measured in KB
-
numberOfCpus - The number of CPUs obtained from JVM runtime
-
numberOfNodes - The number of nodes in the cluster
-
totalMemoryKb - Total memory obtained from JVM runtime measured in KB
Multiple JMX Domains
There can be situations where several CacheManager instances are created in a single VM, or Cache names belonging to different CacheManagers under the same VM clash.
Using different JMX domains for multi cache manager environments should be last resort. Instead, it’s possible to name a cache manager in such way that it can easily be identified and used by monitoring tools. For example:
-
Via XML:
<cache-container statistics="true" name="Hibernate2LC"/>
-
Programmatically:
GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics()
.enable()
.cacheManagerName("Hibernate2LC");
Using either of these options should result on the CacheManager MBean name being: org.infinispan:type=CacheManager,name="Hibernate2LC"
For the time being, you can still set your own jmxDomain if you need to and we also allow duplicate domains, or rather duplicate JMX names, but these should be limited to very special cases where different cache managers within the same JVM are named equally.
Registering MBeans In Non-Default MBean Servers
Let’s discuss where Infinispan registers all these MBeans. By default, Infinispan registers them in the standard JVM MBeanServer platform . However, users might want to register these MBeans in a different MBeanServer instance. For example, an application server might work with a different MBeanServer instance to the default platform one. In such cases, users should implement the MBeanServerLookup interface provided by Infinispan so that the getMBeanServer() method returns the MBeanServer under which Infinispan should register the management MBeans. Once you have your implementation ready, simply configure Infinispan with the fully qualified name of this class. For example:
-
Via XML:
<cache-container statistics="true">
<jmx mbean-server-lookup="com.acme.MyMBeanServerLookup" />
</cache-container>
-
Programmatically:
GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics()
.enable()
.mBeanServerLookup(new com.acme.MyMBeanServerLookup());
Available MBeans
For a complete list of available MBeans, refer to the JMX reference documentation
5. Securing Infinispan Servers
5.1. Security Concepts
5.1.1. Authorization
Just like embedded mode, the server supports cache authorization using the same configuration, e.g.:
<infinispan>
<cache-container default-cache="secured" name="secured">
<security>
<authorization>
<identity-role-mapper />
<role name="admin" permissions="ALL" />
<role name="reader" permissions="READ" />
<role name="writer" permissions="WRITE" />
<role name="supervisor" permissions="READ WRITE EXEC"/>
</authorization>
</security>
<local-cache name="secured">
<security>
<authorization roles="admin reader writer supervisor" />
</security>
</local-cache>
</cache-container>
</infinispan>
5.1.2. Server Realms
Infinispan Server security is built around the concept of security realms. Security Realms are used by the server to provide identity, encryption, authentication and authorization information for the server endpoints. A security realm is identified by a name and combines any number of identities and sub-realms, each one also identified by a name. Bear in mind that the choice of authentication mechanism you select for the protocols limits the type of authentication sources, since the credentials must be in a format supported by the algorithm itself (e.g. pre-digested passwords for the digest algorithm).
Server identities
The Infinispan Server supports two kinds of identities:
-
SSL identities, backed by a keystore.
-
Kerberos identities, backed by a keytab file.
SSL identities and endpoint encryption
A SSL identity is represented by a certificate (or a chain of certificates) stored within a keystore. Infinispan Server automatically recognizes and supports multiple keystore formats: JKS, JCEKS, PKCS12, BKS, BCFKS and UBER. When the server endpoint is configured with a realm which has an SSL identity it will automatically enable encryption.
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<server-identities>
<ssl>
<keystore path="server.p12" relative-to="infinispan.server.config.path" keystore-password="secret" alias="server"/>
</ssl>
</server-identities>
</security-realm>
</security-realms>
</security>
The ideal server certificate is one which is signed by a trusted Certificate Authority (be it a Root or Intermediate CA), and configuring clients to trust that authority.
The following sequence of shell commands uses Java’s keytool
to create a CA certificate, a Server certificate signed by the CA and imports both into a keystore to be used by the server.
# Generate a CA certificate and keystore
keytool -genkeypair -validity 365 -keyalg RSA -keysize 2048 -noprompt -storepass secret -alias ca -dname "CN=CA,OU=Operations,O=Organization,L=ACME" -keystore ca.p12 -ext bc:c
# Export the CA certificate from the keystore
keytool -exportcert -validity 365 -keyalg RSA -keysize 2048 -noprompt -storepass secret -alias ca -keystore ca.p12 -file ca.cer
# Generate a server certificate and keystore
keytool -genkeypair -validity 365 -keyalg RSA -keysize 2048 -noprompt -storepass secret -alias server -dname "CN=Server,OU=Operations,O=Organization,L=ACME" -keystore server.p12
# Create a certificate signing request (CSR) for the server
keytool -certreq -alias server -dname "CN=Server,OU=Operations,O=Organization,L=ACME" -keystore server.p12 -storepass secret -file server.csr
# Sign the CSR with the CA certificate
keytool -gencert -alias ca -keystore ca.p12 -storepass secret -infile server.csr -outfile server.cer
# Import the CA certificate to the server keystore
keytool -importcert -alias ca -keystore server.p12 -noprompt -storepass secret -file ca.cer
# Import the signed server certificate to the server keystore
keytool -importcert -alias server -keystore server.p12 -noprompt -storepass secret -file server.cer
# Create a client truststore that contains the CA public key
keytool -importcert -alias ca -keystore client_truststore.p12 -storepass secret -file ca.cer
# Optionally verify that the server keystore contains the CA and server certificates
keytool -list -v -keystore server.p12 -storepass secret
Place the server.p12
file in the server configuration directory.
To configure the client in order to connect to the server using the Hot Rod protocol, the client needs a trust store containing the public key of the server(s) or of the CA that has signed the key of the server you are going to connect to, unless the key was signed by a Certification Authority (CA) trusted by the JRE. |
The following code snippet shows how to use the client_truststore.p12
we generated above in your clients so that they will trust the server’s certificate:
ConfigurationBuilder clientBuilder = new ConfigurationBuilder();
clientBuilder
.addServer()
.host("127.0.0.1")
.port(11222)
.security()
.ssl()
.trustStoreFileName("client_truststore.p12")
.trustStorePassword("secret".toCharArray());
remoteCacheManager = new RemoteCacheManager(clientBuilder.build());
You can require that clients also present a certificate by adding require-ssl-client-auth="true"
to the endpoints
element:
<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1"
socket-binding="default" security-realm="default" require-ssl-client-auth="true">
<hotrod-connector name="hotrod"/>
<rest-connector name="rest" />
</endpoints>
It is also possible to generate development certificates on server startup. In order to do this, just specify generate-self-signed-certificate-host
in the keystore
element as shown below:
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<server-identities>
<ssl>
<keystore path="server.p12" relative-to="infinispan.server.config.path" keystore-password="secret" alias="server" generate-self-signed-certificate-host="localhost"/>
</ssl>
</server-identities>
</security-realm>
</security-realms>
</security>
There are three basic principles that you should remember when using automatically generated keystores:
|
The SSL identity also defines the way in which the SSL engine is configured, in terms of available protocols and ciphers. The default configuration should work for most scenarios, but it can be fine-tuned if necessary:
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<server-identities>
<ssl>
<keystore path="server.p12" relative-to="infinispan.server.config.path" keystore-password="secret" alias="server"/>
<engine enabled-protocols="TLSv1.2 TLSv1.1" enabled-ciphersuites="SSL_RSA_WITH_AES_128_GCM_SHA256 SSL_RSA_WITH_AES_128_CBC_SHA256"/>
</ssl>
</server-identities>
</security-realm>
</security-realms>
</security>
Make sure you have the correct ciphers for the protocol features you need (e.g. HTTP/2 ALPN)
Kerberos identities
A Kerberos identity is represented by a keytab file which contains pairs of Kerberos principals and encrypted keys (derived from Kerberos password).
While a keytab can contain both user and service account principals, in the context of a Infinispan server we only use the latter.
This allows the server to identify itself to clients as well as acting as an intermediary authentication service between the clients and the Kerberos server.
Since a Kerberos Service Account includes the name of the service in the principal, you usually need to supply one identity per service, i.e. one for HTTP and one for Hot Rod.
For example, you may have a datagrid
server in the INFINISPAN.ORG
domain, you’d need both a hotrod/datagrid@INFINISPAN.ORG
service principal for Hot Rod (note the lowercase service name) and an HTTP/datagrid@INFINISPAN.ORG
service principal for HTTP (note the uppercase service name).
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<server-identities>
<kerberos keytab-path="hotrod.keytab" principal="hotrod/datagrid@INFINISPAN.ORG" required="true"/>
<kerberos keytab-path="http.keytab" principal="HTTP/localhost@INFINISPAN.ORG" required="true"/>
</server-identities>
</security-realm>
</security-realms>
</security>
After creating the service principals in your Kerberos server, you will need to create the relevant keytab files:
- Linux
$ ktutil
ktutil: addent -password -p datagrid@INFINISPAN.ORG -k 1 -e aes256-cts
Password for datagrid@INFINISPAN.ORG: [enter your password]
ktutil: wkt http.keytab
ktutil: quit
- Microsoft Windows
$ ktpass -princ HTTP/datagrid@INFINISPAN.ORG -pass * -mapuser INFINISPAN\USER_NAME
$ ktab -k http.keytab -a HTTP/datagrid@INFINISPAN.ORG
Once you have obtained the keytab files, place them in the server’s configuration directory.
Property realm
Property realms are the simplest type of realm that you can configure in Infinispan:
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<properties-realm groups-attribute="Roles">
<user-properties path="users.properties" relative-to="infinispan.server.config.path" plain-text="true"/>
<group-properties path="groups.properties" relative-to="infinispan.server.config.path"/>
</properties-realm>
</security-realm>
</security-realms>
</security>
They use two property files:
-
a users property file which maps user names to passwords. The latter can be either in plain text or hashed (recommended).
user1=a_password user2=another_password
-
a groups property file which maps users to group names.
group1=user1,user2 group2=user2
Infinispan Server comes with a
user-tool.sh
script (user-tool.bat
for Windows) to ease the process of adding new user/role mappings to the above property files. An example invocation for adding a user with an initial set of groups:- Linux
$ bin/user-tools.sh -a -u myuser -p "qwer1234!" -g supervisor,reader,writer
- Microsoft Windows
$ bin\user-tools.bat -a -u myuser -p "qwer1234!" -g supervisor,reader,writer
Property realms support the following authentication mechanisms:
-
SASL (Hot Rod):
PLAIN
,DIGEST-*
,SCRAM-*
-
HTTP (REST):
Basic
,Digest
LDAP realm
LDAP realms connect to LDAP servers, such as OpenLDAP, Red Hat Directory Server, Apache Directory Server or Microsoft Active Directory, to authenticate users and obtain membership information. Since the layout of entries in an LDAP server varies depending on the type and deployment, the LDAP realm configuration is quite articulate. The following is an example
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<ldap-realm name="ldap" url="ldap://my-ldap-server:10389"
principal="uid=admin,ou=People,dc=infinispan,dc=org" credential="strongPassword"
direct-verification="true">
<identity-mapping rdn-identifier="uid" search-dn="ou=People,dc=infinispan,dc=org">
<attribute-mapping>
<attribute from="cn" to="Roles" filter="(&(objectClass=groupOfNames)(member={1}))"
filter-dn="ou=Roles,dc=infinispan,dc=org"/>
</attribute-mapping>
</identity-mapping>
</ldap-realm>
</security-realm>
</security-realms>
</security>
The ldap-realm
element specifies the LDAP connection url as well as a principal and credential to use.
The principal must be able to perform LDAP queries and access specific attributes, so it should have the necessary privileges.
You can also set additional properties which tune the connection, e.g. timeouts and pooling:
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<ldap-realm name="ldap" url="ldap://my-ldap-server:10389"
principal="uid=admin,ou=People,dc=infinispan,dc=org" credential="strongPassword"
connection-timeout="3000" read-timeout="30000" connection-pooling="true" referral-mode="ignore" page-size="30">
<name-rewriter>
<regex-principal-transformer name="domain-remover" pattern="(.*)@INFINISPAN\.ORG" replacement="$1"/>
</name-rewriter>
<identity-mapping rdn-identifier="uid" search-dn="ou=People,dc=infinispan,dc=org">
<attribute-mapping>
<attribute from="cn" to="Roles" filter="(&(objectClass=groupOfNames)(member={1}))"
filter-dn="ou=Roles,dc=infinispan,dc=org"/>
</attribute-mapping>
<user-password-mapper from="userPassword"/>
</identity-mapping>
</ldap-realm>
</security-realm>
</security-realms>
</security>
It is then necessary to provide a mapping from LDAP entries to identities using the identity-mapping
element.
The first part is to find the user’s entry based on a provided identifier, usually a username.
The rdn-identifier
specifies the LDAP attribute which contains the username (e.g. uid
or sAMAccountName
).
It is also useful to limit the search only to the LDAP subtree which contains the user entries, otherwise it will search the entire tree.
This is achieved by passing a starting context in the search-dn
attribute.
Then we need to decide how to verify a user’s credentials. This can be done in two ways:
-
by adding the
direct-verification="true"
attribute to theldap-realm
element, the realm will try to connect to the LDAP server using the supplied credentials. This is the recommended configuration. -
by adding a
user-password-mapper
element to theidentity-mapping
element specifying which attribute contains the password (e.g.userPassword
)
Some authentication mechanisms (most notably the Kerberos-based ones: GSSAPI
, GS2-KRB5
and Negotiate
) supply a username which needs to be cleaned up before it can be used to search LDAP.
In these situation you should add a name-rewriter
to the ldap-realm
which extracts the username from the principal. For example, the following rewriter uses a regular expression and group capturing to remove the domain part:
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<ldap-realm name="ldap" url="ldap://${org.infinispan.test.host.address}:10389" principal="uid=admin,ou=People,dc=infinispan,dc=org" credential="strongPassword">
<name-rewriter>
<regex-principal-transformer name="domain-remover" pattern="(.*)@INFINISPAN\.ORG" replacement="$1"/>
</name-rewriter>
<identity-mapping rdn-identifier="uid" search-dn="ou=People,dc=infinispan,dc=org">
<attribute-mapping>
<attribute from="cn" to="Roles" filter="(&(objectClass=groupOfNames)(member={1}))" filter-dn="ou=Roles,dc=infinispan,dc=org" />
</attribute-mapping>
<user-password-mapper from="userPassword" />
</identity-mapping>
</ldap-realm>
</security-realm>
</security-realms>
</security>
Next, we will want to retrieve all the groups a user is a member of, since this information may not be stored as part of a user entry. There are usually two ways in which this is implemented:
-
Membership information is stored under group entries (which usually have class
groupOfNames
) in themember
attribute (this is the typical layout in OpenLDAP). In this case you will need to use an attribute filter as follows:<attribute filter="(&(objectClass=groupOfNames)(member={1}))" filter-dn="ou=Roles,dc=infinispan,dc=org" from="cn" to="Roles" />
The above will search for entries matching the supplied
filter
(which matches groups with amember
attribute equal to the user’s DN) under thefilter-dn
. It will then extract the group entry’s CN (as specified byfrom
) and add it to the user’sRoles
. -
Membership information is stored in the user entry in the
memberOf
attribute (typical in Microsoft Active Directory). In this case use an attribute reference as follows:<attribute-reference reference="memberOf" from="cn" to="Roles" />
The above will get all
memberOf
attributes from the user and extract the CN (as specified byfrom
) and add them to the user’sRoles
.
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<ldap-realm name="ldap" url="ldap://my-ldap-server:10389"
principal="CN=LdapAdmin,CN=Users,DC=INFINISPAN,DC=ORG" credential="strongPassword"
direct-verification="true">
<identity-mapping rdn-identifier="sAMAccountName" search-dn="cn=Users,dc=INFINISPAN,dc=ORG">
<attribute-mapping>
<attribute-reference reference="memberOf" from="cn" to="Roles"/>
</attribute-mapping>
</identity-mapping>
</ldap-realm>
</security-realm>
</security-realms>
</security>
LDAP Realms support the following authentication mechanisms directly:
-
SASL (Hot Rod):
PLAIN
,DIGEST-*
,SCRAM-*
-
HTTP (REST):
Basic
,Digest
LDAP Realms can also be used to fill-in authorization information for users obtained through realms which don’t provide this directly (e.g. Trust, Kerberos)
Trust Store realm
A Trust Store realm is backed by a keystore containing the public certificates of all the clients you want to allow to connect. It is used with client-certificate authentication mechanisms (EXTERNAL for Hot Rod and CLIENT_CERT for HTTP) so it requires that the server’s realm also includes an SSL server identity as well as requiring client certificate authentication on the endpoint.
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<server-identities>
<ssl>
<keystore path="server.p12" relative-to="infinispan.server.config.path" keystore-password="secret"
alias="server"/>
</ssl>
</server-identities>
<truststore-realm path="trust.p12" relative-to="infinispan.server.config.path" keystore-password="secret"/>
</security-realm>
</security-realms>
</security>
You should use keytool
to import all of the client certificates into the trust store.
Token realm
The Token realm relies on external validation services to validate tokens. It requires a provider compatible with RFC-7662 (OAuth2 Token Introspection), such as KeyCloak.
<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1">
<security-realms>
<security-realm name="default">
<token-realm name="token" auth-server-url="https://oauth-server/auth/">
<oauth2-introspection
introspection-url="https://oauth-server/auth/realms/infinispan/protocol/openid-connect/token/introspect"
client-id="infinispan-server" client-secret="1fdca4ec-c416-47e0-867a-3d471af7050f"/>
</token-realm>
</security-realm>
</security-realms>
</security>
The auth-server-url
attribute on the realm specifies the URL of the authentication server.
The oauth2-introspection
element specifies the introspection-url
as well as a client-id
and client-secret
which will be used for interpreting the tokens.
Token realms support the following authentication mechanisms:
-
SASL (Hot Rod):
OAUTHBEARER
-
HTTP (REST):
Bearer
5.2. Hot Rod Authentication
The Hot Rod protocol supports authentication by leveraging the SASL mechanisms. The supported SASL mechanisms (usually shortened as mechs) are:
-
PLAIN
- This is the most insecure mech, since credentials are sent over the wire in plain-text format, however it is the simplest to get to work. In combination with encryption (i.e. TLS) it can be used safely. It is equivalent to theBASIC
HTTP mechanism. -
DIGEST-*
- This family of mechs hashes the credentials before sending them over the wire, so it is more secure thanPLAIN
. The supported implementations, which differ in the hashing algorithm, are:DIGEST-MD5
,DIGEST-SHA
,DIGEST-SHA-256
,DIGEST-SHA-384
andDIGEST-SHA-512
ordered by increasing strength. These mechs are comparable to theDIGEST
HTTP mechanism. -
SCRAM-*
- This family of mechs is similar toDIGEST-*
, but increases security by using an additional salt for increased security. The supported implementations, which differ in the hashing algorithm, are:SCRAM-SHA
,SCRAM-SHA-256
,SCRAM-SHA-384
andSCRAM-SHA-512
ordered by increasing strength. -
GSSAPI
- This mech uses Kerberos tickets, and therefore requires the presence of a properly configured Kerberos Domain Controller (such as Microsoft Active Directory), and a correspondingkerberos
server identity in the realm. It is also usually used in combination with anldap-realm
for retrieving user membership information. These mechs are equivalent with theSPNEGO
HTTP scheme. -
GS2-KRB5
- An improvement overGSSAPI
, it has the same requirements and is configured in the same way. -
EXTERNAL
- This mech obtains credentials from the underlying transport (i.e. from a X.509 client certificate) and therefore requires encryption using client-certificates to be enabled. It is equivalent to theCLIENT_CERT
HTTP mechanism. -
OAUTHBEARER
- This mech enables the use of OAuth tokens as a way to. It requires a token realm. It is equivalent to theBEARER_TOKEN
HTTP mechanism.
The following configuration enables authentication against the default
realm, using various SCRAM and DIGEST SASL mechanisms and only enables the auth QoP (see SASL Quality of Protection):
<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1"
socket-binding="default" security-realm="default">
<hotrod-connector name="hotrod">
<authentication>
<sasl mechanisms="SCRAM-SHA-512 SCRAM-SHA-384 SCRAM-SHA-256 SCRAM-SHA-1 DIGEST-SHA-512 DIGEST-SHA-384 DIGEST-SHA-256 DIGEST-SHA DIGEST-MD5 PLAIN"
server-name="infinispan" qop="auth"/>
</authentication>
</hotrod-connector>
</endpoints>
Notice the server-name attribute: it is the name that the server declares to incoming clients and therefore the client configuration must match.
It is particularly important when using GSSAPI
or GS2-KRB5
mechs as it is equivalent to the Kerberos service name.
The Kerberos mechanisms also need to indicate the identity that will be used:
<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1"
socket-binding="default" security-realm="default">
<hotrod-connector name="hotrod">
<authentication>
<sasl mechanisms="GSSAPI GS2-KRB5" server-name="datagrid" server-principal="hotrod/datagrid@INFINISPAN.ORG"/>
</authentication>
</hotrod-connector>
</endpoints>
5.2.1. SASL Quality of Protection
While the main purpose of SASL is to provide authentication, some mechanisms also support integrity and privacy protection, also known as Quality of Protection (or qop). During authentication negotiation, ciphers are exchanged between client and server, and they can be used to add checksums and encryption to all subsequent traffic. You can tune the required level of qop as follows:
QOP | Description |
---|---|
auth |
Authentication only |
auth-int |
Authentication with integrity protection |
auth-conf |
Authentication with integrity and privacy protection |
5.2.2. SASL Policies
You can further refine the way a mechanism is chosen by tuning the SASL policies. This will effectively include / exclude mechanisms based on whether they match the desired policies.
Policy | Description |
---|---|
forward-secrecy |
Specifies that the selected SASL mechanism must support forward secrecy between sessions. This means that breaking into one session will not automatically provide information for breaking into future sessions. |
pass-credentials |
Specifies that the selected SASL mechanism must require client credentials. |
no-plain-text |
Specifies that the selected SASL mechanism must not be susceptible to simple plain passive attacks. |
no-active |
Specifies that the selected SASL mechanism must not be susceptible to active (non-dictionary) attacks. The mechanism might require mutual authentication as a way to prevent active attacks. |
no-dictionary |
Specifies that the selected SASL mechanism must not be susceptible to passive dictionary attacks. |
no-anonymous |
Specifies that the selected SASL mechanism must not accept anonymous logins. |
Each policy’s value is either "true" or "false". If a policy is absent, then the chosen mechanism need not have that characteristic (equivalent to setting the policy to "false"). One notable exception is the no-anonymous policy which, if absent, defaults to true, thus preventing anonymous connections.
It is possible to have mixed anonymous and authenticated connections to the endpoint, delegating actual access logic to cache authorization configuration. To do so, set the no-anonymous policy to false and turn on cache authorization. |
The following configuration selects all available mechanisms, but effectively only enables GSSAPI, since it is the only one that respects all chosen policies:
<hotrod-connector socket-binding="hotrod" cache-container="default">
<authentication security-realm="ApplicationRealm">
<sasl server-name="myhotrodserver" mechanisms="PLAIN DIGEST-MD5 GSSAPI EXTERNAL" qop="auth">
<policy>
<no-active value="true" />
<no-anonymous value="true" />
<no-plain-text value="true" />
</policy<>
</sasl>
</authentication>
</hotrod-connector>
5.3. REST Authentication
The REST connector supports authentication by leveraging the HTTP authentication schemes. The supported mechanism names are:
-
BASIC
- This mechanism corresponds to theBasic
HTTP scheme. This is the most insecure scheme, since credentials are sent over the wire in plain-text format, however it is the simplest to get to work. In combination with encryption (i.e. TLS) it can be used safely. It is comparable to thePLAIN
SASL mechanism. -
DIGEST
- This mechanism corresponds to theDigest
HTTP scheme. It is more secure thanBASIC
because it uses hashing and a nonce to protect against replay attacks. The server supports multiple hashing algorithms:SHA-512
,SHA-256
andMD5
. It is comparable to theDIGEST-*
family of SASL mechanisms. -
SPNEGO
- This mechanism corresponds to theNegotiate
HTTP scheme and uses Kerberos tickets, and therefore requires the presence of a properly configured Kerberos Domain Controller (such as Microsoft Active Directory), and a correspondingkerberos
server identity in the realm. It is also usually used in combination with anldap-realm
for retrieving user membership information. It is comparable to theGSSAPI
andGS2-KRB5
SASL mechanisms. -
BEARER_TOKEN
- This mechanism corresponds to theBearer
HTTP scheme and enables the use of OAuth tokens as a way to authenticate users. It requires a token realm. It is equivalent to theOAUTHBEARER
SASL mechanism. -
CLIENT_CERT
- This is a pseudo-mechanism which relies on the client certificate to provide authentication information. It is equivalent to theEXTERNAL
SASL mechanism.
The following configuration enables authentication against the default
realm, using both BASIC
and DIGEST
mechanisms:
<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1"
socket-binding="default" security-realm="default">
<rest-connector name="rest">
<authentication mechanisms="DIGEST BASIC"/>
</rest-connector>
</endpoints>
The SPNEGO
mechanism also needs to indicate the Kerberos identity that will be used:
<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:server:10.1 https://infinispan.org/schemas/infinispan-server-10.1.xsd"
xmlns="urn:infinispan:server:10.1"
socket-binding="default" security-realm="default">
<rest-connector name="rest">
<authentication mechanisms="SPNEGO" server-principal="HTTP/localhost@INFINISPAN.ORG"/>
</rest-connector>
</endpoints>
6. Remotely Executing Server-Side Tasks
Define and add tasks to Infinispan servers that you can invoke from the Infinispan command line interface, REST API, or from Hot Rod clients.
You can implement tasks as custom Java classes or define scripts in languages such as JavaScript.
6.1. Creating Server Tasks
Create custom task implementations and add them to Infinispan servers.
6.1.1. Server Tasks
Infinispan server tasks are classes that extend the
org.infinispan.tasks.ServerTask
interface and generally include the following
method calls:
setTaskContext()
-
Allows access to execution context information including task parameters, cache references on which tasks are executed, and so on. In most cases, implementations store this information locally and use it when tasks are actually executed.
getName()
-
Returns unique names for tasks. Clients invoke tasks with these names.
getExecutionMode()
-
Returns the execution mode for tasks.
-
TaskExecutionMode.ONE_NODE
only the node that handles the request executes the script. Although scripts can still invoke clustered operations. -
TaskExecutionMode.ALL_NODES
Infinispan uses clustered executors to run scripts across nodes. For example, server tasks that invoke stream processing need to be executed on a single node because stream processing is distributed to all nodes.
-
call()
-
Computes a result. This method is defined in the
java.util.concurrent.Callable
interface and is invoked with server tasks.
Server task implementations must adhere to service loader pattern requirements. For example, implementations must have a zero-argument constructors. |
The following HelloTask
class implementation provides an example task that
has one parameter:
package example;
import org.infinispan.tasks.ServerTask;
import org.infinispan.tasks.TaskContext;
public class HelloTask implements ServerTask<String> {
private TaskContext ctx;
@Override
public void setTaskContext(TaskContext ctx) {
this.ctx = ctx;
}
@Override
public String call() throws Exception {
String name = (String) ctx.getParameters().get().get("name");
return "Hello " + name;
}
@Override
public String getName() {
return "hello-task";
}
}
6.1.2. Deploying Server Tasks to Infinispan Servers
Add your custom server task classes to Infinispan servers.
Stop any running Infinispan servers. Infinispan does not support runtime deployment of custom classes.
-
Package your server task implementation in a JAR file.
-
Add a
META-INF/services/org.infinispan.tasks.ServerTask
file that contains the fully qualified names of server tasks, for example:example.HelloTask
-
Copy the JAR file to the
$ISPN_HOME/server/lib
directory of your Infinispan server. -
Add your classes to the deserialization whitelist in your Infinispan configuration. Alternatively set the whitelist using system properties.
6.2. Creating Server Scripts
Create custom scripts and add them to Infinispan servers.
6.2.1. Server Scripts
Infinispan server scripting is based on the javax.script
API and is
compatible with any JVM-based ScriptEngine implementation. Nashorn is the
default JDK ScriptEngine and provides JavaScript capabilities.
The following script provides a simple example that runs on a single Infinispan server, has one parameter, and uses JavaScript:
// mode=local,language=javascript,parameters=[greetee]
"Hello " + greetee
When you run the preceding script, you pass a value for the greetee
parameter
and Infinispan returns "Hello ${value}"
.
Script Metadata
Metadata provides additional information about scripts that Infinispan servers use when running scripts.
Script metadata are property=value
pairs that you add to comments in the
first lines of scripts, such as the following example:
// name=test, language=javascript
// mode=local, parameters=[a,b,c]
-
Use comment styles that match the scripting language (
//
,;;
,#
). -
Separate
property=value
pairs with commas. -
Separate values with single (') or double (") quote characters.
Property | Description |
---|---|
|
Defines the exection mode and has the following values:
|
|
Specifies the ScriptEngine that executes the script. |
|
Specifies filename extensions as an alternative method to set the ScriptEngine. |
|
Specifies roles that users must have to execute scripts. |
|
Specifies an array of valid parameter names for this script. Invocations which specify parameters not included in this list cause exceptions. |
|
Optionally sets the MediaType (MIME type) for storing data as well as parameter and return values. This property is useful for remote clients that support particular data formats only. Currently you can set only |
Script Bindings
Infinispan exposes internal objects as bindings for script execution.
Binding | Description |
---|---|
|
Specifies the cache against which the script is run. |
|
Specifies the marshaller to use for serializing data to the cache. |
|
Specifies the |
|
Specifies the instance of the script manager that runs the script. You can use this binding to run other scripts from a script. |
Script Parameters
Infinispan lets you pass named parameters as bindings for running scripts.
Parameters are name,value
pairs, where name
is a string and value
is any
value that the marshaller can interpret.
The following example script has two parameters, multiplicand
and
multiplier
. The script takes the value of multiplicand
and multiplies it
with the value of multiplier
.
// mode=local,language=javascript
multiplicand * multiplier
When you run the preceding script, Infinispan responds with the result of the expression evaluation.
6.2.2. Adding Scripts to Infinispan Servers
Use the command line interface to add scripts to Infinispan servers.
Infinispan servers store scripts in the ___script_cache
cache. If you enable
cache authorization, users must have the ___script_manager
role to access
___script_cache
.
-
Define scripts as required.
For example, create a file named
multiplication.js
that runs on a single Infinispan server, has two parameters, and uses JavaScript to multiply a given value:// mode=local,language=javascript multiplicand * multiplier
-
Open a CLI connection to Infinispan and use the
task
command to upload your scripts as in the following example:[//containers/default]> task upload --file=multiplication.js multiplication
-
Verify that your scripts are available.
[//containers/default]> ls tasks multiplication
6.2.3. Programmatically Creating Scripts
Add scripts with the Hot Rod RemoteCache
interface as in the following
example:
RemoteCache<String, String> scriptCache = cacheManager.getCache("___script_cache");
scriptCache.put("multiplication.js",
"// mode=local,language=javascript\n" +
"multiplicand * multiplier\n");
6.3. Running Server-Side Tasks and Scripts
Execute tasks and custom scripts on Infinispan servers.
6.3.1. Running Tasks and Scripts
Use the command line interface to run tasks and scripts on Infinispan servers.
-
Open a CLI connection to Infinispan.
Use the task
command to run tasks and scripts on Infinispan servers, as in the following examples:
-
Execute a script named
multipler.js
and specify two parameters:[//containers/default]> task exec multipler.js -Pmultiplicand=10 -Pmultiplier=20 200.0
-
Execute a task named
@@cache@names
to retrieve a list of all available caches://containers/default]> task exec @@cache@names ["___protobuf_metadata","mycache","___script_cache"]
6.3.2. Programmatically Running Scripts
Call the execute()
method to run scripts with the Hot Rod RemoteCache
interface, as in the following example:
RemoteCache<String, Integer> cache = cacheManager.getCache();
// Create parameters for script execution.
Map<String, Object> params = new HashMap<>();
params.put("multiplicand", 10);
params.put("multiplier", 20);
// Run the script with the parameters.
Object result = cache.execute("multiplication.js", params);
6.3.3. Programmatically Running Tasks
Call the execute()
method to run tasks with the Hot Rod RemoteCache
interface, as in the following example:
// Add configuration for a locally running server.
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.addServer().host("127.0.0.1").port(11222);
// Connect to the server.
RemoteCacheManager cacheManager = new RemoteCacheManager(builder.build());
// Retrieve the remote cache.
RemoteCache<String, String> cache = cacheManager.getCache();
// Create task parameters.
Map<String, String> parameters = new HashMap<>();
parameters.put("name", "developer");
// Run the server task.
String greet = cache.execute("hello-task", parameters);
System.out.println(greet);
7. Performing Rolling Upgrades
Upgrade Infinispan without downtime or data loss. You can perform rolling upgrades for Infinispan servers to start using a more recent version of Infinispan.
This section explains how to upgrade Infinispan servers, see the appropriate documentation for your Hot Rod client for upgrade procedures. |
7.1. Rolling Upgrades
From a high-level, you do the following to perform rolling upgrades:
-
Set up a target cluster. The target cluster is the Infinispan version to which you want to migrate data. The source cluster is the Infinispan deployment that is currently in use. After the target cluster is running, you configure all clients to point to it instead of the source cluster.
-
Synchronize data from the source cluster to the target cluster.
7.2. Setting Up Target Clusters
-
Start the target cluster with unique network properties or a different JGroups cluster name to keep it separate from the source cluster.
-
Configure a
RemoteCacheStore
on the target cluster for each cache you want to migrate from the source cluster.RemoteCacheStore
settings-
-
remote-server
must point to the source cluster via theoutbound-socket-binding
property. -
remoteCacheName
must match the cache name on the source cluster. -
hotrod-wrapping
must betrue
(enabled). -
shared
must betrue
(enabled). -
purge
must befalse
(disabled). -
passivation
must befalse
(disabled). -
protocol-version
matches the Hot Rod protocol version of the source cluster.ExampleRemoteCacheStore
Configuration<distributed-cache> <remote-store cache="MyCache" socket-timeout="60000" tcp-no-delay="true" protocol-version="2.5" shared="true" hotrod-wrapping="true" purge="false" passivation="false"> <remote-server outbound-socket-binding="remote-store-hotrod-server"/> </remote-store> </distributed-cache> ... <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}"> ... <outbound-socket-binding name="remote-store-hotrod-server"> <remote-destination host="198.51.100.0" port="11222"/> </outbound-socket-binding> ... </socket-binding-group>
-
-
Configure the target cluster to handle all client requests instead of the source cluster:
-
Configure all clients to point to the target cluster instead of the source cluster.
-
Restart each client node.
The target cluster lazily loads data from the source cluster on demand via
RemoteCacheStore
.
-
7.3. Synchronizing Data from Source Clusters
-
Call the
synchronizeData()
method in theTargetMigrator
interface. Do one of the following on the target cluster for each cache that you want to migrate:- JMX
-
Invoke the
synchronizeData
operation and specify thehotrod
parameter on theRollingUpgradeManager
MBean. - CLI
-
$ bin/cli.sh --connect controller=127.0.0.1:9990 -c "/subsystem=datagrid-infinispan/cache-container=clustered/distributed-cache=MyCache:synchronize-data(migrator-name=hotrod)"
Data migrates to all nodes in the target cluster in parallel, with each node receiving a subset of the data.
Use the following parameters to tune the operation:
-
read-batch
configures the number of entries to read from the source cluster at a time. The default value is10000
. -
write-threads
configures the number of threads used to write data. The default value is the number of processors available.For example:
synchronize-data(migrator-name=hotrod, read-batch=100000, write-threads=3)
-
-
Disable the
RemoteCacheStore
on the target cluster. Do one of the following:- JMX
-
Invoke the
disconnectSource
operation and specify thehotrod
parameter on theRollingUpgradeManager
MBean. - CLI
-
$ bin/cli.sh --connect controller=127.0.0.1:9990 -c "/subsystem=datagrid-infinispan/cache-container=clustered/distributed-cache=MyCache:disconnect-source(migrator-name=hotrod)"
-
Decommission the source cluster.