
:guide-id: multi-cluster-deploy-infinispan-kubernetes-crossdc
:guide-parent: multi-cluster
:guide-title: Deploying {jdgserver_name} for HA with the {jdgserver_name} Operator
:guide-summary: Deploy {jdgserver_name} for high availability in multi availability zones on {kubernetes}.
:guide-priority: 999
:guide-tile-visible: false
:version: 26.4.0

include::../../attributes.adoc[]

[[multi-cluster-deploy-infinispan-kubernetes-crossdc]]
= Deploying {jdgserver_name} for HA with the {jdgserver_name} Operator

ifeval::["" == "true"]
WARNING: This {section} is describing a feature which is currently in preview.
ifeval::["" == ""]
Please provide your feedback while we’re continuing to work on this.
endif::[]
ifeval::["" != ""]
Please provide your feedback by link:[joining this discussion] while we’re continuing to work on this.
endif::[]
endif::[]


include::../partials/infinispan/infinispan-attributes.adoc[]

This {section} describes the procedures required to deploy {jdgserver_name} in a multi-cluster environment (cross-site).
For simplicity, this topic uses the minimum configuration possible that allows {project_name} to be used with an external {jdgserver_name}.

This {section} assumes two {ocp} clusters named `{site-a}` and `{site-b}`.

This is a building block following the concepts described in the link:{links_high-availability_multi-cluster-concepts_url}[{links_high-availability_multi-cluster-concepts_name}] {section}.
See the link:{links_high-availability_multi-cluster-introduction_url}[{links_high-availability_multi-cluster-introduction_name}] {section} for an overview.


[IMPORTANT]
====
ifeval::[{project_product} != true]
Only Infinispan version 15.0.19.Final or more recent patch releases are supported for external {jdgserver_name} deployments.
endif::[]
ifeval::[{project_product} == true]
Only {jdgserver_name} version {jdgserver_min_version} or more recent patch releases are supported for external {jdgserver_name} deployments.
endif::[]
====

== Architecture

This setup deploys two synchronously replicating {jdgserver_name} clusters in two sites with a low-latency network connection.
An example of this scenario could be two availability zones in one AWS region.

{project_name}, loadbalancer and database have been removed from the following diagram for simplicity.

image::high-availability/infinispan-crossdc-az.dio.svg[]

== Prerequisites

include::../partials/infinispan/infinispan-prerequisites.adoc[]

== Procedure

include::../partials/infinispan/infinispan-install-operator.adoc[]
include::../partials/infinispan/infinispan-credentials.adoc[]
+
These commands must be executed on both {ocp} clusters.

. Create a service account.
+
A service account is required to establish a connection between clusters.
The {ispn-operator} uses it to inspect the network configuration from the remote site and to configure the local {jdgserver_name} cluster accordingly.
+
For more details, see the {infinispan-operator-docs}#managed-cross-site-connections_cross-site[Managing Cross-Site Connections] documentation.
+
.. Create a `service-account-token` secret type as follows.
The same YAML file can be used in both {ocp} clusters.
+
.xsite-sa-secret-token.yaml
[source,yaml,subs="+attributes"]
----
apiVersion: v1
kind: Secret
metadata:
  name: ispn-xsite-sa-token #<1>
  annotations:
    kubernetes.io/service-account.name: "{sa}" #<2>
type: kubernetes.io/service-account-token
----
<1> The secret name.
<2> The service account name.
.. Create the service account and generate an access token in both {ocp} clusters.
+
.Create the service account in `{site-a}`
[source,bash,subs="+attributes"]
----
kubectl create sa -n {ns} {sa}
oc policy add-role-to-user view -n {ns} -z {sa}
kubectl create -f xsite-sa-secret-token.yaml
kubectl get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > {site-a}-token.txt
----
+
.Create the service account in `{site-b}`
[source,bash,subs="+attributes"]
----
kubectl create sa -n {ns} {sa}
oc policy add-role-to-user view -n {ns} -z {sa}
kubectl create -f xsite-sa-secret-token.yaml
kubectl get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > {site-b}-token.txt
----
+
.. The next step is to deploy the token from `{site-a}` into `{site-b}` and the reverse:
+
.Deploy `{site-b}` token into `{site-a}`
[source,bash,subs="+attributes"]
----
kubectl create secret generic -n {ns} {sa-secret} \
  --from-literal=token="$(cat {site-b}-token.txt)"
----
+
.Deploy `{site-a}` token into `{site-b}`
[source,bash,subs="+attributes"]
----
kubectl create secret generic -n {ns} {sa-secret} \
  --from-literal=token="$(cat {site-a}-token.txt)"
----

. Create TLS secrets
+
In this {section}, {jdgserver_name} uses an {ocp} Route for the cross-site communication.
It uses the SNI extension of TLS to direct the traffic to the correct Pods.
To achieve that, JGroups use TLS sockets, which require a Keystore and Truststore with the correct certificates.
+
For more information, see the {infinispan-operator-docs}#securing-cross-site-connections_cross-site[Securing Cross Site Connections] documentation or this https://developers.redhat.com/learn/openshift/cross-site-and-cross-applications-red-hat-openshift-and-red-hat-data-grid[Red Hat Developer Guide].
+
Upload the Keystore and the Truststore in an {ocp} Secret.
The secret contains the file content, the password to access it, and the type of the store.
Instructions for creating the certificates and the stores are beyond the scope of this {section}.
+
To upload the Keystore as a Secret, use the following command:
+
.Deploy a Keystore
[source,bash,subs="+attributes"]
----
kubectl -n {ns} create secret generic {ks-secret} \
  --from-file=keystore.p12="./certs/keystore.p12" \ # <1>
  --from-literal=password=secret \ #<2>
  --from-literal=type=pkcs12 #<3>
----
<1> The filename and the path to the Keystore.
<2> The password to access the Keystore.
<3> The Keystore type.
+
To upload the Truststore as a Secret, use the following command:
+
.Deploy a Truststore
[source,bash,subs="+attributes"]
----
kubectl -n {ns} create secret generic {ts-secret} \
        --from-file=truststore.p12="./certs/truststore.p12" \  # <1>
        --from-literal=password=caSecret \  # <2>
        --from-literal=type=pkcs12  # <3>
----
<1> The filename and the path to the Truststore.
<2> The password to access the Truststore.
<3> The Truststore type.
+
NOTE: Keystore and Truststore must be uploaded in both {ocp} clusters.

. Create a Cluster for {jdgserver_name} with Cross-Site enabled
+
The {infinispan-operator-docs}#setting-up-xsite[Setting Up Cross-Site] documentation provides all the information on how to create and configure your {jdgserver_name} cluster with cross-site enabled, including the previous steps.
+
A basic example is provided in this {section} using the credentials, tokens, and TLS Keystore/Truststore created by the commands from the previous steps.
+
--
.The `Infinispan` CR for `{site-a}`
[source,yaml]
----
include::../examples/generated/ispn-site-a.yaml[tag=infinispan-crossdc]
----
<1> The cluster name
<2> Allows the cluster to be monitored by Prometheus.
<3> If using a custom credential, configure here the secret name.
<4> The name of the local site, in this case `{site-a}`.
<5> Exposing the cross-site connection using {ocp} Route.
<6> The secret name where the Keystore exists as defined in the previous step.
<7> The alias of the certificate inside the Keystore.
<8> The secret key (filename) of the Keystore as defined in the previous step.
<9> The secret name where the Truststore exists as defined in the previous step.
<10> The Truststore key (filename) of the Keystore as defined in the previous step.
<11> The remote site's name, in this case  `{site-b}`. You may use this value in {project_name} option `cache-remote-backup-sites` for the automatic cache creation.
<12> The namespace of the {jdgserver_name} cluster from the remote site.
<13> The {ocp} API URL for the remote site.
<14> The secret with the access token to authenticate into the remote site.
--
+
For `{site-b}`, the `Infinispan` CR looks similar to the above.
Note the differences in point 4, 11 and 13.
+
.The `Infinispan` CR for `{site-b}`
[source,yaml]
----
include::../examples/generated/ispn-site-b.yaml[tag=infinispan-crossdc]
----

. Creating the caches for {project_name}.
+
{project_name} automatically creates the necessary caches on the first startup if they do not exist.
+
[IMPORTANT]
====
It requires {project_name} to be deployed in both clusters because the {project_name} does not start until all caches are present in both clusters.
====
+
To {infinispan-operator-docs}#creating-caches_creating-caches[`Cache` CR] is the recommend way to proceed in {kubernetes}.
To take effect, the `Cache` CR must be deployed before any {project_name} Pod starts.
+
The following example shows the `Cache` CR for `{site-a}`.
+
--
. In `{site-a}` create a `Cache` CR for each of the caches mentioned above with the following content.
+
.Cache `actionTokens`
[source,yaml]
----
include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-actionTokens]
----
+
.Cache `authenticationSessions`
[source,yaml]
----
include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-authenticationSessions]
----
+
.Cache `loginFailures`
[source,yaml]
----
include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-loginFailures]
----
+
.Cache `work`
[source,yaml]
----
include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-work]
----
<1> The transaction mode.
<2> The locking mode used by the transaction.
<3> The remote site name.
<4> The cross-site communication strategy, in this case, `SYNC`.
<5> The cross-site replication timeout.
<6> The cross-site replication failure policy.
--
+
The example above is the recommended configuration to achieve the best data consistency.
+
====
*Background information*

Deadlocks may occur in an active-active setup as entries are modified concurrently in both sites.

The `transaction.mode: NON_DURABLE_XA` ensures that the transaction is rolled back keeping the data consistent if this occurs.
The setting `backup.failurePolicy: FAIL` is required in this case.
It will throw an error that allows the transaction to be safely rolled back.
When this occurs, {project_name} will attempt a retry.

The `transaction.locking: PESSIMISTIC` is the only supported locking mode; `OPTIMISTIC` is not recommended due to its network costs.
The same settings also prevent that one site is updated while the other site is unreachable.

The `backup.strategy: SYNC` ensures the data is visible and stored in the other site when the {project_name} request is completed.

NOTE: The `locking.acquireTimeout` can be reduced to fail fast in a deadlock scenario.
The `backup.timeout` must always be higher than the `locking.acquireTimeout`.
====
+
For `{site-b}`, the `Cache` CR is similar, except for the `backups.<name>` outlined in point 3 of the above diagram.
+
.Example for `actionTokens` cache in `{site-b}`
[source,yaml]
----
include::../examples/generated/ispn-site-b.yaml[tag=infinispan-cache-actionTokens]
----
== Verifying the deployment

Confirm that the {jdgserver_name} cluster is formed, and the cross-site connection is established between the {ocp} clusters.

.Wait until the {jdgserver_name} cluster is formed
[source,bash,subs="+attributes"]
----
kubectl wait --for condition=WellFormed --timeout=300s infinispans.infinispan.org -n {ns} {cluster-name}
----

.Wait until the {jdgserver_name} cross-site connection is established
[source,bash,subs="+attributes"]
----
kubectl wait --for condition=CrossSiteViewFormed --timeout=300s infinispans.infinispan.org -n {ns} {cluster-name}
----

[#connecting-infinispan-to-keycloak]
== Connecting {jdgserver_name} with {project_name}

Now that the {jdgserver_name} server is running, here are the relevant {project_name} CR changes necessary to connect it to {project_name}. These changes will be required in the link:{links_high-availability_multi-cluster-deploy-keycloak-kubernetes_url}[{links_high-availability_multi-cluster-deploy-keycloak-kubernetes_name}] {section}.

. Create a Secret with the username and password to connect to the external {jdgserver_name} deployment:
+
[source,yaml]
----
include::../examples/generated/keycloak-ispn.yaml[tag=keycloak-ispn-secret]
----

. Extend the {project_name} Custom Resource with `additionalOptions` as shown below.
+
[NOTE]
====
All the memory, resource and database configurations are skipped from the CR below as they have been described in the link:{links_high-availability_multi-cluster-deploy-keycloak-kubernetes_url}[{links_high-availability_multi-cluster-deploy-keycloak-kubernetes_name}] {section} already.
Administrators should leave those configurations untouched.
====
+
[source,yaml]
----
include::../examples/generated/keycloak-ispn.yaml[tag=keycloak-ispn]
----
<1> The hostname of the remote {jdgserver_name} cluster.
<2> The port of the remote {jdgserver_name} cluster.
This is optional, and it defaults to `11222`.
<3> The Secret `name` and `key` with the {jdgserver_name} username credential.
<4> The Secret `name` and `key` with the {jdgserver_name} password credential.
<5> Optionally, the name of the remote site. The caches are only created if they do not already exist.
+
[IMPORTANT]
====
Using the option `cache-remote-backup-sites` only creates the caches in the local site.
You must deploy the `KeycloakCR` in the other cluster too, to create the caches there, otherwise {project_name} fails to start until they exist.
====

=== Architecture

This connects {project_name} to {jdgserver_name} using TCP connections secured by TLS 1.3.
It uses the {project_name}'s truststore to verify {jdgserver_name}'s server certificate.
As {project_name} is deployed using its Operator on OpenShift in the prerequisites listed below, the Operator already added the `service-ca.crt` to the truststore which is used to sign {jdgserver_name}'s server certificates.
In other environments, add the necessary certificates to {project_name}'s truststore.

== Next steps

After the AWS Aurora database and {jdgserver_name} are deployed and running, use the procedure in the link:{links_high-availability_multi-cluster-deploy-keycloak-kubernetes_url}[{links_high-availability_multi-cluster-deploy-keycloak-kubernetes_name}] {section} to deploy {project_name} and connect it to all previously created building blocks.


== Relevant options

[cols="12a,4",role="options"]
|===
| |Value

|
[.options-key]#`+cache-remote-backup-sites+`# 

[.options-description]#Configures a list of backup sites names to where the external Infinispan cluster backups the Keycloak data.#

[role="options-extended"]
--




*CLI:* `+--cache-remote-backup-sites+` +
*Env:* `+KC_CACHE_REMOTE_BACKUP_SITES+`
--

Available only when remote host is set


|

|
[.options-key]#`+cache-remote-host+`# 

[.options-description]#The hostname of the external Infinispan cluster.#

[role="options-extended"]
--
[.options-description-extended]#Available only when feature `multi-site` or `clusterless` is set.#



*CLI:* `+--cache-remote-host+` +
*Env:* `+KC_CACHE_REMOTE_HOST+`
--



|

|
[.options-key]#`+cache-remote-password+`# 

[.options-description]#The password for the authentication to the external Infinispan cluster.#

[role="options-extended"]
--
[.options-description-extended]#It is optional if connecting to an unsecure external Infinispan cluster. If the option is specified, `cache-remote-username` is required as well.#



*CLI:* `+--cache-remote-password+` +
*Env:* `+KC_CACHE_REMOTE_PASSWORD+`
--

Available only when remote host is set


|

|
[.options-key]#`+cache-remote-port+`# 

[.options-description]#The port of the external Infinispan cluster.#

[role="options-extended"]
--




*CLI:* `+--cache-remote-port+` +
*Env:* `+KC_CACHE_REMOTE_PORT+`
--

Available only when remote host is set


|[.options-default]#`+11222+`# (default)

|
[.options-key]#`+cache-remote-tls-enabled+`# 

[.options-description]#Enable TLS support to communicate with a secured remote Infinispan server.#

[role="options-extended"]
--
[.options-description-extended]#Recommended to be enabled in production.#



*CLI:* `+--cache-remote-tls-enabled+` +
*Env:* `+KC_CACHE_REMOTE_TLS_ENABLED+`
--

Available only when remote host is set


|
`+true+` (default), `+false+`

|
[.options-key]#`+cache-remote-username+`# 

[.options-description]#The username for the authentication to the external Infinispan cluster.#

[role="options-extended"]
--
[.options-description-extended]#It is optional if connecting to an unsecure external Infinispan cluster. If the option is specified, `cache-remote-password` is required as well.#



*CLI:* `+--cache-remote-username+` +
*Env:* `+KC_CACHE_REMOTE_USERNAME+`
--

Available only when remote host is set


|


|===
