
:guide-id: multi-cluster-operate-synchronize
:guide-parent: multi-cluster
:guide-title: Synchronizing sites
:guide-summary: Synchronize an offline site with an online site.
:guide-priority: 999
:guide-tile-visible: false
:version: 26.4.0

include::../../attributes.adoc[]

[[multi-cluster-operate-synchronize]]
= Synchronizing sites

ifeval::["" == "true"]
WARNING: This {section} is describing a feature which is currently in preview.
ifeval::["" == ""]
Please provide your feedback while we’re continuing to work on this.
endif::[]
ifeval::["" != ""]
Please provide your feedback by link:[joining this discussion] while we’re continuing to work on this.
endif::[]
endif::[]


include::../partials/infinispan/infinispan-attributes.adoc[]

== When to use this procedure

Use this when the state of {jdgserver_name} clusters of two sites become disconnected and the contents of the caches are out-of-sync.
Perform this for example after a split-brain or when one site has been taken offline for maintenance.

At the end of the procedure, the data on the secondary site have been discarded and replaced by the data of the active site. All caches in the offline site are cleared to prevent invalid cache contents.

== Procedures

=== {jdgserver_name} Cluster

For the context of this {section}, `{keep-site-name}` is the currently active site and `{stale-site-name}` is an offline site that is not part
of the AWS Global Accelerator EndpointGroup and is therefore not receiving user requests.

WARNING: Transferring state may impact {jdgserver_name} cluster performance by increasing the response time and/or resources usage.

The first procedure is to delete the stale data from the offline site.

. Login into the offline site.

. Shutdown {project_name}.
This will clear all {project_name} caches and prevents the {project_name} state from being out-of-sync with {jdgserver_name}.
+
When deploying {project_name} using the {project_name} Operator, change the number of {project_name} instances in the {project_name} Custom Resource to 0.

. Connect into {jdgserver_name} Cluster using the {jdgserver_name} CLI tool:
+
.Command:
[source,bash,subs="+attributes"]
----
kubectl -n {ns} exec -it pods/{cluster-name}-0 -- ./bin/cli.sh --trustall --connect https://127.0.0.1:11222
----
+
It asks for the username and password for the {jdgserver_name} cluster.
Those credentials are the one set in the link:{links_high-availability_multi-cluster-deploy-infinispan-kubernetes-crossdc_url}[{links_high-availability_multi-cluster-deploy-infinispan-kubernetes-crossdc_name}] {section} in the configuring credentials section.
+
.Output:
[source,bash,subs="+attributes"]
----
Username: developer
Password:
[{cluster-name}-0-29897@ISPN//containers/default]>
----
+
NOTE: The pod name depends on the cluster name defined in the {jdgserver_name} CR.
The connection can be done with any pod in the {jdgserver_name} cluster.
. Disable the replication from {stale-site} site to the {keep-site} site by running the following command.
It prevents the clear request to reach the {keep-site} site and delete all the correct cached data.
+
.Command:
[source,bash,subs="+attributes"]
----
site take-offline --all-caches --site={keep-site-name}
----
+
.Output:
[source,bash,subs="+attributes"]
----
{
  "authenticationSessions" : "ok",
  "work" : "ok",
  "loginFailures" : "ok",
  "actionTokens" : "ok"
}
----

. Check the replication status is `offline`.
+
.Command:
[source,bash,subs="+attributes"]
----
site status --all-caches --site={keep-site-name}
----
+
.Output:
[source,bash,subs="+attributes"]
----
{
  "status" : "offline"
}
----
+
If the status is not `offline`, repeat the previous step.
+
WARNING: Make sure the replication is `offline` otherwise the clear data will clear both sites.

. Clear all the cached data in {stale-site} site using the following commands:
+
.Command:
[source,bash,subs="+attributes"]
----
clearcache actionTokens
clearcache authenticationSessions
clearcache loginFailures
clearcache work
----
+
These commands do not print any output.

. Re-enable the cross-site replication from {stale-site} site to the {keep-site} site.
+
.Command:
[source,bash,subs="+attributes"]
----
site bring-online --all-caches --site={keep-site-name}
----
+
.Output:
[source,bash,subs="+attributes"]
----
{
  "authenticationSessions" : "ok",
  "work" : "ok",
  "loginFailures" : "ok",
  "actionTokens" : "ok"
}
----

. Check the replication status is `online`.
+
.Command:
[source,bash,subs="+attributes"]
----
site status --all-caches --site={keep-site-name}
----
+
.Output:
[source,bash,subs="+attributes"]
----
{
  "status" : "online"
}
----

Now we are ready to transfer the state from the active site to the offline site.

. Login into your Active site

. Connect into {jdgserver_name} Cluster using the {jdgserver_name} CLI tool:
+
.Command:
[source,bash,subs="+attributes"]
----
kubectl -n {ns} exec -it pods/{cluster-name}-0 -- ./bin/cli.sh --trustall --connect https://127.0.0.1:11222
----
+
It asks for the username and password for the {jdgserver_name} cluster.
Those credentials are the one set in the link:{links_high-availability_multi-cluster-deploy-infinispan-kubernetes-crossdc_url}[{links_high-availability_multi-cluster-deploy-infinispan-kubernetes-crossdc_name}] {section} in the configuring credentials section.
+
.Output:
[source,bash,subs="+attributes"]
----
Username: developer
Password:
[{cluster-name}-0-29897@ISPN//containers/default]>
----
+
NOTE: The pod name depends on the cluster name defined in the {jdgserver_name} CR.
The connection can be done with any pod in the {jdgserver_name} cluster.

. Trigger the state transfer from the {keep-site} site to the {stale-site} site.
+
.Command:
[source,bash,subs="+attributes"]
----
site push-site-state --all-caches --site={stale-site-name}
----
+
.Output:
[source,bash,subs="+attributes"]
----
{
  "authenticationSessions" : "ok",
  "work" : "ok",
  "loginFailures" : "ok",
  "actionTokens" : "ok"
}
----

. Check the replication status is `online` for all caches.
+
.Command:
[source,bash,subs="+attributes"]
----
site status --all-caches --site={stale-site-name}
----
+
.Output:
[source,bash,subs="+attributes"]
----
{
  "status" : "online"
}
----

. Wait for the state transfer to complete by checking the output of `push-site-status` command for all caches.
+
.Command:
[source,bash,subs="+attributes"]
----
site push-site-status --cache=actionTokens
site push-site-status --cache=authenticationSessions
site push-site-status --cache=loginFailures
site push-site-status --cache=work
----
+
.Output:
[source,bash,subs="+attributes"]
----
{
  "{stale-site-name}" : "OK"
}
{
  "{stale-site-name}" : "OK"
}
{
  "{stale-site-name}" : "OK"
}
{
  "{stale-site-name}" : "OK"
}
----
+
Check the table in {infinispan-xsite-docs}#rest_v2_xsite_state_push_cross-site-operations-rest[this section for the Cross-Site Documentation] for the possible status values.
+
If an error is reported, repeat the state transfer for that specific cache.
+
.Command:
[source,bash,subs="+attributes"]
----
site push-site-state --cache=<cache-name> --site={stale-site-name}
----

. Clear/reset the state transfer status with the following command
+
.Command:
[source,bash,subs="+attributes"]
----
site clear-push-site-status --cache=actionTokens
site clear-push-site-status --cache=authenticationSessions
site clear-push-site-status --cache=loginFailures
site clear-push-site-status --cache=work
----
+
.Output:
[source,bash,subs="+attributes"]
----
"ok"
"ok"
"ok"
"ok"
----

Now the state is available in the offline site, {project_name} can be started again:

. Login into your secondary site.

. Startup {project_name}.
+
When deploying {project_name} using the {project_name} Operator, change the number of {project_name} instances in the {project_name} Custom Resource to the original value.

=== AWS Aurora Database

No action required.

=== AWS Global Accelerator

Once the two sites have been synchronized, it is safe to add the previously offline site back to the Global Accelerator
EndpointGroup following the steps in the link:{links_high-availability_multi-cluster-operate-site-online_url}[{links_high-availability_multi-cluster-operate-site-online_name}] {section}.

== Further reading

See link:{links_high-availability_multi-cluster-concepts-infinispan-cli-batch_url}[{links_high-availability_multi-cluster-concepts-infinispan-cli-batch_name}].


