
:guide-id: multi-cluster-concepts-memory-and-cpu-sizing
:guide-parent: multi-cluster
:guide-title: Concepts for sizing CPU and memory resources
:guide-summary: Understand concepts for avoiding resource exhaustion and congestion.
:guide-priority: 999
:guide-tile-visible: false
:version: 26.4.0

include::../../attributes.adoc[]

[[multi-cluster-concepts-memory-and-cpu-sizing]]
= Concepts for sizing CPU and memory resources

ifeval::["" == "true"]
WARNING: This {section} is describing a feature which is currently in preview.
ifeval::["" == ""]
Please provide your feedback while we’re continuing to work on this.
endif::[]
ifeval::["" != ""]
Please provide your feedback by link:[joining this discussion] while we’re continuing to work on this.
endif::[]
endif::[]


Use this as a starting point to size a product environment.
Adjust the values for your environment as needed based on your load tests.

[#multi-cluster-performance-recommendations]
== Performance recommendations

include::../partials/concepts/perf_recommendations.adoc[]

[#multi-cluster-measture-running-instance]
=== Measuring the activity of a running {project_name} instance

Sizing of a {project_name} instance depends on the actual and forecasted numbers for password-based user logins, refresh token requests, and client credential grants as described in the previous section.

To retrieve the actual numbers of a running {project_name} instance for these three key inputs, use the metrics {project_name} provides:

* The user event metric `keycloak_user_events_total` for event type `login` includes both password-based logins and cookie-based logins, still it can serve as a first approximate input for this sizing guide.
* To find out number of password validations performed by {project_name} use the metric `keycloak_credentials_password_hashing_validations_total`.
The metric also contains tags providing some details about the hashing algorithm used and the outcome of the validation.
Here is the list of available tags: `realm`, `algorithm`, `hashing_strength`, `outcome`.
* Use the user event metric `keycloak_user_events_total` for the event types `refresh_token` and `client_login` for refresh token requests and client credential grants respectively.

See the link:{links_observability_event-metrics_url}[{links_observability_event-metrics_name}] and link:{links_observability_metrics-for-troubleshooting-http_url}[{links_observability_metrics-for-troubleshooting-http_name}] {sections} for more information.

These metrics are crucial for tracking daily and weekly fluctuations in user activity loads,
identifying emerging trends that may indicate the need to resize the system and
validating sizing calculations.
By systematically measuring and evaluating these user event metrics,
you can ensure your system remains appropriately scaled and responsive to changes in user behavior and demand.

[#multi-cluster-single-site-calculation]
=== Calculation example (single cluster)

include::../partials/concepts/perf_single_cluster_calculation.adoc[]

=== Sizing a multi-cluster setup

To create the sizing an active-active Keycloak setup with two AZs in one AWS region, following these steps:

* Create the same number of Pods with the same memory sizing as above on the second site.

* The database sizing remains unchanged. Both sites will connect to the same database writer instance.

In regard to the sizing of CPU requests and limits, there are different approaches depending on the expected failover behavior:

Fast failover and more expensive::
Keep the CPU requests and limits as above for the second site. This way any remaining site can take over the traffic from the primary site immediately without the need to scale.

Slower failover and more cost-effective::
Reduce the CPU requests and limits as above by 50% for the second site. When one of the sites fails, scale the remaining site from 3 Pod to 6 Pods either manually, automated, or using a Horizontal Pod Autoscaler. This requires enough spare capacity on the cluster or cluster auto-scaling capabilities.

Alternative setup for some environments::
Reduce the CPU requests by 50% for the second site, but keep the CPU limits as above. This way, the remaining site can take the traffic, but only at the downside that the Nodes will experience CPU pressure and therefore slower response times during peak traffic.
The benefit of this setup is that the number of Pods does not need to scale during failovers which is simpler to set up.

== Reference architecture

The following setup was used to retrieve the settings above to run tests of about 10 minutes for different scenarios:

* OpenShift 4.17.x deployed on AWS via ROSA.
* Machine pool with `c7g.2xlarge` instances.^*^
* {project_name} deployed with the Operator and 3 pods in a high-availability setup with two sites in active/active mode.
* OpenShift's reverse proxy runs in the passthrough mode where the TLS connection of the client is terminated at the Pod.
* Database Amazon Aurora PostgreSQL in a multi-AZ setup.
* Default user password hashing with Argon2 and 5 hash iterations and minimum memory size 7 MiB https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#argon2id[as recommended by OWASP] (which is the default).
* Client credential grants do not use refresh tokens (which is the default).
* Database seeded with 20,000 users and 20,000 clients.
* Infinispan local caches at default of 10,000 entries, so not all clients and users fit into the cache, and some requests will need to fetch the data from the database.
* All authentication sessions in distributed caches as per default, with two owners per entries, allowing one failing Pod without losing data.
* All user and client sessions are stored in the database and are not cached in-memory as this was tested in a multi-cluster setup.
Expect a slightly higher performance for single-site setups as a fixed number of user and client sessions will be cached.
* OpenJDK 21

^*^ For non-ARM CPU architectures on AWS (`c7i`/`c7a` vs. `c7g`) we found that client credential grants and refresh token workloads were able to deliver up to two times the number of operations per CPU core, while password hashing was delivering a constant number of operations per CPU core. Depending on your workload and your cloud pricing, please run your own tests and make your own calculations for mixed workloads to find out which architecture delivers a better pricing for you.


