[[Suspend,_Resume_and_Graceful_shutdown]]
= Suspend, Resume and Graceful shutdown

[[core-concepts]]
== Core Concepts

WildFly introduces the ability to suspend and resume servers. This can
be combined with shutdown to enable the server to gracefully finish
processing all active requests and then shut down. When a server is
suspended it will immediately stop accepting new requests, but wait for
existing requests to complete. A suspended server can be resumed at any
point, and will begin processing requests immediately. Suspending and
resuming has no effect on deployment state (e.g. if a server is
suspended singleton Jakarta Enterprise Beans's will not be destroyed). As of WildFly 11 it is
also possible to start a server in suspended mode which means it will
not accept requests until it has been resumed. Servers will also be
suspended during the boot process, so no requests will be accepted until
the startup process is 100% complete.

Suspend/Resume has no effect on management operations; management
operations can still be performed while a server is suspended. If you
wish to perform a management operation that will affect the operation of
the server you can suspend the server, perform the operation, and then
resume the server. This allows all requests to finish, and makes sure
that no requests are running while the management changes are taking
place.

[NOTE]
====
If you perform a management operation while the server is suspended,
and the response to that operation includes the
`operation-requires-reload` or `operation-requires-restart` response
headers, then the operation will not take full effect until that
reload or restart is done. Simply resuming the server will not be
sufficient to cause the change to take effect.
====

When a server is suspending it goes through four different states:

* *RUNNING* - The normal state, the server is accepting requests and
running normally
* *PRE_SUSPEND* - In PRE_SUSPEND the server will notify external parties
that it is about to suspend, for example mod_cluster will notify the
load balancer that the deployment is suspending. Requests are still
accepted in this phase.
* *SUSPENDING* - All new requests are rejected, and the server is
waiting for all active requests to finish. If there are no active
requests at suspend time this phase will be skipped.
* *SUSPENDED* - All requests have completed, and the server is
suspended.

[[starting-suspended]]
== Starting Suspended

In order to start into suspended mode when using a standalone server you
need to add *--start-mode=suspend* to the command line. It is also
possible to specify the start-mode in the *reload* operation to cause
the server to reload into suspended mode (other possible values for
start-mode are *normal* and *admin-only*).

In domain mode servers can be started in suspended mode by passing the
*suspend=true* parameter to any command that causes a server to start,
restart or reload (e.g. :start-servers(suspend=true)).

[[the-request-controller-subsystem]]
== The Request Controller Subsystem

WildFly introduces a new subsystem called the Request Controller
Subsystem. This optional subsystem tracks all requests at their entry
point, which is how the graceful shutdown mechanism knows when all requests
are done. (It also allows you to provide a global limit on the total
number of running requests).

If this subsystem is not present suspend/resume will be limited. In
general things that happen in the PRE_SUSPEND phase will work as normal
(stopping message delivery, notifying the load balancer); however the
server will not wait for all requests to complete and instead will move
straight to SUSPENDED mode.

There is a small performance penalty associated with the request
controller subsystem (about on par with enabling statistics), so if you
do not require the suspend/resume functionality this subsystem can be
removed to get a small performance boost.

[[subsystem-integrations]]
== Subsystem Integrations

Suspend/Resume is a service provided by the WildFly platform that any
subsystem may choose to integrate with. Some subsystems integrate
directly with the suspend controller, while others integrate through the
request controller subsystem.

The following subsystems support graceful shutdown. Note that only
subsystems that provide an external entry point to the server need
graceful shutdown support. For example the Jakarta RESTful Web Services subsystem does not
require suspend/resume support as all access to Jakarta RESTful Web Services is through the
web connector.

* *Undertow* - Undertow will wait for all requests to finish.
* *mod_cluster* - The mod_cluster subsystem will notify the load
balancer that the server is suspending in the PRE_SUSPEND phase.
* *Jakarta Enterprise Beans* - Jakarta Enterprise Beans will wait
for all remote Jakarta Enterprise Beans requests and MDB message
deliveries to finish. Delivery to MDB's is stopped in the PRE_SUSPEND
phase. Jakarta Enterprise Beans timers are suspended, and missed timers will be activated
when the server is resumed.
* *Batch* - Batch jobs will be stopped at a checkpoint while the server
is suspending. They will be restarted from that checkpoint when the
server returns to running mode.
* *EE Concurrency* - The server will wait for all active jobs to finish.
All jobs that have already been queued will be skipped.
* *Transactions* - The transaction subsystem waits for all running
transactions to finish while the server is suspending. During that time
the server refuses to start any new transaction. But any in-flight
transaction will be serviced - e.g. the server will accept any
incoming remote call which carries the context of a transaction already
started at the suspending server.

[[transactions-and-Jakarta-Enterprise-Beans]]
=== Transactions and Jakarta Enterprise Beans
When you work with Jakarta Enterprise Beans you have to enable the graceful shutdown
functionality by setting the attribute `enable-graceful-txn-shutdown` to
`true`. For example, in the ejb3 subsystem section of `standalone.xml`:

[source,xml,options="nowrap"]
----
<enable-graceful-txn-shutdown value="false"/>
----

By *default* graceful shutdown is *disabled* for the ejb subsystem.
The reason for this is that the behavior might be unwelcome in cluster
environments, as the server notifies remote clients that the node is no
longer available for remote calls only after the transactions are
finished. During that brief window of time, the client of a cluster may
send a new request to a node that is shutting down and it will refuse the
request because it is not related to an existing transaction.
If this attribute `enable-graceful-txn-shutdown` is set to `false`, we
disable the graceful behavior and Jakarta Enterprise Beans clients will not attempt to invoke
the node when it suspends, regardless of active transactions.

[[standalone-mode]]
== Standalone Mode

Suspend/Resume can be controlled via the following CLI operations
and commands in standalone mode:

`:suspend(suspend-timeout=x)`

Suspends the server. If the timeout is specified it will wait in the
SUSPENDING phase up to the specified number of seconds for all requests
to finish. If there is no timeout specified or the value is less than
zero it will wait indefinitely.

`:resume`

Resumes a previously suspended server. The server should be able to
begin serving requests immediately.

`:read-attribute(name=suspend-state)`

Returns the current suspend state of the server.

`shutdown --suspend-timeout=x`

If a timeout parameter is passed to the shutdown command then a graceful
shutdown will be performed. The server will be suspended, and will wait
in SUSPENDING state up to the specified number of seconds for all requests
to finish before shutting down. A timeout value of less than zero means
it will wait indefinitely.

[[domain-mode]]
== Domain Mode

Domain mode has similar operations as standalone mode, however they can be
applied at global, server group, server and host levels:

*Whole Domain*

`:suspend-servers(suspend-timeout=x)`

`:resume-servers`

`:stop-servers(suspend-timeout=x)`

*Server Group*

`/server-group=main-server-group:suspend-servers(suspend-timeout=x)`

`/server-group=main-server-group:resume-servers`

`/server-group=main-server-group:stop-servers(suspend-timeout=x)`

*Server*

`/host=primary/server-config=server-one:suspend(suspend-timeout=x)`

`/host=primary/server-config=server-one:resume`

`/host=primary/server-config=server-one:stop(suspend-timeout=x)`

*Host level*

`/host=primary:suspend-servers(suspend-timeout=x)`

`/host=primary:resume-servers`

`/host=primary:shutdown(suspend-timeout=x)`

Note that even though the host controller itself is being shut down, the suspend-timeout attribute for the shutdown operation at host level is applied to the servers only and not to the host controller itself.


[[graceful-shutdown-from-an-os-signal]]
== Graceful Shutdown via an OS Signal

If you use an OS signal like `TERM` to shut down your WildFly standalone server
process, e.g. via `kill -15 <pid>`, the WildFly server will shut down gracefully.
By default, the behavior will be analogous to a CLI `shutdown --suspend-timeout=0` command;
that is the process will not wait in SUSPENDING state for in-flight requests to
complete before proceeding to SUSPENDED state and then shutting down. A different
timeout can be configured by setting the `org.wildfly.sigterm.suspend.timeout`
system property. The value of the property should be an integer indicating the maximum
number of seconds to wait for in-flight requests to complete. A value of `-1` means
the server should wait indefinitely.

Graceful shutdown via an OS signal will not work if the server JVM is configured
to disable signal handling (i.e. with the `-Xrs` argument to java). It also won't
work if the method used to terminate the process doesn't result in a signal the
JVM can respond to (e.g. `kill -9`).

In a managed domain, Process Controller and Host Controller processes will not attempt
any sort of graceful shutdown in response to a signal. A domain mode server may, but
the proper way to control the lifecycle of a domain mode server process is via the
management API and its managing Host Controller, not via direct signals to the server
process.

[[non-graceful-startup]]
== Non-graceful Startup

By default, WildFly starts up gracefully, meaning that incoming requests are queued or cleanly rejected
until the server is ready to process them. In some instances, though, it may be desirable to allow the
server to begin to process requests at the earliest possible moment. One such example might be
when two deployed applications need to interact with one another during the deployment or
application startup. In one such scenario, Application A needs to make a REST request to
Application B to get information vital to its own startup. Under a graceful startup, the request
to Application B will block until the server is fully started. However, the server can't fully
start, as Application A is waiting for data from Application B before its deploy/startup can
complete. In this situation, a deadlock occurs, and the server startup times out.

A non-graceful startup is intended to address this situation in that it will allow WildFly to
begin attempting to answer requests as soon as possible. In the scenario above, assuming
Application B has successfully deployed/started, Application A can also start immediately, as its
request will be fulfilled. Note, however, that a race condition can occur: if Application B is
not yet deployed (e.g., the deploy order is incorrect, or B has not finished starting), then
Application A may still fail to start since Application B is not available. WildFly users making
use of non-graceful startups must be aware of this and take steps to remedy those scenarios. With
a non-graceful startup, however, WildFly will no longer be the cause of a deployment failure in
such a configuration.

Some discussion here of how this relates to reloading and restarting, as well as to suspended starts, is
important. When reloading, the `ApplicationServerService` is stopped, and a new one started. It is equivalent
to if it was being started the first time: all the same stuff happens, but it happens faster because a lot of
classloading and static initialization doesn't have to happen again. This includes honoring the value of
`graceful-startup`, so if the server was initially started non-gracefully, it will be reloaded in the same manner.

Restarting the server is similar. A restart means a new JVM, so all the initialization happens again, exactly
as it did on the first start. When restarting in domain mode, the Host Controller simply rereads the config
file and does the same thing it did the first time. In standalone, the restart is driven by `standalone.[sh|ps1|bat]`.
The running JVM exits with a specific exit code, which the script recognizes and starts a new server, using the
same parameters as the first start, so if you start a server non-gracefully, you will restart a server
non-gracefully.

Finally, there's `start-mode=suspend`. In the event that an administrator specifies a suspended start as well as a
non-graceful start, the suspended start will "win". That is to say, the server will start in a suspended mode,
the `graceful-start=false` will be disregarded, and the server will log a message indicating that this is happening.
