[[Observability_Tracing]]
= OpenTelemetry Subsystem

ifdef::env-github[]
:tip-caption: :bulb:
:note-caption: :information_source:
:important-caption: :heavy_exclamation_mark:
:caution-caption: :fire:
:warning-caption: :warning:
endif::[]

[[required-extension-opentelemetry]]
== Extension

This extension is not included in any of the standalone configurations included in the WildFly distribution.
To enable, the administrator must run the following CLI commands:

[source,options=nowrap]
-----
[standalone@localhost:9990 /]  /extension=org.wildfly.extension.opentelemetry:add()
[standalone@localhost:9990 /]  /subsystem=opentelemetry:add()
-----

[IMPORTANT]
The OpenTelemetry subsystem provides support for all three "pillars" of observability: traces, metrics, and logging. It is important for administrators to note that WildFly also supports two other metrics-reporting subsystems, `metrics` and `micrometer`. While these subsystems may all be run concurrently, it is not advisable, as this will likely have an impact on server performance due to the duplicated metrics collection.

To disable WildFly Metrics, then, issue these commands:

[source,options="nowrap"]
----
[standalone@localhost:9990 /] /subsystem=metrics:remove()
[standalone@localhost:9990 /] /extension=org.wildfly.extension.metrics:remove()
[standalone@localhost:9990 /] reload
----

and to disable Micrometer, issue these commands:

[source,options="nowrap"]
----
[standalone@localhost:9990 /] /subsystem=micrometer:remove()
[standalone@localhost:9990 /] /extension=org.wildfly.extension.micrometer:remove()
[standalone@localhost:9990 /] reload
----

If, however, the use of Micrometer is desired for metrics and OpenTelemetry for tracing, for example, OpenTelemetry metrics can be disabled via `otel.metrics.exporter=none` in a MicroProfile Config `ConfigSource` or by setting a system either when starting the server or via the server config file:

[source,options="nowrap"]
-----
[standalone@localhost:9990 /] /system-property=otel.metrics.exporter:add(value=none)
[standalone@localhost:9990 /] reload
-----

== Configuration

Systems administrators can configure a number of aspects of OpenTelemetry: the exporter, span processor, and sampler.

=== Exporter

The exporter can be selected and configured using the `exporter` child element, which supports these attributes:

* `exporter`: WildFly currently supports only one exporter
** `otlp`: The default, which use the OpenTelemetry protocol
* `endpoint`: The URL via which OpenTelemetry will push traces. The default is OTLP's gRPC-based endpoint, `http://localhost:4317`

.IMPORTANT CHANGE
[NOTE]
====
Earlier versions of WildFly supported `jaeger` as a valid exporter type. Jaeger support, however, has been dropped by
the OpenTelemetry project upstream, so its support has been removed from WildFly as well. Any server configurations with
`jaeger` still configured will fail to start the `opentelemetry` subsystem and apps using OpenTelemetry will fail to
deploy  until the server is reconfigured to use `otlp`. You can, however, start the server in admin-only mode in order
to reconfigure the value. Of course, editing the  XML config file is also a valid option should you prefer that approach.

Note also that OTLP has a different default value for the endpoint, so that will need to be configured appropriately for
your environment.
====

=== Span Processor
The span process is configured via the `span-processor` element, which supports the following attributes:

* `type`: The type of span processor to use.
** `batch`: The default processor, which sends traces in batches as configured via the remaining attributes
** `simple`: Traces are pushed to the exporter as they finish.
* `batch-delay`: The amount of time, in milliseconds, to wait before traces are published (default: 5000)
* `max-queue-size`: The maximum size of the queue before traces are dropped (default: 2048)
* `max-export-batch-size`: The maximum number of traces that are published in each batch, which must be smaller or equal to `max-queue-size (default: 512)
* `export-timeout`: The maximum amount of time in milliseconds to allow for an export to complete before being cancelled (default: 30000)

=== Sampler

The sampler is configured via the `sampler` element:

* `type`: The type of sampler to use
** `on`: Always on (all traces are recorded)
** `off`: Always off (no traces are recorded)
** `ratio`: Return a ratio of the traces (e.g., 1 trace in 10000).
* `ratio`: The value used to configure the `ratio` sampler, which must be within [0.0, 1.0].For example, if 1 trace in 10,000 is to be exported, this value would be `0.0001`.

=== Example Configuration

The following XML is an example of the full configuration, including default values (WildFly does not typically persist
default values, so what you see in the configuration file may look different):

[source,xml,options="nowrap"]
-----
<subsystem xmlns="urn:wildfly:opentelemetry:1.0"
        service-name="example">
    <exporter
        type="otlp"
        endpoint="http://localhost:4317"/>
    <span-processor
        type="batch"
        batch-delay="4500"
        max-queue-size="128"
        max-export-batch-size="512"
        export-timeout="45"/>
    <sampler
        type="on"/>
</subsystem>
-----

== Application Usage

All incoming REST requests are automatically traced, so no work needs be done in user applications.If a REST request is received and the OpenTelemetry context propagation header (`traceparent`) is present, the request will traced as part of the remote trace context automatically.

Likewise, all Jakarta REST Client calls will have the trace context added to outgoing request headers so that requests to external applications can be traced correctly (assuming the remote system properly handles OpenTelemetry trace context propagation).If the REST Client call is made to another application on the local WildFly server, or a remote server of the same version or later, the trace context will propagate automatically as described above.

While automatic tracing may be sufficient in many cases, it will often be desirable to have traces occur throughout the user application.To support that, WildFly makes available the `io.opentelemetry.api.OpenTelemetry` and
`io.opentelemetry.api.trace.Tracer` instances, via CDI injection.A user application, then is able to create arbitrary spans as part of the server-managed trace:

[source,java,options="nowrap"]
-----
@Path("/myEndpoint")
public class MyEndpoint {
    @Inject
    private Tracer tracer;

    @GET
    public Response doSomeWork() {
        final Span span = tracer.spanBuilder("Doing some work")
                .startSpan();
        span.makeCurrent();
        doSomeMoreWork();
        span.addEvent("Make request to external system.");
        makeExternalRequest();
        span.addEvent("All the work is done.");
        span.end();

        return Response.ok().build();
}
-----

== Component Reference

OpenTelemetry support is provided via the https://opentelemetry.io[OpenTelemetry] project.
