[[Micrometer_Metrics]]
= Micrometer Metrics Subsystem

ifdef::env-github[]
:tip-caption: :bulb:
:note-caption: :information_source:
:important-caption: :heavy_exclamation_mark:
:caution-caption: :fire:
:warning-caption: :warning:
endif::[]

https://micrometer.io/[Micrometer] is a vendor-neutral observability facade that provides a generic, reusable API for registering and recording metrics related to application performance. This extension provides an integration with Micrometer, exposing its API to deployed applications so that they may expose application-specific metrics in addition to the server metrics added by the extension.

NOTE: Standard WildFly continues to use the existing metrics subsystem, so this extension must be manually added and configured. See below for details.

[[micrometer-required-extension-metrics]]
== Extension

This `org.wildfly.extension.micrometer` extension is available to all the standalone configurations included in the WildFly distribution, but must be added manually:

[source,options="nowrap"]
----
[standalone@localhost:9990 /] /extension=org.wildfly.extension.micrometer:add
[standalone@localhost:9990 /] /subsystem=micrometer:add(endpoint="http://localhost:4318/v1/metrics")
----

This subsystem exposes metrics from the WildFly Management Model and JVM MBeans, as well as end-user deployments via the Micrometer API now exposed by the server. Metrics are made available via Micrometer Registries, and WildFly supports two of these:

- OTLP - a push-based registry that publishes data periodically to the configured OpenTelemetry Collector
- Prometheus - a pull-based registry that allows for metrics to be pulled (or scraped) by an external process

Note that this an alternative to the existing WildFly Metrics extension. While they may be run concurrently, it is not advisable, as this will likely have an impact on server performance due to the duplicated metrics collection. To disable WildFly Metrics, issue these commands:

[source,options="nowrap"]
----
[standalone@localhost:9990 /] /subsystem=metrics:remove
[standalone@localhost:9990 /] /extension=org.wildfly.extension.metrics:remove
[standalone@localhost:9990 /] reload
----

Additionally, WildFly support for OpenTelemetry and MicroProfile Telemetry now includes support for metrics as well. Running these in conjunction with Micrometer poses the same performance concerns mentioned above, so administrators/developers are advised to choose one subsystem and remove the other. If, however, Micrometer metrics and OpenTelemetry tracing/logging are both desired, OpenTelemetry/MicroProfile Telemetry metrics can be disabled via a system property:

[source,options="nowrap"]
-----
[standalone@localhost:9990 /] /system-property=otel.metrics.exporter:add(value=none)
[standalone@localhost:9990 /] reload
-----

or via `otel.metrics.exporter=none` in a MicroProfile Config `ConfigSource`.

[IMPORTANT]
If no registry is explicitly added and configured, no metrics will be exported. WildFly administrators *MUST* choose at least one or the other, but _may_ choose both, if desired.

=== OTLP Registry

To add the OTLP registry, execute this command:

[source,options="nowrap"]
----
[standalone@localhost:9990 /] /subsystem=micrometer/registry=otlp:add(endpoint="http://localhost:4318/v1/metrics")
----

There are two attributes for the OTLP registry, `endpoint` and `step`, which are described below, but note the `endpoint` attribute is required.

[IMPORTANT]
It is assumed that the server administrator will provision and secure the collector, which is outside the scope of this document.

=== Prometheus Registry

To install the Prometheus registry, execute this command:

[source,options="nowrap"]
----
[standalone@localhost:9990 /] /subsystem=micrometer/registry=prometheus:add(context="/metrics")
----

This will create a context on the management interface from which metrics can be scraped. For example, the above configuration would result in a URL that looks like `http://localhost:9090/metrics`. If you have not disabled the WildFly Metrics subsystem, you will not be able to use `/metrics` as your context, as that is already configured. To disable the WildFly Metrics Subsystem, see the `remove` commands above.

== Management Model

The `/subsystem=micrometer` resource defines one attribute, that is shared by the two supported registries:

* `exposed-subsystems` - a list of strings corresponding to the names of subsystems that exposes their metrics in the
HTTP metrics endpoints. By default, it is not defined (there will be no metrics exposed by subsystem). The special wildcard "{asterisk}" can be used to expose metrics from _all_ subsystems. The standalone configuration sets this attribute to "{asterisk}".

=== OTLP Registry Model

The `/subsystem=micrometer/registry=otlp` resource defines two attributes:

* `endpoint` - the URL of the metrics collector endpoint (default: http://localhost:4318/v1/metrics)
* `step` - the step size, or reporting frequency, to use (in seconds).

=== Prometheus Registry Model

The `/subsystem=micrometer/registry=prometheus` resource defines two attributes:

* `context` - (*required*) The context on which to make metrics available for scraping. This will be exposed on the management interface (e.g, `http://localhost:9090`)
* `security-enabled` - Whether or not to enable security on the Micrometer Prometheus context. This will use the same user credentials as the web console and command line interface tools. The default is `true`.

== Exposed Metrics

The following types of metrics are gathered and published by Micrometer:

* Metrics from JVM MBeans (read directly from the JMX MBeans)
* WildFly metrics from the management model subsystem and deployment subtrees.
* Any application-specific metrics provided via the injected Micrometer `MeterRegistry` instance.

=== WildFly Metrics Description

WildFly metrics names are based on the subsystem that provides them, as well as the name of the attribute from the management model.

For example Undertow exposes a metric attribute `request-count` for every `Servlet` in a deployment. This attribute will be exposed with the name `undertow_request_count`. Other information such as the name of the `Servlet` are added to the tags of the metric.

NOTE: Prometheus has its own https://prometheus.io/docs/practices/naming/[metrics naming scheme], so metrics names exposed via that registry will adhere to that and may differ from the description above.

The https://github.com/wildfly/quickstart/tree/main/helloworld[helloworld quickstart] demonstrates the use of CDI and Servlet in WildFly. A corresponding metric will be exposed for it with the name and tags (formatted for readability):

[source]
-----
undertow_request_count_total{
    deployment="helloworld.war",
    servlet="org.jboss.as.quickstarts.helloworld.HelloWorldServlet",
    subdeployment="helloworld.war"} 4.0
-----

[NOTE]
Some subsystems (such as `undertow` or `messaging-activemq`) do not enable their statistics by default as they have an impact on performance and memory usage. These subsystems provide a `statistics-enabled` attribute that must be set to `true` to enable them. For convenience, WildFly standalone configuration provides expressions to enable the statistics by setting a System property `-Dwildfly.statistics-enabled=true` to enable statistics on the subsystems provided by the configuration.

== Use in Applications

Unlike the previous metrics systems, this new extension exposes an API (that of https://micrometer.io[Micrometer]) to applications in order to allow developers to record and export metrics out of the box. To do so, application developers will need to inject a `MeterRegistry` instance:

[source,java]
-----
package com.redhat.wildfly.micrometerdemo;

import jakarta.enterprise.context.RequestScoped;
import jakarta.inject.Inject;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import io.micrometer.core.instrument.MeterRegistry;

@RequestScoped
@Path("/endpoint")
public class Endpoint {
    @Inject
    private MeterRegistry registry;

    @GET
    public String method() {
        registry.counter("dummy").increment();
        return "Counter is " + registry.counter("dummy").count();
    }
}
-----

This provides the application with a `MeterRegistry` instance that will have any recorded metrics exported with the system metrics WildFly already exposes. There is no need for an application to include the Micrometer dependencies in the application archive, as they are provided by the server out-of-the-box:

[source,xml]
-----
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
    <version>${version.micrometer}</version>
    <scope>provided</scope>
</dependency>
-----
