> ## Documentation Index > Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Micrometer > Metrics from Java apps instrumented with Micrometer, including Spring Boot, sent to the CXDOT Collector over OTLP. [Micrometer](https://micrometer.io/) is the metrics instrumentation library Spring Boot uses for Java Virtual Machine (JVM) apps. Use Micrometer's OTLP registry to send your services' metrics to the [CXDOT Collector](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector) running on the same node. For the Micrometer integration, there is nothing to configure in the CXDOT Collector. Each service pushes its own metrics over OTLP to the [OTLP integration](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/otlp.md), which the CXDOT Collector runs by default. Because the metrics arrive through the OTLP integration, they carry `cxdot.integration.name=otlp`, not `micrometer`. The Micrometer integration stays listed as available after its metrics arrive, and it reports no health status. Adding the OTLP registry is additive. Spring Boot sends every metric to every registry the app includes, so your existing StatsD or Datadog registry keeps sending what it sends today. The CXDOT Collector supports Micrometer 1.11 or greater. ## Supported telemetry types The CXDOT Collector supports these telemetry types from Micrometer: | Type | Supported | | :-: | :-: | | Logs | No | | Metrics | Yes | | Traces | No | | Events | No | ## Prerequisites Sending Micrometer metrics over OTLP has the following prerequisites: * Find each service's Spring Boot version: look for `spring-boot` in its Gradle or Maven dependencies. The setup differs by Spring Boot line. * Micrometer 1.11 or greater, the first version that sends delta temporality. Spring Boot 3.1 and later include it. Spring Boot 2 and 3.0 include an older version. See [Upgrade Micrometer](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/micrometer.md#upgrade-micrometer). Upgrading Micrometer doesn't require upgrading Spring Boot. * Run the CXDOT Collector on each node with host networking, the Helm chart's default. OTLP ingest is on by default, so apps send to their node's IP address on port `4318`. ## Configure Follow the section for each service's Spring Boot line, then confirm collection with the steps in Validate. ### Spring Boot 2 (2.5 to 2.7) Spring Boot 2 includes Micrometer 1.7 to 1.9 and doesn't configure the OTLP registry itself. To send metrics from a Spring Boot 2 service, follow these steps: 1. Upgrade Micrometer to 1.11 or greater. See [Upgrade Micrometer](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/micrometer.md#upgrade-micrometer). 2. Add the Micrometer OTLP registry. For example, add the following to the `dependencies` block of `build.gradle`: ```groovy theme={null} implementation 'io.micrometer:micrometer-registry-otlp' ``` Keep any registry the app already uses, such as StatsD, until you stop sending to it. 3. Define the OTLP registry as a bean. Spring Boot adds it beside your other registries and applies your common tags and distribution settings to it. For example, add the following class to your app: ```java theme={null} import io.micrometer.core.instrument.Clock; import io.micrometer.registry.otlp.OtlpConfig; import io.micrometer.registry.otlp.OtlpMeterRegistry; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.env.Environment; @Configuration(proxyBeanMethods = false) class OtlpMetricsConfig { @Bean OtlpMeterRegistry otlpMeterRegistry(Environment env, Clock clock) { OtlpConfig config = key -> env.getProperty("management.otlp.metrics.export." + key.substring("otlp.".length()).replaceAll("([A-Z])", "-$1").toLowerCase()); return new OtlpMeterRegistry(config, clock); } } ``` 4. Point the registry at the CXDOT Collector on the same node, and set delta temporality. For example, set these environment variables on the app's container: ```yaml theme={null} env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: MANAGEMENT_OTLP_METRICS_EXPORT_URL value: http://$(HOST_IP):4318/v1/metrics - name: MANAGEMENT_OTLP_METRICS_EXPORT_AGGREGATIONTEMPORALITY value: delta ``` If your services already set `HOST_IP` or `DD_AGENT_HOST` for StatsD, reuse it: both hold the node's address. The same settings work as `management.otlp.metrics.export.url` and `management.otlp.metrics.export.aggregation-temporality` in your app's configuration. 5. Set `OTEL_SERVICE_NAME` to the service's name, and any other resource attributes with `OTEL_RESOURCE_ATTRIBUTES`. The bean doesn't take the name from `spring.application.name`. 6. Optional: Turn on percentile histograms, so Palo Alto Networks Cortex XCOR can calculate percentiles across all of a service's instances. Set `management.metrics.distribution.percentiles-histogram.http.server.requests=true` for each timer you want percentiles for. If the service also sends to Datadog through the StatsD registry, wait until you turn StatsD off. The setting makes StatsD send timers as distributions. Datadog then stops receiving the timers' `.avg`, `.count`, `.max`, `.median`, and `.95percentile` metrics. Client-side percentiles, the `management.metrics.distribution.percentiles` setting, don't reach Cortex XCOR from a timer that also has service level objective (SLO) boundaries. The OTLP registry sends that timer's histogram instead of the percentiles. Use percentile histograms for those timers. 7. Optional: To get each timer's exact maximum, register a gauge in your app that reads it. For example, use a Micrometer `TimeGauge` over `Timer.max()`. Without a gauge, you can only estimate a timer's maximum from its percentile histogram buckets. ### Spring Boot 3 (3.0 to 3.5) Spring Boot 3 configures the OTLP registry for you. Spring Boot 3.1 and later include Micrometer 1.11 or greater and have an aggregation temporality setting. Spring Boot 3.0 includes Micrometer 1.10 and has no such setting, so it needs two extra steps, marked in the following list. To send metrics from a Spring Boot 3 service, follow these steps: 1. Spring Boot 3.0 only: Upgrade Micrometer to 1.11 or greater, since 3.0 includes 1.10. See [Upgrade Micrometer](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/micrometer.md#upgrade-micrometer). 2. Add the Micrometer OTLP registry. For example, add the following to the `dependencies` block of `build.gradle`: ```groovy theme={null} implementation 'io.micrometer:micrometer-registry-otlp' ``` Spring Boot manages the version. Keep any registry the app already uses, such as StatsD, until you stop sending to it. 3. Spring Boot 3.0 only: Define the registry's configuration as a bean. Spring Boot 3.0 has no aggregation temporality setting, and builds the registry from your bean instead of its own. For example, add the following class to your app: ```java theme={null} import io.micrometer.registry.otlp.OtlpConfig; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.env.Environment; @Configuration(proxyBeanMethods = false) class OtlpMetricsConfig { @Bean OtlpConfig otlpConfig(Environment env) { return key -> env.getProperty("management.otlp.metrics.export." + key.substring("otlp.".length()).replaceAll("([A-Z])", "-$1").toLowerCase()); } } ``` With the bean, also set `OTEL_SERVICE_NAME` to the service's name, and any other resource attributes with `OTEL_RESOURCE_ATTRIBUTES`. 4. Point the registry at the CXDOT Collector on the same node, and set delta temporality. For example, set these environment variables on the app's container: ```yaml theme={null} env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: MANAGEMENT_OTLP_METRICS_EXPORT_URL value: http://$(HOST_IP):4318/v1/metrics - name: MANAGEMENT_OTLP_METRICS_EXPORT_AGGREGATIONTEMPORALITY value: delta ``` If your services already set `HOST_IP` or `DD_AGENT_HOST` for StatsD, reuse it: both hold the node's address. The same settings work as `management.otlp.metrics.export.url` and `management.otlp.metrics.export.aggregation-temporality` in your app's configuration. 5. Optional: Turn on percentile histograms, so Cortex XCOR can calculate percentiles across all of a service's instances. Set `management.metrics.distribution.percentiles-histogram.http.server.requests=true` for each timer you want percentiles for. If the service also sends to Datadog through the StatsD registry, wait until you turn StatsD off. The setting makes StatsD send timers as distributions. Datadog then stops receiving the timers' `.avg`, `.count`, `.max`, `.median`, and `.95percentile` metrics. Client-side percentiles, the `management.metrics.distribution.percentiles` setting, don't reach Cortex XCOR from a timer that also has service level objective (SLO) boundaries. The OTLP registry sends that timer's histogram instead of the percentiles. Use percentile histograms for those timers. 6. Optional: To get each timer's exact maximum, register a gauge in your app that reads it. For example, use a Micrometer `TimeGauge` over `Timer.max()`. Without a gauge, you can only estimate a timer's maximum from its percentile histogram buckets. ### Spring Boot 4 (4.0 and 4.1) Spring Boot 4 includes Micrometer 1.16 or greater, and configures the OTLP registry once the app includes Spring Boot's OpenTelemetry module. To send metrics from a Spring Boot 4 service, follow these steps: 1. Add the Micrometer OTLP registry and Spring Boot's OpenTelemetry module. For example, add the following to the `dependencies` block of `build.gradle`: ```groovy theme={null} implementation 'io.micrometer:micrometer-registry-otlp' implementation 'org.springframework.boot:spring-boot-opentelemetry' ``` `spring-boot-starter-opentelemetry` also includes the module. Without it, Spring Boot 4 doesn't configure the OTLP registry, and logs nothing about it. Keep any registry the app already uses, such as StatsD, until you stop sending to it. 2. Point the registry at the CXDOT Collector on the same node, and set delta temporality. For example, set these environment variables on the app's container: ```yaml theme={null} env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: MANAGEMENT_OTLP_METRICS_EXPORT_URL value: http://$(HOST_IP):4318/v1/metrics - name: MANAGEMENT_OTLP_METRICS_EXPORT_AGGREGATIONTEMPORALITY value: delta ``` If your services already set `HOST_IP` or `DD_AGENT_HOST` for StatsD, reuse it: both hold the node's address. The same settings work as `management.otlp.metrics.export.url` and `management.otlp.metrics.export.aggregation-temporality` in your app's configuration. On Spring Boot 4.1, OpenTelemetry environment variables such as `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_METRIC_EXPORT_INTERVAL` override these settings. If the app's container sets them for another OpenTelemetry exporter, set `management.opentelemetry.map-environment-variables=false` so the registry keeps your settings. 3. Optional: Turn on percentile histograms, so Cortex XCOR can calculate percentiles across all of a service's instances. Set `management.metrics.distribution.percentiles-histogram.http.server.requests=true` for each timer you want percentiles for. If the service also sends to Datadog through the StatsD registry, wait until you turn StatsD off. The setting makes StatsD send timers as distributions. Datadog then stops receiving the timers' `.avg`, `.count`, `.max`, `.median`, and `.95percentile` metrics. Client-side percentiles, the `management.metrics.distribution.percentiles` setting, don't reach Cortex XCOR from a timer that also has service level objective (SLO) boundaries. The OTLP registry sends that timer's histogram instead of the percentiles. Use percentile histograms for those timers. 4. Optional: For each timer's exact maximum: * Spring Boot 4.1: Set `management.otlp.metrics.export.publish-max-gauge-for-histograms=true`. The registry then also sends each timer's maximum as a gauge, such as `http.server.requests.max`. * Spring Boot 4.0: Register a gauge in your app that reads the timer's maximum. For example, use a Micrometer `TimeGauge` over `Timer.max()`. ### Upgrade Micrometer Spring Boot 2 and 3.0 include a Micrometer version older than 1.11. To upgrade Micrometer, follow these steps: 1. Check the version. In Gradle, run `./gradlew dependencyInsight --dependency micrometer-core`. In Maven, run `mvn dependency:tree -Dincludes=io.micrometer`. 2. Set the version with the form that matches how your build manages dependency versions: * Gradle with the Spring dependency management plugin (`io.spring.dependency-management`). Add the following to `build.gradle`: ```groovy theme={null} ext['micrometer.version'] = '1.13.15' ``` * Gradle with Spring Boot's bill of materials imported as a platform. Add the following to the `dependencies` block. Use `enforcedPlatform` so it overrides the version Spring Boot's bill of materials sets: ```groovy theme={null} implementation enforcedPlatform('io.micrometer:micrometer-bom:1.13.15') ``` * Maven with `spring-boot-starter-parent` as the parent. Add the following to `pom.xml`: ```xml theme={null} 1.13.15 ``` * Maven importing `spring-boot-dependencies` as a bill of materials. Import the Micrometer bill of materials first, so its versions win: ```xml theme={null} io.micrometer micrometer-bom 1.13.15 pom import ``` 3. Check the version again. If it still shows the old version, the form doesn't match how your build manages versions. The build ignores a property it doesn't read and reports no error. ### Validate To validate collection, follow these steps: 1. In the [Live Telemetry Analyzer](https://docs-xcor.paloaltonetworks.com/investigate/analyze/telemetry-analyzer.md), filter on `__name__=jvm.memory.used`. Confirm that time series from your app arrive. 2. In [Metrics Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/explorer.md), run the following query: ```text theme={null} sum by ("service.name") (rate({"http.server.requests_count"}[5m])) ``` Confirm that the query reports a value for your app. ### Troubleshooting * No OTLP metrics from a Spring Boot 2 app: Confirm that the app defines the registry bean, and that Micrometer is 1.11 or greater. * No OTLP metrics from a Spring Boot 4 app: Confirm that the app includes `spring-boot-opentelemetry`. On Spring Boot 4.1, also read the URL in the app's `Failed to publish metrics to OTLP receiver` warnings. An `OTEL_EXPORTER_OTLP_ENDPOINT` set for another exporter, often on port `4317`, overrides the registry's URL. In that case, set `management.opentelemetry.map-environment-variables=false`. * Micrometer still reports the old version after you upgrade: The Gradle or Maven form doesn't match your build. For example, `ext['micrometer.version']` has no effect when your build imports Spring Boot's bill of materials as a Gradle platform. Use the form in [Upgrade Micrometer](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/micrometer.md#upgrade-micrometer) that matches your build. * Counters and timers look cumulative: The app isn't sending delta temporality. On Spring Boot 2, use the registry bean exactly as shown. A bean that maps only exact property names misses the `aggregation-temporality` setting. On Spring Boot 3.0, Spring Boot ignores the setting unless the app defines the `OtlpConfig` bean. Before Spring Boot 4.1, `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` doesn't replace the setting. * Timer maximums are missing: Cortex XCOR keeps a histogram's count, sum, and buckets, but not its maximum, whatever the temporality. Send the maximum as a gauge, as the optional maximum step for your Spring Boot line describes. * Metrics carry `service.name="unknown_service"`: On Spring Boot 2 and 3.0, set `OTEL_SERVICE_NAME` on the app. * Request counts are double what you expect: The app's metrics reach the CXDOT Collector by two paths, for example over OTLP and as DogStatsD forwarded by another agent. Send each app's metrics by one path. ## Turn off your previous registry Keep your existing registry until the service's dashboards and monitors work with the OTLP metrics. Their queries don't change on their own: the OTLP registry sends different metric names and shapes, so rewrite each query with the forms in Query Micrometer metrics. While both registries send, expect the old and new charts to agree in level but not point for point, because each registry samples gauges at its own moments. Queries on Datadog's `.median` and `.95percentile` timer metrics, and on client-side percentiles of a timer with SLO boundaries, need percentile histograms on that timer. Without them, the OTLP registry sends nothing to calculate those values from. With the StatsD registry, turning on percentile histograms also changes what Datadog receives, so turn them on as part of the cutover. When the dashboards and monitors work, turn off the old registry and restart the service: | Registry | Spring Boot 3 and 4 | Spring Boot 2 | | - | - | - | | StatsD | `management.statsd.metrics.export.enabled=false` | `management.metrics.export.statsd.enabled=false` | | Datadog | `management.datadog.metrics.export.enabled=false` | `management.metrics.export.datadog.enabled=false` | ## Query Micrometer metrics Cortex XCOR keeps Micrometer's dotted metric names, so queries quote them. Your Micrometer tags carry over as labels of the same name. The following table lists queries for common questions about a service, and the timer setting each query needs: | You want | Query | Needs | | - | - | - | | Request rate | `sum(rate({"http.server.requests_count"}[5m]))` | none | | Average latency | `sum(rate({"http.server.requests_sum"}[5m])) / sum(rate({"http.server.requests_count"}[5m]))` | none | | p95 latency | `histogram_quantile(0.95, sum by (le) (rate({"http.server.requests_bucket"}[5m])))` | Percentile histograms | | Share of requests under 200 ms | `histogram_fraction(0, 200, sum by (le) (rate({"http.server.requests_bucket"}[5m])))` | An SLO boundary at 200 ms, or percentile histograms | | Maximum latency, estimated | `histogram_quantile(1, sum by (le) (rate({"http.server.requests_bucket"}[5m])))` | Percentile histograms | | Maximum latency, exact | `max({"http.server.requests.max"})` | The maximum gauge, on Spring Boot 4.1 | | A gauge | `{"jvm.memory.used"}` | none | | A counter | `rate({"logback.events"}[5m])` | none | Cortex XCOR stores a histogram's count, sum, and buckets, but not its maximum. Without a gauge, a timer's maximum is an estimate: the upper bound of the bucket that holds the slowest request. On Spring Boot 4.1, set `management.otlp.metrics.export.publish-max-gauge-for-histograms=true`. The registry then also sends each timer's maximum as a gauge named after the timer, such as `{"http.server.requests.max"}`. On earlier Spring Boot lines, register a gauge in your app that reads the timer's maximum. For example, use a Micrometer `TimeGauge` over `Timer.max()`. ## Related topics - [CXDOT Collector integrations](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.