Micrometer

Micrometer is the metrics instrumentation library Spring Boot uses for Java Virtual Machine (JVM) apps. Use Micrometer’s OTLP registry to send your services’ metrics to the CXDOT Collector running on the same node. For the Micrometer integration, there is nothing to configure in the CXDOT Collector. Each service pushes its own metrics over OTLP to the OTLP integration, which the CXDOT Collector runs by default. Because the metrics arrive through the OTLP integration, they carry cxdot.integration.name=otlp, not micrometer. The Micrometer integration stays listed as available after its metrics arrive, and it reports no health status. Adding the OTLP registry is additive. Spring Boot sends every metric to every registry the app includes, so your existing StatsD or Datadog registry keeps sending what it sends today. The CXDOT Collector supports Micrometer 1.11 or greater.

​

Supported telemetry types

The CXDOT Collector supports these telemetry types from Micrometer:

​

Prerequisites

Sending Micrometer metrics over OTLP has the following prerequisites:

• Find each service’s Spring Boot version: look for spring-boot in its Gradle or Maven dependencies. The setup differs by Spring Boot line.
• Micrometer 1.11 or greater, the first version that sends delta temporality. Spring Boot 3.1 and later include it. Spring Boot 2 and 3.0 include an older version. See Upgrade Micrometer. Upgrading Micrometer doesn’t require upgrading Spring Boot.
• Run the CXDOT Collector on each node with host networking, the Helm chart’s default. OTLP ingest is on by default, so apps send to their node’s IP address on port 4318.

​

Configure

Follow the section for each service’s Spring Boot line, then confirm collection with the steps in Validate.

​

Spring Boot 2 (2.5 to 2.7)

Spring Boot 2 includes Micrometer 1.7 to 1.9 and doesn’t configure the OTLP registry itself. To send metrics from a Spring Boot 2 service, follow these steps:

1. Upgrade Micrometer to 1.11 or greater. See Upgrade Micrometer.
2. Add the Micrometer OTLP registry. For example, add the following to the dependencies block of build.gradle:
   implementation 'io.micrometer:micrometer-registry-otlp'
   
   Keep any registry the app already uses, such as StatsD, until you stop sending to it.
3. Define the OTLP registry as a bean. Spring Boot adds it beside your other registries and applies your common tags and distribution settings to it. For example, add the following class to your app:
   import io.micrometer.core.instrument.Clock;
   import io.micrometer.registry.otlp.OtlpConfig;
   import io.micrometer.registry.otlp.OtlpMeterRegistry;
   import org.springframework.context.annotation.Bean;
   import org.springframework.context.annotation.Configuration;
   import org.springframework.core.env.Environment;
   
   @Configuration(proxyBeanMethods = false)
   class OtlpMetricsConfig {
       @Bean
       OtlpMeterRegistry otlpMeterRegistry(Environment env, Clock clock) {
           OtlpConfig config = key -> env.getProperty("management.otlp.metrics.export."
                   + key.substring("otlp.".length()).replaceAll("([A-Z])", "-$1").toLowerCase());
           return new OtlpMeterRegistry(config, clock);
       }
   }
   
4. Point the registry at the CXDOT Collector on the same node, and set delta temporality. For example, set these environment variables on the app’s container:
   env:
     - name: HOST_IP
       valueFrom:
         fieldRef:
           fieldPath: status.hostIP
     - name: MANAGEMENT_OTLP_METRICS_EXPORT_URL
       value: http://$(HOST_IP):4318/v1/metrics
     - name: MANAGEMENT_OTLP_METRICS_EXPORT_AGGREGATIONTEMPORALITY
       value: delta
   
   If your services already set HOST_IP or DD_AGENT_HOST for StatsD, reuse it: both hold the node’s address. The same settings work as management.otlp.metrics.export.url and management.otlp.metrics.export.aggregation-temporality in your app’s configuration.
5. Set OTEL_SERVICE_NAME to the service’s name, and any other resource attributes with OTEL_RESOURCE_ATTRIBUTES. The bean doesn’t take the name from spring.application.name.
6. Optional: Turn on percentile histograms, so Palo Alto Networks Cortex XCOR can calculate percentiles across all of a service’s instances. Set management.metrics.distribution.percentiles-histogram.http.server.requests=true for each timer you want percentiles for. If the service also sends to Datadog through the StatsD registry, wait until you turn StatsD off. The setting makes StatsD send timers as distributions. Datadog then stops receiving the timers’ .avg, .count, .max, .median, and .95percentile metrics. Client-side percentiles, the management.metrics.distribution.percentiles setting, don’t reach Cortex XCOR from a timer that also has service level objective (SLO) boundaries. The OTLP registry sends that timer’s histogram instead of the percentiles. Use percentile histograms for those timers.
7. Optional: To get each timer’s exact maximum, register a gauge in your app that reads it. For example, use a Micrometer TimeGauge over Timer.max(). Without a gauge, you can only estimate a timer’s maximum from its percentile histogram buckets.

​

Spring Boot 3 (3.0 to 3.5)

Spring Boot 3 configures the OTLP registry for you. Spring Boot 3.1 and later include Micrometer 1.11 or greater and have an aggregation temporality setting. Spring Boot 3.0 includes Micrometer 1.10 and has no such setting, so it needs two extra steps, marked in the following list. To send metrics from a Spring Boot 3 service, follow these steps:

1. Spring Boot 3.0 only: Upgrade Micrometer to 1.11 or greater, since 3.0 includes 1.10. See Upgrade Micrometer.
2. Add the Micrometer OTLP registry. For example, add the following to the dependencies block of build.gradle:
   implementation 'io.micrometer:micrometer-registry-otlp'
   
   Spring Boot manages the version. Keep any registry the app already uses, such as StatsD, until you stop sending to it.
3. Spring Boot 3.0 only: Define the registry’s configuration as a bean. Spring Boot 3.0 has no aggregation temporality setting, and builds the registry from your bean instead of its own. For example, add the following class to your app:
   import io.micrometer.registry.otlp.OtlpConfig;
   import org.springframework.context.annotation.Bean;
   import org.springframework.context.annotation.Configuration;
   import org.springframework.core.env.Environment;
   
   @Configuration(proxyBeanMethods = false)
   class OtlpMetricsConfig {
       @Bean
       OtlpConfig otlpConfig(Environment env) {
           return key -> env.getProperty("management.otlp.metrics.export."
                   + key.substring("otlp.".length()).replaceAll("([A-Z])", "-$1").toLowerCase());
       }
   }
   
   With the bean, also set OTEL_SERVICE_NAME to the service’s name, and any other resource attributes with OTEL_RESOURCE_ATTRIBUTES.
4. Point the registry at the CXDOT Collector on the same node, and set delta temporality. For example, set these environment variables on the app’s container:
   env:
     - name: HOST_IP
       valueFrom:
         fieldRef:
           fieldPath: status.hostIP
     - name: MANAGEMENT_OTLP_METRICS_EXPORT_URL
       value: http://$(HOST_IP):4318/v1/metrics
     - name: MANAGEMENT_OTLP_METRICS_EXPORT_AGGREGATIONTEMPORALITY
       value: delta
   
   If your services already set HOST_IP or DD_AGENT_HOST for StatsD, reuse it: both hold the node’s address. The same settings work as management.otlp.metrics.export.url and management.otlp.metrics.export.aggregation-temporality in your app’s configuration.
5. Optional: Turn on percentile histograms, so Cortex XCOR can calculate percentiles across all of a service’s instances. Set management.metrics.distribution.percentiles-histogram.http.server.requests=true for each timer you want percentiles for. If the service also sends to Datadog through the StatsD registry, wait until you turn StatsD off. The setting makes StatsD send timers as distributions. Datadog then stops receiving the timers’ .avg, .count, .max, .median, and .95percentile metrics. Client-side percentiles, the management.metrics.distribution.percentiles setting, don’t reach Cortex XCOR from a timer that also has service level objective (SLO) boundaries. The OTLP registry sends that timer’s histogram instead of the percentiles. Use percentile histograms for those timers.
6. Optional: To get each timer’s exact maximum, register a gauge in your app that reads it. For example, use a Micrometer TimeGauge over Timer.max(). Without a gauge, you can only estimate a timer’s maximum from its percentile histogram buckets.

​

Spring Boot 4 (4.0 and 4.1)

Spring Boot 4 includes Micrometer 1.16 or greater, and configures the OTLP registry once the app includes Spring Boot’s OpenTelemetry module. To send metrics from a Spring Boot 4 service, follow these steps:

1. Add the Micrometer OTLP registry and Spring Boot’s OpenTelemetry module. For example, add the following to the dependencies block of build.gradle:
   implementation 'io.micrometer:micrometer-registry-otlp'
   implementation 'org.springframework.boot:spring-boot-opentelemetry'
   
   spring-boot-starter-opentelemetry also includes the module. Without it, Spring Boot 4 doesn’t configure the OTLP registry, and logs nothing about it. Keep any registry the app already uses, such as StatsD, until you stop sending to it.
2. Point the registry at the CXDOT Collector on the same node, and set delta temporality. For example, set these environment variables on the app’s container:
   env:
     - name: HOST_IP
       valueFrom:
         fieldRef:
           fieldPath: status.hostIP
     - name: MANAGEMENT_OTLP_METRICS_EXPORT_URL
       value: http://$(HOST_IP):4318/v1/metrics
     - name: MANAGEMENT_OTLP_METRICS_EXPORT_AGGREGATIONTEMPORALITY
       value: delta
   
   If your services already set HOST_IP or DD_AGENT_HOST for StatsD, reuse it: both hold the node’s address. The same settings work as management.otlp.metrics.export.url and management.otlp.metrics.export.aggregation-temporality in your app’s configuration. On Spring Boot 4.1, OpenTelemetry environment variables such as OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_METRIC_EXPORT_INTERVAL override these settings. If the app’s container sets them for another OpenTelemetry exporter, set management.opentelemetry.map-environment-variables=false so the registry keeps your settings.
3. Optional: Turn on percentile histograms, so Cortex XCOR can calculate percentiles across all of a service’s instances. Set management.metrics.distribution.percentiles-histogram.http.server.requests=true for each timer you want percentiles for. If the service also sends to Datadog through the StatsD registry, wait until you turn StatsD off. The setting makes StatsD send timers as distributions. Datadog then stops receiving the timers’ .avg, .count, .max, .median, and .95percentile metrics. Client-side percentiles, the management.metrics.distribution.percentiles setting, don’t reach Cortex XCOR from a timer that also has service level objective (SLO) boundaries. The OTLP registry sends that timer’s histogram instead of the percentiles. Use percentile histograms for those timers.
4. Optional: For each timer’s exact maximum:
   • Spring Boot 4.1: Set management.otlp.metrics.export.publish-max-gauge-for-histograms=true. The registry then also sends each timer’s maximum as a gauge, such as http.server.requests.max.
   • Spring Boot 4.0: Register a gauge in your app that reads the timer’s maximum. For example, use a Micrometer TimeGauge over Timer.max().

​

Upgrade Micrometer

Spring Boot 2 and 3.0 include a Micrometer version older than 1.11. To upgrade Micrometer, follow these steps:

1. Check the version. In Gradle, run ./gradlew dependencyInsight --dependency micrometer-core. In Maven, run mvn dependency:tree -Dincludes=io.micrometer.
2. Set the version with the form that matches how your build manages dependency versions:
   • Gradle with the Spring dependency management plugin (io.spring.dependency-management). Add the following to build.gradle:
     ext['micrometer.version'] = '1.13.15'
     
   • Gradle with Spring Boot’s bill of materials imported as a platform. Add the following to the dependencies block. Use enforcedPlatform so it overrides the version Spring Boot’s bill of materials sets:
     implementation enforcedPlatform('io.micrometer:micrometer-bom:1.13.15')
     
   • Maven with spring-boot-starter-parent as the parent. Add the following to pom.xml:
     <properties>
       <micrometer.version>1.13.15</micrometer.version>
     </properties>
     
   • Maven importing spring-boot-dependencies as a bill of materials. Import the Micrometer bill of materials first, so its versions win:
     <dependencyManagement>
       <dependencies>
         <dependency>
           <groupId>io.micrometer</groupId>
           <artifactId>micrometer-bom</artifactId>
           <version>1.13.15</version>
           <type>pom</type>
           <scope>import</scope>
         </dependency>
         <!-- then spring-boot-dependencies -->
       </dependencies>
     </dependencyManagement>
     
3. Check the version again. If it still shows the old version, the form doesn’t match how your build manages versions. The build ignores a property it doesn’t read and reports no error.

​

Validate

To validate collection, follow these steps:

1. In the Live Telemetry Analyzer, filter on __name__=jvm.memory.used. Confirm that time series from your app arrive.
2. In Metrics Explorer, run the following query:
   sum by ("service.name") (rate({"http.server.requests_count"}[5m]))
   
   Confirm that the query reports a value for your app.

​

Troubleshooting

• No OTLP metrics from a Spring Boot 2 app: Confirm that the app defines the registry bean, and that Micrometer is 1.11 or greater.
• No OTLP metrics from a Spring Boot 4 app: Confirm that the app includes spring-boot-opentelemetry. On Spring Boot 4.1, also read the URL in the app’s Failed to publish metrics to OTLP receiver warnings. An OTEL_EXPORTER_OTLP_ENDPOINT set for another exporter, often on port 4317, overrides the registry’s URL. In that case, set management.opentelemetry.map-environment-variables=false.
• Micrometer still reports the old version after you upgrade: The Gradle or Maven form doesn’t match your build. For example, ext['micrometer.version'] has no effect when your build imports Spring Boot’s bill of materials as a Gradle platform. Use the form in Upgrade Micrometer that matches your build.
• Counters and timers look cumulative: The app isn’t sending delta temporality. On Spring Boot 2, use the registry bean exactly as shown. A bean that maps only exact property names misses the aggregation-temporality setting. On Spring Boot 3.0, Spring Boot ignores the setting unless the app defines the OtlpConfig bean. Before Spring Boot 4.1, OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE doesn’t replace the setting.
• Timer maximums are missing: Cortex XCOR keeps a histogram’s count, sum, and buckets, but not its maximum, whatever the temporality. Send the maximum as a gauge, as the optional maximum step for your Spring Boot line describes.
• Metrics carry service.name="unknown_service": On Spring Boot 2 and 3.0, set OTEL_SERVICE_NAME on the app.
• Request counts are double what you expect: The app’s metrics reach the CXDOT Collector by two paths, for example over OTLP and as DogStatsD forwarded by another agent. Send each app’s metrics by one path.

​

Turn off your previous registry

Keep your existing registry until the service’s dashboards and monitors work with the OTLP metrics. Their queries don’t change on their own: the OTLP registry sends different metric names and shapes, so rewrite each query with the forms in Query Micrometer metrics. While both registries send, expect the old and new charts to agree in level but not point for point, because each registry samples gauges at its own moments. Queries on Datadog’s .median and .95percentile timer metrics, and on client-side percentiles of a timer with SLO boundaries, need percentile histograms on that timer. Without them, the OTLP registry sends nothing to calculate those values from. With the StatsD registry, turning on percentile histograms also changes what Datadog receives, so turn them on as part of the cutover. When the dashboards and monitors work, turn off the old registry and restart the service:

​

Query Micrometer metrics

Cortex XCOR keeps Micrometer’s dotted metric names, so queries quote them. Your Micrometer tags carry over as labels of the same name. The following table lists queries for common questions about a service, and the timer setting each query needs: Cortex XCOR stores a histogram’s count, sum, and buckets, but not its maximum. Without a gauge, a timer’s maximum is an estimate: the upper bound of the bucket that holds the slowest request. On Spring Boot 4.1, set management.otlp.metrics.export.publish-max-gauge-for-histograms=true. The registry then also sends each timer’s maximum as a gauge named after the timer, such as {"http.server.requests.max"}. On earlier Spring Boot lines, register a gauge in your app that reads the timer’s maximum. For example, use a Micrometer TimeGauge over Timer.max().