Skip to main content
The Consul integration requires CXDOT Collector 1.4.0 or greater. Consul is a distributed system for connecting and configuring applications across dynamic infrastructure. Use the Consul CXDOT Collector integration to collect cluster and catalog health metrics from Consul agents in your environment.

Supported telemetry types

The Consul integration supports these telemetry types:

Prerequisites

Meet these requirements before configuring the integration:
  • Make each Consul agent HTTP API reachable from the Collector tier that scrapes it (node-local agents on the node Collector, static targets on the cluster Collector).
  • When Consul access control lists are enabled, provide a token with read access to the agent, catalog, and health endpoints.

Configure

The Consul integration is enabled by default. You can configure targets either by annotating agent pods in your Kubernetes cluster or by listing static endpoints in your Helm values. To configure the Consul integration, follow these steps:
  1. To collect from Consul agent pods in Kubernetes, add a CXDOT integration annotation to the pod template. The annotation key must end with the container name, and its value can reference an access control list token stored in a Kubernetes Secret in the same namespace. For example, add the following annotation to your Consul agents:
    The Collector replaces ${target.host} with the pod IP address, ${target.port.http} with the container port named http, and ${k8s_secret:consul-monitoring/token} with the token key of the consul-monitoring Secret. To allow the Collector to read the Secret, add the pod’s namespace to apiServer.metadataServer.secretAllowlist in the Helm values for your CXDOT Collector. The Collector also discovers Consul agent pods that carry the app: consul label and expose the HTTP API on port 8500, which HashiCorp’s official chart sets by default. A CXDOT integration annotation takes precedence over the label for its container. To stop collecting from a container, set consul: {enabled: false} in the annotation. For more information, see autodiscovery.
  2. Optional: To collect from Consul agents outside your Kubernetes cluster, configure static targets: list reachable host:port values or full URLs in endpoints. A nonempty list disables discovery for that instance. The Collector scrapes only those agents. When access control lists are enabled, set acl_token. For example:
  3. Optional: To combine annotated pods and static targets, configure separate instances. The unnamed instance discovers pods, and the named instance scrapes the static targets:
  4. Optional: To collect Raft and Serf metrics, send Consul’s own telemetry to the DogStatsD listener in the CXDOT Collector. The Consul integration doesn’t collect these metrics. Consul agents push them through a telemetry sink, and the Datadog integration receives them. The Datadog integration runs by default on the node Collector. Consul 2.0.2 through 2.0.4 don’t send Raft or Serf metrics to any telemetry sink. By default, HashiCorp’s chart installs the Consul version that matches the chart version, so chart versions 2.0.2 through 2.0.4 install an affected Consul version. To collect these metrics, upgrade to a later Consul release, which restores them. Until one is available, set global.image in the Helm values file for your Consul release to Consul 2.0.1, which also sends them. Set telemetry.dogstatsd_addr in each agent’s configuration to the node IP address and the DogStatsD listener port, which is 9125 by default, not 8125. HashiCorp’s official chart replaces the HOST_IP placeholder in server.extraConfig with the IP address of each server’s node. For example, add the following to the Helm values file for your Consul release:
    Consul sends DogStatsD metrics to one address only, so telemetry.dogstatsd_addr replaces the DogStatsD destination that your Consul agents already use rather than adding a second one. If a Datadog Agent already receives Consul’s DogStatsD metrics and forwards metrics to the CXDOT Collector, leave Consul pointed at the Agent and skip this step. Otherwise, set global.metrics.datadog.dogstatsd.enabled to false in the Helm values file for your Consul release. While that value is true, HashiCorp’s chart rejects dogstatsd_addr in server.extraConfig. The node IP address reaches the node Collector only while nodeCollector.hostNetwork is true, the CXDOT Collector chart’s default. If you set it to false, use the address of the node Collector Service, cxdot-node-collector.<namespace>.svc, instead of HOST_IP, where <namespace> is the namespace of your CXDOT Collector release. Only Consul servers report Raft metrics. If you run Consul client agents (client.enabled: true), add the same telemetry block to client.extraConfig to also collect their Serf metrics:
    By default, Consul sends all of its telemetry to the DogStatsD sink, not only Raft and Serf metrics. That includes Go runtime, gossip, RPC, and HTTP metrics. To send only Raft and Serf metrics, allow the consul.raft and consul.serf prefixes with Consul’s telemetry.prefix_filter setting, and set telemetry.filter_default to false to drop everything else. The filter applies to every Consul telemetry sink, not only DogStatsD. HashiCorp’s chart rejects prefix_filter and disable_hostname in server.extraConfig, so set the server filter through global.metrics instead. Filter entries match the start of each metric name, and Consul adds the agent’s host name to gauge names unless disable_hostname is true, so also set global.metrics.disableAgentHostName to true. Setting global.metrics.enabled also turns on the chart’s other metrics features, such as Prometheus scrape annotations on Consul pods. For example:
    Client agents report Serf metrics but not Raft metrics. For client agents, add "prefix_filter": ["+consul.serf"], "filter_default": false, and "disable_hostname": true to the telemetry block in client.extraConfig. In Consul’s own configuration, each allowed prefix starts with +. Use only the DogStatsD sink. Don’t point Consul’s statsd_address or statsite_address sinks at the CXDOT Collector: those sinks add label values to metric names, such as consul.serf.member.join.lan, and send no tags.
  5. Optional: To attach pod metadata to Consul’s telemetry, set the DD_ENTITY_ID environment variable on the Consul pods. Without it, Consul’s metrics identify only the node that sent them, and a server and a client agent on the same node report the same time series. With it, each metric arrives with the Kubernetes metadata of the pod that sent it. The CXDOT Collector admission controller sets DD_ENTITY_ID on pods that carry the admission.cxdot.paloaltonetworks.com/enabled: "true" label. Set apiServer.admissionController.enabled to true in the Helm values file for your CXDOT Collector, then add the label to the Consul pods in the Helm values file for your Consul release:
    If the Datadog Cluster Agent admission controller already sets DD_ENTITY_ID on the Consul pods, such as during a migration, skip this step.

Validate

  1. In the Live Telemetry Analyzer, add the following label filters:
    • Set Label to __name__ and Value to cxdot.integration.target.health.
    • Set Label to cxdot.integration.name and Value to consul.
    Group the results by cxdot.integration.target, and confirm that the metric reports 1 for each target.
  2. In Metrics Explorer, run the following query:
    Confirm that each agent you expect appears. Only agents where consul.mode="leader" also emit catalog metrics for the data center.
  3. Optional: If you send Consul Raft and Serf metrics to the CXDOT Collector, run the following query in Metrics Explorer after you write a key to the Consul key-value store:
    Confirm that the query returns a result. Only the Raft leader reports this metric.

Troubleshooting

  • Consul metrics don’t appear and Collector logs report HTTP 403 responses: Confirm that the ACL token has read access to the agent, catalog, and health endpoints.
  • Catalog metrics are missing for an agent, but consul.peers appears: Only the Raft leader for a data center reports catalog metrics. Other agents report peer count only.
  • Raft and Serf metrics don’t appear: Confirm that the Datadog integration and its DogStatsD intake are enabled, and that the port in telemetry.dogstatsd_addr matches the DogStatsD listener port, 9125 by default. The CXDOT Collector doesn’t listen on port 8125 by default.
  • Raft and Serf metrics don’t appear, but other Consul telemetry such as Go runtime metrics does: Consul 2.0.2 through 2.0.4 don’t send Raft or Serf metrics to any telemetry sink, so upgrade to a later release, or use 2.0.1 until one is available.

Disable the Consul integration

Deleting every consul block restores the default unnamed instance. To remove custom settings and stop collection, retain a disabled configuration:

Configuration reference

Configure one Consul integration instance with the following settings. In Helm values, place these settings under collection.cxdot.integrations.consul. In a Collector configuration file, place them under cxdot.integrations.consul.

Optional settings

  • enabled Type: boolean. Optional. Default: true. Whether to enable this Consul integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it.
  • endpoints Type: array of object. Optional. Default: []. Consul targets. Specify a target as host:port to use HTTP. Specify a full HTTP or HTTPS URL to control the scheme.
  • endpoints[].endpoint Type: string. Required. Consul target as either host:port or a full HTTP or HTTPS URL.
  • acl_token Type: string. Optional. Consul access control list token sent in the X-Consul-Token header. Provide a token with read access to the agent, catalog, and health endpoints when access control lists are enabled. The Collector masks this sensitive value in diagnostic output, logs, and errors. Discovery annotations can provide a target-specific token.
  • collection_interval Type: duration. Optional. Default: 15s. How often the Collector collects metrics from each Consul target.
  • timeout Type: duration. Optional. Default: 15s. Maximum time allowed to collect metrics from one Consul target. The value must not exceed collection_interval.
  • max_services Type: integer. Optional. Default: 50. Minimum: 1. Cap on how many catalog services one collection interrogates. Each service costs one health-endpoint request, so an unbounded catalog would turn a single collection into thousands of requests against the agent. Services are taken in name order once the catalog exceeds the cap, so the truncated set stays the same between collections rather than shifting with the agent’s response ordering.
  • network_latency_checks Type: boolean. Optional. Default: false. Whether the Consul leader reports consul.net.node.latency, each node’s estimated round-trip times to the other nodes in its network segment, computed from network coordinates. Reading coordinates requires the node:read access control list permission for every node. For a target discovered through a Datadog annotation, the annotation’s network_latency_checks field applies instead.
  • tls Type: object. Optional. Transport layer security (TLS) settings for the HTTPS targets in endpoints. Certificate verification is enabled unless insecure_skip_verify is true. These settings don’t apply to HTTP targets. Discovery annotations can provide target-specific TLS settings.
  • tls.ca_file Type: string. Optional. Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty uses system root CA. (optional)
  • tls.ca_pem Type: string. Optional. In memory PEM encoded cert. (optional)
  • tls.cert_file Type: string. Optional. Path to the TLS cert to use for TLS required connections. (optional)
  • tls.cert_pem Type: string. Optional. In memory PEM encoded TLS cert to use for TLS required connections. (optional)
  • tls.cipher_suites Type: array of string. Optional. CipherSuites is a list of TLS cipher suites that the TLS transport can use. If left blank, a safe default list is used. See https://go.dev/src/crypto/tls/cipher_suites.go for a list of supported cipher suites.
  • tls.curve_preferences Type: array of string. Optional. contains the elliptic curves that will be used in an ECDHE handshake, in preference order Defaults to empty list and “crypto/tls” defaults are used, internally.
  • tls.include_system_ca_certs_pool Type: boolean. Optional. If true, load system CA certificates pool in addition to the certificates configured in this struct.
  • tls.insecure Type: boolean. Optional. In gRPC and HTTP when set to true, this is used to disable the client transport security. See https://godoc.org/google.golang.org/grpc#WithInsecure for gRPC. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional, default false)
  • tls.insecure_skip_verify Type: boolean. Optional. InsecureSkipVerify will enable TLS but not verify the certificate.
  • tls.key_file Type: string. Optional. Path to the TLS key to use for TLS required connections. (optional)
  • tls.key_pem Type: string. Optional. In memory PEM encoded TLS key to use for TLS required connections. (optional)
  • tls.max_version Type: string. Optional. MaxVersion sets the maximum TLS version that is acceptable. If not set, refer to crypto/tls for defaults. (optional)
  • tls.min_version Type: string. Optional. MinVersion sets the minimum TLS version that is acceptable. If not set, TLS 1.2 will be used. (optional)
  • tls.reload_interval Type: duration. Optional. ReloadInterval specifies the duration after which the certificate will be reloaded If not set, it will never be reloaded (optional)
  • tls.server_name_override Type: string. Optional. ServerName requested by client for virtual hosting. This sets the ServerName in the TLSConfig. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional)
  • tls.tpm Type: object. Optional. Trusted platform module configuration
  • tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
  • tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
  • tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
  • tls.tpm.path Type: string. Optional. The path to the TPM device or Unix domain socket. For instance /dev/tpm0 or /dev/tpmrm0.