Supported telemetry types
The Consul integration supports these telemetry types:Prerequisites
Meet these requirements before configuring the integration:- Make each Consul agent HTTP API reachable from the Collector tier that scrapes it (node-local agents on the node Collector, static targets on the cluster Collector).
- When Consul access control lists are enabled, provide a token with read access to the agent, catalog, and health endpoints.
Configure
The Consul integration is enabled by default. You can configure targets either by annotating agent pods in your Kubernetes cluster or by listing static endpoints in your Helm values. To configure the Consul integration, follow these steps:-
To collect from Consul agent pods in Kubernetes, add a CXDOT integration annotation
to the pod template. The annotation key must end with the container name, and its
value can reference an access control list token stored in a Kubernetes Secret in the
same namespace. For example, add the following annotation to your Consul agents:
The Collector replaces
${target.host}with the pod IP address,${target.port.http}with the container port namedhttp, and${k8s_secret:consul-monitoring/token}with thetokenkey of theconsul-monitoringSecret. To allow the Collector to read the Secret, add the pod’s namespace toapiServer.metadataServer.secretAllowlistin the Helm values for your CXDOT Collector. The Collector also discovers Consul agent pods that carry theapp: consullabel and expose the HTTP API on port8500, which HashiCorp’s official chart sets by default. A CXDOT integration annotation takes precedence over the label for its container. To stop collecting from a container, setconsul: {enabled: false}in the annotation. For more information, see autodiscovery. -
Optional: To collect from Consul agents outside your Kubernetes cluster, configure
static targets: list reachable
host:portvalues or full URLs inendpoints. A nonempty list disables discovery for that instance. The Collector scrapes only those agents. When access control lists are enabled, setacl_token. For example: -
Optional: To combine annotated pods and static targets, configure separate
instances. The unnamed instance discovers pods, and the named instance
scrapes the static targets:
-
Optional: To collect Raft and Serf metrics, send Consul’s own telemetry to
the DogStatsD listener in the CXDOT Collector. The Consul integration doesn’t
collect these metrics. Consul agents push them through a telemetry sink, and
the Datadog integration receives them. The Datadog integration runs by
default on the node Collector.
Consul 2.0.2 through 2.0.4 don’t send Raft or Serf metrics to any telemetry
sink. By default, HashiCorp’s chart installs the Consul version that matches
the chart version, so chart versions 2.0.2 through 2.0.4 install an affected
Consul version. To collect these metrics, upgrade to a later Consul release,
which restores them. Until one is available, set
global.imagein the Helm values file for your Consul release to Consul 2.0.1, which also sends them. Settelemetry.dogstatsd_addrin each agent’s configuration to the node IP address and the DogStatsD listener port, which is9125by default, not8125. HashiCorp’s official chart replaces theHOST_IPplaceholder inserver.extraConfigwith the IP address of each server’s node. For example, add the following to the Helm values file for your Consul release:Consul sends DogStatsD metrics to one address only, sotelemetry.dogstatsd_addrreplaces the DogStatsD destination that your Consul agents already use rather than adding a second one. If a Datadog Agent already receives Consul’s DogStatsD metrics and forwards metrics to the CXDOT Collector, leave Consul pointed at the Agent and skip this step. Otherwise, setglobal.metrics.datadog.dogstatsd.enabledtofalsein the Helm values file for your Consul release. While that value istrue, HashiCorp’s chart rejectsdogstatsd_addrinserver.extraConfig. The node IP address reaches the node Collector only whilenodeCollector.hostNetworkistrue, the CXDOT Collector chart’s default. If you set it tofalse, use the address of the node Collector Service,cxdot-node-collector.<namespace>.svc, instead ofHOST_IP, where<namespace>is the namespace of your CXDOT Collector release. Only Consul servers report Raft metrics. If you run Consul client agents (client.enabled: true), add the sametelemetryblock toclient.extraConfigto also collect their Serf metrics:By default, Consul sends all of its telemetry to the DogStatsD sink, not only Raft and Serf metrics. That includes Go runtime, gossip, RPC, and HTTP metrics. To send only Raft and Serf metrics, allow theconsul.raftandconsul.serfprefixes with Consul’stelemetry.prefix_filtersetting, and settelemetry.filter_defaulttofalseto drop everything else. The filter applies to every Consul telemetry sink, not only DogStatsD. HashiCorp’s chart rejectsprefix_filteranddisable_hostnameinserver.extraConfig, so set the server filter throughglobal.metricsinstead. Filter entries match the start of each metric name, and Consul adds the agent’s host name to gauge names unlessdisable_hostnameistrue, so also setglobal.metrics.disableAgentHostNametotrue. Settingglobal.metrics.enabledalso turns on the chart’s other metrics features, such as Prometheus scrape annotations on Consul pods. For example:Client agents report Serf metrics but not Raft metrics. For client agents, add"prefix_filter": ["+consul.serf"],"filter_default": false, and"disable_hostname": trueto thetelemetryblock inclient.extraConfig. In Consul’s own configuration, each allowed prefix starts with+. Use only the DogStatsD sink. Don’t point Consul’sstatsd_addressorstatsite_addresssinks at the CXDOT Collector: those sinks add label values to metric names, such asconsul.serf.member.join.lan, and send no tags. -
Optional: To attach pod metadata to Consul’s telemetry, set the
DD_ENTITY_IDenvironment variable on the Consul pods. Without it, Consul’s metrics identify only the node that sent them, and a server and a client agent on the same node report the same time series. With it, each metric arrives with the Kubernetes metadata of the pod that sent it. The CXDOT Collector admission controller setsDD_ENTITY_IDon pods that carry theadmission.cxdot.paloaltonetworks.com/enabled: "true"label. SetapiServer.admissionController.enabledtotruein the Helm values file for your CXDOT Collector, then add the label to the Consul pods in the Helm values file for your Consul release:If the Datadog Cluster Agent admission controller already setsDD_ENTITY_IDon the Consul pods, such as during a migration, skip this step.
Validate
-
In the
Live Telemetry Analyzer,
add the following label filters:
- Set Label to
__name__and Value tocxdot.integration.target.health. - Set Label to
cxdot.integration.nameand Value toconsul.
cxdot.integration.target, and confirm that the metric reports1for each target. - Set Label to
-
In
Metrics Explorer,
run the following query:
Confirm that each agent you expect appears. Only agents where
consul.mode="leader"also emit catalog metrics for the data center. -
Optional: If you send Consul Raft and Serf metrics to the CXDOT Collector,
run the following query in Metrics Explorer after you write a key to the
Consul key-value store:
Confirm that the query returns a result. Only the Raft leader reports this metric.
Troubleshooting
- Consul metrics don’t appear and Collector logs report HTTP
403responses: Confirm that the ACL token has read access to the agent, catalog, and health endpoints. - Catalog metrics are missing for an agent, but
consul.peersappears: Only the Raft leader for a data center reports catalog metrics. Other agents report peer count only. - Raft and Serf metrics don’t appear: Confirm that the Datadog integration and
its DogStatsD intake are enabled, and that the port in
telemetry.dogstatsd_addrmatches the DogStatsD listener port,9125by default. The CXDOT Collector doesn’t listen on port8125by default. - Raft and Serf metrics don’t appear, but other Consul telemetry such as Go runtime metrics does: Consul 2.0.2 through 2.0.4 don’t send Raft or Serf metrics to any telemetry sink, so upgrade to a later release, or use 2.0.1 until one is available.
Disable the Consul integration
Deleting everyconsul block restores the default unnamed instance. To remove
custom settings and stop collection, retain a disabled configuration:
Configuration reference
Configure one Consul integration instance with the following settings. In Helm values, place these settings undercollection.cxdot.integrations.consul. In a Collector configuration file,
place them under cxdot.integrations.consul.
Optional settings
-
enabledType:boolean. Optional. Default:true. Whether to enable this Consul integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it. -
endpointsType:array of object. Optional. Default:[]. Consul targets. Specify a target ashost:portto use HTTP. Specify a full HTTP or HTTPS URL to control the scheme. -
endpoints[].endpointType:string. Required. Consul target as eitherhost:portor a full HTTP or HTTPS URL. -
acl_tokenType:string. Optional. Consul access control list token sent in theX-Consul-Tokenheader. Provide a token with read access to the agent, catalog, and health endpoints when access control lists are enabled. The Collector masks this sensitive value in diagnostic output, logs, and errors. Discovery annotations can provide a target-specific token. -
collection_intervalType:duration. Optional. Default:15s. How often the Collector collects metrics from each Consul target. -
timeoutType:duration. Optional. Default:15s. Maximum time allowed to collect metrics from one Consul target. The value must not exceedcollection_interval. -
max_servicesType:integer. Optional. Default:50. Minimum:1. Cap on how many catalog services one collection interrogates. Each service costs one health-endpoint request, so an unbounded catalog would turn a single collection into thousands of requests against the agent. Services are taken in name order once the catalog exceeds the cap, so the truncated set stays the same between collections rather than shifting with the agent’s response ordering. -
network_latency_checksType:boolean. Optional. Default:false. Whether the Consul leader reportsconsul.net.node.latency, each node’s estimated round-trip times to the other nodes in its network segment, computed from network coordinates. Reading coordinates requires thenode:readaccess control list permission for every node. For a target discovered through a Datadog annotation, the annotation’snetwork_latency_checksfield applies instead. -
tlsType:object. Optional. Transport layer security (TLS) settings for the HTTPS targets inendpoints. Certificate verification is enabled unlessinsecure_skip_verifyis true. These settings don’t apply to HTTP targets. Discovery annotations can provide target-specific TLS settings. -
tls.ca_fileType:string. Optional. Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty uses system root CA. (optional) -
tls.ca_pemType:string. Optional. In memory PEM encoded cert. (optional) -
tls.cert_fileType:string. Optional. Path to the TLS cert to use for TLS required connections. (optional) -
tls.cert_pemType:string. Optional. In memory PEM encoded TLS cert to use for TLS required connections. (optional) -
tls.cipher_suitesType:array of string. Optional. CipherSuites is a list of TLS cipher suites that the TLS transport can use. If left blank, a safe default list is used. See https://go.dev/src/crypto/tls/cipher_suites.go for a list of supported cipher suites. -
tls.curve_preferencesType:array of string. Optional. contains the elliptic curves that will be used in an ECDHE handshake, in preference order Defaults to empty list and “crypto/tls” defaults are used, internally. -
tls.include_system_ca_certs_poolType:boolean. Optional. If true, load system CA certificates pool in addition to the certificates configured in this struct. -
tls.insecureType:boolean. Optional. In gRPC and HTTP when set to true, this is used to disable the client transport security. See https://godoc.org/google.golang.org/grpc#WithInsecure for gRPC. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional, default false) -
tls.insecure_skip_verifyType:boolean. Optional. InsecureSkipVerify will enable TLS but not verify the certificate. -
tls.key_fileType:string. Optional. Path to the TLS key to use for TLS required connections. (optional) -
tls.key_pemType:string. Optional. In memory PEM encoded TLS key to use for TLS required connections. (optional) -
tls.max_versionType:string. Optional. MaxVersion sets the maximum TLS version that is acceptable. If not set, refer to crypto/tls for defaults. (optional) -
tls.min_versionType:string. Optional. MinVersion sets the minimum TLS version that is acceptable. If not set, TLS 1.2 will be used. (optional) -
tls.reload_intervalType:duration. Optional. ReloadInterval specifies the duration after which the certificate will be reloaded If not set, it will never be reloaded (optional) -
tls.server_name_overrideType:string. Optional. ServerName requested by client for virtual hosting. This sets the ServerName in the TLSConfig. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional) -
tls.tpmType:object. Optional. Trusted platform module configuration -
tls.tpm.authType:string. Optional. Authorization value for the trusted platform module key. -
tls.tpm.enabledType:boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key. -
tls.tpm.owner_authType:string. Optional. Owner authorization value for the trusted platform module. -
tls.tpm.pathType:string. Optional. The path to the TPM device or Unix domain socket. For instance /dev/tpm0 or /dev/tpmrm0.