Supported telemetry types
The Docker integration supports these telemetry types:Prerequisites
The Docker integration has the following prerequisites:-
Run Docker Engine on the hosts or Kubernetes nodes you collect from. Most Kubernetes
clusters run containerd instead, which exposes no Docker daemon for this integration
to read. To confirm which runtime your nodes run, run the following command and read
the
CONTAINER-RUNTIMEcolumn: -
Run your hosts or nodes on Linux with cgroups v2, which most current distributions
use. On a cgroups v1 machine the integration still reports container and image counts
and the per-container metrics the Docker daemon serves, but not the fields read from
the machine’s cgroup tree: block I/O operation counts, memory reclaim events, kernel
and swap memory, and open file descriptors. To confirm which version a host or node
uses, run the following command on it and confirm that it reports
cgroup2fs: - Make the Docker daemon reachable from the Collector, either through the Unix socket on the machine the Collector runs on or over a TCP address the daemon listens on.
Configure
The Docker integration is disabled by default. Follow the steps for where the CXDOT Collector runs: directly on each Docker host, or on Kubernetes with the CXDOT Collector Helm chart.Configure on a host
To configure the Docker integration on a host, follow these steps:-
Enable the integration. Run the Collector in host mode on each Docker host and add
the following to the Collector configuration file:
Each Collector reads the Docker daemon on its own host and reports the containers that daemon runs. For more information about running the Collector on a host, see install the CXDOT Collector on Linux.
-
Give the Collector access to the Docker socket. Run the Collector as root, or as an
account in the
dockergroup, which owns the socket. For example, add the account that runs the Collector to thedockergroup:ReplaceCOLLECTOR_USERwith the name of that account, then restart the Collector. An account other than root also needscxdot.debug.socket_pathset to a path in a directory that account can write to. The default debug socket directory is under/var/run, and a Collector that can’t create it doesn’t start. Running as root is also required to emitcontainer.file_descriptor.count. Membership in thedockergroup doesn’t grant it. A Collector that doesn’t run as root reports every other metric and skips this one. -
Optional: Collect from a daemon at another address, such as one that listens on a TCP
port. For example, add the following to the Collector configuration file:
Open file descriptors are counted on the host the Collector runs on, so a daemon at any address other than a Unix socket,
localhost, or127.0.0.1reports nocontainer.file_descriptor.count. -
Optional: Exclude the containers of images you don’t want reported, such as
short-lived build agents. For example, add the following to the Collector
configuration file:
Configure on Kubernetes
To configure the Docker integration on Kubernetes, follow these steps:-
Enable the integration. For example, add the following to the
values.yamlfor your CXDOT Collector Helm chart:Each Collector reads the Docker daemon on its own node and reports the containers that daemon runs. That work runs on the node Collector. For more information about Collector tiers and node placement, see architecture. -
Allow the node Collector to run as root. For example, add the following to the
values.yamlfor your CXDOT Collector Helm chart:runAsRootis required for a Unix socket endpoint.runAsRootis also required to emitcontainer.file_descriptor.count. Leavecollect_open_file_descriptorsout of yourvalues.yamlto keep the default, where the Collector reports the count when it can read it and skips it when it can’t. Writingcollect_open_file_descriptors: truemakes the metric a requirement instead: the chart then refuses to deploy unless the Collector runs as root and reads a daemon on its own node. -
Optional: Collect from a daemon at another address, such as one that listens on a TCP
port. For example, add the following to the
values.yamlfor your CXDOT Collector Helm chart:Open file descriptors are counted on the node the Collector runs on, so a daemon at any address other than a Unix socket,localhost, or127.0.0.1reports nocontainer.file_descriptor.count. Don’t setcollect_open_file_descriptors: truefor such an endpoint, because the chart then refuses to deploy. -
Optional: Exclude the containers of images you don’t want reported, such as
short-lived build agents. For example, add the following to the
values.yamlfor your CXDOT Collector Helm chart:
Validate
To validate the Docker integration, follow these steps:-
In
Live Telemetry Analyzer,
filter for
__name__=cxdot.integration.target.healthandcxdot.integration.name=docker. Confirm that the metric reports1, meaning the Collector reached the daemon and listed its containers and images. A value of0means that one of those listings failed. Thehost.nameattribute identifies which host’s or node’s daemon a reading describes. -
In
Metrics Explorer,
run the following query:
Confirm that the query returns one time series per host or node, each counting the containers running there.
Troubleshooting
- The health metric reports
0and the Collector logsclient version 1.44 is too new: The daemon runs a Docker Engine release older than 25.0, which serves an older API version than the integration requests by default. Setapi_versionto the maximum API version the message reports, whichdocker versionalso shows for the server. The integration doesn’t fully support Docker Engine releases older than 29.0, so some metrics can be missing on them. - Every other metric arrives, but
container.file_descriptor.countis missing: Descriptor counts come from each process’s own entry under/proc, on the machine the containers run on, and only that process’s own user and root can read them. A Collector that doesn’t run as root reads none of them and reports no count for any container. A Collector reading a daemon on another machine reports no count either, whatever it runs as, because the counts are on that machine. For a daemon on the same machine, run the Collector as root, as shown in Configure, or setcollect_open_file_descriptorstofalseto stop collecting the metric.
Configuration reference
Configure one Docker integration instance with the following settings. In Helm values, place these settings undercollection.cxdot.integrations.docker. In a Collector configuration file,
place them under cxdot.integrations.docker.
Optional settings
-
enabledType:boolean. Optional. Default:true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it. -
endpointType:string. Optional. Default:unix:///var/run/docker.sock. The Docker daemon the Collector reads container stats from. Either a Unix socket such asunix:///var/run/docker.sock, or a TCP address such astcp://host:2375. The default connects to the daemon on the node the Collector runs on: directly inhostmode, and in Kubernetes because the Helm chart mounts that node’s socket into the Collector. -
api_versionType:string. Optional. Default:1.44. Docker Engine API version the Collector requests from the daemon. A daemon rejects a version newer than its own API version, whichdocker versionreports for the server. The Collector supports API versions 1.25 and later. The default 1.44 works with Docker Engine 25.0 and later. -
excluded_imagesType:array of string. Optional. Default:[]. Images whose containers aren’t reported. Each entry is a literal name, a glob, or a/regex/, matched against the whole image reference the container was created from, tag included, rather than againstcontainer.image.name. For example,nginxexcludes containers created fromnginxbut not fromnginx:1.27. Listnginx:*as well to exclude every tag. -
env_vars_to_metric_labelsType:object. Optional. Default:{}. Container environment variables to report as resource attributes, as pairs of an environment variable name and the attribute name its value is reported under. The Collector adds each attribute to every series from a container that sets the variable, and counts containers incontainer.countseparately for each combination of these attributes, for stopped containers as well as running ones. For example, mapMESOS_TASK_IDtomesos.task.idto identify the scheduler task each container runs. Each attribute name can appear only once, and the Collector rejects a configuration that maps two variables to the same attribute. -
collection_intervalType:duration. Optional. Default:10s. How often the Collector collects metrics from the Docker daemon. -
timeoutType:duration. Optional. Default:10s. Maximum time the Collector waits for the Docker daemon to respond during one collection. This value must not exceedcollection_interval. If the deadline expires, that collection fails. -
collect_open_file_descriptorsType:boolean. Optional. Default:true. Whether to collectcontainer.file_descriptor.count, the number of file descriptors open across all of a container’s processes. If true, the Collector reports the metric. If false, it reports everything else and omits this one. The cost of this metric grows with the number of processes in the reported containers and the descriptors each has open, because every collection counts them again. Working out which container a process belongs to takes a read for every process on the node. That answer is cached and rechecked only every few dozen collections, so its cost is paid mostly on the first collection. Turning the metric off leaves every other field read underroot_pathcollecting normally. The Collector can read descriptor counts only for a daemon on its own node, and only when it runs as root, because another user’s open descriptors are readable only by root. On a host, run the Collector as root. In Kubernetes, setnodeCollector.securityContext.runAsRootin the Helm chart. Left unset, the setting takes its default without making the metric a requirement: a count the Collector can’t read is skipped, with a warning, rather than reported. Written out astrue, it becomes a requirement: the Helm chart fails to render unless the daemon is on the same node as the Collector andrunAsRootis set. -
collect_container_sizeType:boolean. Optional. Default:false. Whether to collect the size of each running container’s file system:container.filesystem.writable_layer.usage, the size of its writable layer, andcontainer.filesystem.rootfs.usage, the size of its image and writable layer together. If true, the Collector reports both. If false, it reports neither. Computing the sizes makes the Docker daemon walk the writable layer of every container, which is slow on a node with many containers or large writable layers. To limit that load, the Collector requests and reports the sizes on one collection in five, so each size has one value every fivecollection_intervalperiods. A collection that requests sizes and runs pasttimeoutfails like any other. A container whose size the daemon can’t compute is skipped. -
root_pathType:string. Optional. Root of the node file system when the Collector runs in a container with the node root bind-mounted, for example/hostfs. In Kubernetes the Helm chart mounts it and sets this value, so it rarely needs setting by hand. The Collector reads it for the per-container cgroup fields the Docker daemon’s stats API doesn’t report on cgroups v2: block I/O operation counts, memory reclaim events, kernel and swap memory, and open file descriptors. Those fields come from the node’s cgroup tree instead. Inhostmode against a daemon on the same node, meaning a Unix socket or atcp://endpoint on a loopback host such as127.0.0.1orlocalhost, the Collector runs on the node it reports on, so this value defaults to/and those fields are read from that node’s own/procand/sys/fs/cgroup. It stays unset for atcp://endpoint naming another host, because these fields can’t be collected without access to that host’s file system. It also stays unset innodemode outside the Helm chart, where nothing mounts the node root or sets this value for you. When this value is unset, those fields aren’t collected. Set it to""to switch the fields off where the default would otherwise apply. -
containersType:object. Optional. Settings for Docker container metrics collection. -
containers.enabledType:boolean. Optional. Default:true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it.