Skip to main content
The Docker integration requires CXDOT Collector 1.4.0 or greater. Docker is a platform for building and running applications in containers. Use the Docker integration with the CXDOT Collector to collect per-container resource metrics and container counts from Docker daemons running in your environment. The Docker integration supports Docker Engine 29.0 or greater.

Supported telemetry types

The Docker integration supports these telemetry types:

Prerequisites

The Docker integration has the following prerequisites:
  • Run Docker Engine on the hosts or Kubernetes nodes you collect from. Most Kubernetes clusters run containerd instead, which exposes no Docker daemon for this integration to read. To confirm which runtime your nodes run, run the following command and read the CONTAINER-RUNTIME column:
  • Run your hosts or nodes on Linux with cgroups v2, which most current distributions use. On a cgroups v1 machine the integration still reports container and image counts and the per-container metrics the Docker daemon serves, but not the fields read from the machine’s cgroup tree: block I/O operation counts, memory reclaim events, kernel and swap memory, and open file descriptors. To confirm which version a host or node uses, run the following command on it and confirm that it reports cgroup2fs:
  • Make the Docker daemon reachable from the Collector, either through the Unix socket on the machine the Collector runs on or over a TCP address the daemon listens on.

Configure

The Docker integration is disabled by default. Follow the steps for where the CXDOT Collector runs: directly on each Docker host, or on Kubernetes with the CXDOT Collector Helm chart.

Configure on a host

To configure the Docker integration on a host, follow these steps:
  1. Enable the integration. Run the Collector in host mode on each Docker host and add the following to the Collector configuration file:
    Each Collector reads the Docker daemon on its own host and reports the containers that daemon runs. For more information about running the Collector on a host, see install the CXDOT Collector on Linux.
  2. Give the Collector access to the Docker socket. Run the Collector as root, or as an account in the docker group, which owns the socket. For example, add the account that runs the Collector to the docker group:
    Replace COLLECTOR_USER with the name of that account, then restart the Collector. An account other than root also needs cxdot.debug.socket_path set to a path in a directory that account can write to. The default debug socket directory is under /var/run, and a Collector that can’t create it doesn’t start. Running as root is also required to emit container.file_descriptor.count. Membership in the docker group doesn’t grant it. A Collector that doesn’t run as root reports every other metric and skips this one.
  3. Optional: Collect from a daemon at another address, such as one that listens on a TCP port. For example, add the following to the Collector configuration file:
    Open file descriptors are counted on the host the Collector runs on, so a daemon at any address other than a Unix socket, localhost, or 127.0.0.1 reports no container.file_descriptor.count.
  4. Optional: Exclude the containers of images you don’t want reported, such as short-lived build agents. For example, add the following to the Collector configuration file:

Configure on Kubernetes

To configure the Docker integration on Kubernetes, follow these steps:
  1. Enable the integration. For example, add the following to the values.yaml for your CXDOT Collector Helm chart:
    Each Collector reads the Docker daemon on its own node and reports the containers that daemon runs. That work runs on the node Collector. For more information about Collector tiers and node placement, see architecture.
  2. Allow the node Collector to run as root. For example, add the following to the values.yaml for your CXDOT Collector Helm chart:
    runAsRoot is required for a Unix socket endpoint. runAsRoot is also required to emit container.file_descriptor.count. Leave collect_open_file_descriptors out of your values.yaml to keep the default, where the Collector reports the count when it can read it and skips it when it can’t. Writing collect_open_file_descriptors: true makes the metric a requirement instead: the chart then refuses to deploy unless the Collector runs as root and reads a daemon on its own node.
  3. Optional: Collect from a daemon at another address, such as one that listens on a TCP port. For example, add the following to the values.yaml for your CXDOT Collector Helm chart:
    Open file descriptors are counted on the node the Collector runs on, so a daemon at any address other than a Unix socket, localhost, or 127.0.0.1 reports no container.file_descriptor.count. Don’t set collect_open_file_descriptors: true for such an endpoint, because the chart then refuses to deploy.
  4. Optional: Exclude the containers of images you don’t want reported, such as short-lived build agents. For example, add the following to the values.yaml for your CXDOT Collector Helm chart:

Validate

To validate the Docker integration, follow these steps:
  1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health and cxdot.integration.name=docker. Confirm that the metric reports 1, meaning the Collector reached the daemon and listed its containers and images. A value of 0 means that one of those listings failed. The host.name attribute identifies which host’s or node’s daemon a reading describes.
  2. In Metrics Explorer, run the following query:
    Confirm that the query returns one time series per host or node, each counting the containers running there.

Troubleshooting

  • The health metric reports 0 and the Collector logs client version 1.44 is too new: The daemon runs a Docker Engine release older than 25.0, which serves an older API version than the integration requests by default. Set api_version to the maximum API version the message reports, which docker version also shows for the server. The integration doesn’t fully support Docker Engine releases older than 29.0, so some metrics can be missing on them.
  • Every other metric arrives, but container.file_descriptor.count is missing: Descriptor counts come from each process’s own entry under /proc, on the machine the containers run on, and only that process’s own user and root can read them. A Collector that doesn’t run as root reads none of them and reports no count for any container. A Collector reading a daemon on another machine reports no count either, whatever it runs as, because the counts are on that machine. For a daemon on the same machine, run the Collector as root, as shown in Configure, or set collect_open_file_descriptors to false to stop collecting the metric.

Configuration reference

Configure one Docker integration instance with the following settings. In Helm values, place these settings under collection.cxdot.integrations.docker. In a Collector configuration file, place them under cxdot.integrations.docker.

Optional settings

  • enabled Type: boolean. Optional. Default: true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it.
  • endpoint Type: string. Optional. Default: unix:///var/run/docker.sock. The Docker daemon the Collector reads container stats from. Either a Unix socket such as unix:///var/run/docker.sock, or a TCP address such as tcp://host:2375. The default connects to the daemon on the node the Collector runs on: directly in host mode, and in Kubernetes because the Helm chart mounts that node’s socket into the Collector.
  • api_version Type: string. Optional. Default: 1.44. Docker Engine API version the Collector requests from the daemon. A daemon rejects a version newer than its own API version, which docker version reports for the server. The Collector supports API versions 1.25 and later. The default 1.44 works with Docker Engine 25.0 and later.
  • excluded_images Type: array of string. Optional. Default: []. Images whose containers aren’t reported. Each entry is a literal name, a glob, or a /regex/, matched against the whole image reference the container was created from, tag included, rather than against container.image.name. For example, nginx excludes containers created from nginx but not from nginx:1.27. List nginx:* as well to exclude every tag.
  • env_vars_to_metric_labels Type: object. Optional. Default: {}. Container environment variables to report as resource attributes, as pairs of an environment variable name and the attribute name its value is reported under. The Collector adds each attribute to every series from a container that sets the variable, and counts containers in container.count separately for each combination of these attributes, for stopped containers as well as running ones. For example, map MESOS_TASK_ID to mesos.task.id to identify the scheduler task each container runs. Each attribute name can appear only once, and the Collector rejects a configuration that maps two variables to the same attribute.
  • collection_interval Type: duration. Optional. Default: 10s. How often the Collector collects metrics from the Docker daemon.
  • timeout Type: duration. Optional. Default: 10s. Maximum time the Collector waits for the Docker daemon to respond during one collection. This value must not exceed collection_interval. If the deadline expires, that collection fails.
  • collect_open_file_descriptors Type: boolean. Optional. Default: true. Whether to collect container.file_descriptor.count, the number of file descriptors open across all of a container’s processes. If true, the Collector reports the metric. If false, it reports everything else and omits this one. The cost of this metric grows with the number of processes in the reported containers and the descriptors each has open, because every collection counts them again. Working out which container a process belongs to takes a read for every process on the node. That answer is cached and rechecked only every few dozen collections, so its cost is paid mostly on the first collection. Turning the metric off leaves every other field read under root_path collecting normally. The Collector can read descriptor counts only for a daemon on its own node, and only when it runs as root, because another user’s open descriptors are readable only by root. On a host, run the Collector as root. In Kubernetes, set nodeCollector.securityContext.runAsRoot in the Helm chart. Left unset, the setting takes its default without making the metric a requirement: a count the Collector can’t read is skipped, with a warning, rather than reported. Written out as true, it becomes a requirement: the Helm chart fails to render unless the daemon is on the same node as the Collector and runAsRoot is set.
  • collect_container_size Type: boolean. Optional. Default: false. Whether to collect the size of each running container’s file system: container.filesystem.writable_layer.usage, the size of its writable layer, and container.filesystem.rootfs.usage, the size of its image and writable layer together. If true, the Collector reports both. If false, it reports neither. Computing the sizes makes the Docker daemon walk the writable layer of every container, which is slow on a node with many containers or large writable layers. To limit that load, the Collector requests and reports the sizes on one collection in five, so each size has one value every five collection_interval periods. A collection that requests sizes and runs past timeout fails like any other. A container whose size the daemon can’t compute is skipped.
  • root_path Type: string. Optional. Root of the node file system when the Collector runs in a container with the node root bind-mounted, for example /hostfs. In Kubernetes the Helm chart mounts it and sets this value, so it rarely needs setting by hand. The Collector reads it for the per-container cgroup fields the Docker daemon’s stats API doesn’t report on cgroups v2: block I/O operation counts, memory reclaim events, kernel and swap memory, and open file descriptors. Those fields come from the node’s cgroup tree instead. In host mode against a daemon on the same node, meaning a Unix socket or a tcp:// endpoint on a loopback host such as 127.0.0.1 or localhost, the Collector runs on the node it reports on, so this value defaults to / and those fields are read from that node’s own /proc and /sys/fs/cgroup. It stays unset for a tcp:// endpoint naming another host, because these fields can’t be collected without access to that host’s file system. It also stays unset in node mode outside the Helm chart, where nothing mounts the node root or sets this value for you. When this value is unset, those fields aren’t collected. Set it to "" to switch the fields off where the default would otherwise apply.
  • containers Type: object. Optional. Settings for Docker container metrics collection.
  • containers.enabled Type: boolean. Optional. Default: true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it.