containerd

The containerd integration requires CXDOT Collector 1.8.0 or greater. containerd is a container runtime that manages the full lifecycle of containers on a node. Use the containerd integration with the CXDOT Collector to collect per-container resource metrics and image sizes from the containerd daemons running in your environment. The containerd integration supports containerd 1.6 or greater.

​

Supported telemetry types

The containerd integration supports these telemetry types:

​

Prerequisites

The containerd integration has the following prerequisites:

• Run containerd as the container runtime on the nodes you collect from. To confirm which runtime your nodes run, run the following command and read the CONTAINER-RUNTIME column:
  kubectl get nodes -o wide
  
• Run your nodes on Linux with cgroups v2, which most current distributions use. On a cgroups v1 node the integration still reports each container’s uptime and CPU limit, but none of the cgroup readings: CPU time, memory, block I/O, and process counts. To confirm which version a node uses, run the following command on that node and confirm that it reports cgroup2fs:
  stat -fc %T /sys/fs/cgroup
  

​

Configure

The containerd integration is disabled by default. To configure the containerd integration, follow these steps:

1. Enable the integration. For example, add the following to the Helm values file for your CXDOT Collector:
   config:
     integrations:
       containerd:
         enabled: true
   
   Each Collector reads the containerd daemon on its own node and reports the containers that daemon runs, along with the size of every image in its store. Metrics from a container that Kubernetes created arrive with that pod’s Kubernetes metadata attached. A container created outside Kubernetes carries none of that metadata.
2. Allow the node Collector to run as root. For example, add the following to the Helm values file for your CXDOT Collector:
   nodeCollector:
     securityContext:
       runAsRoot: true
   
   The containerd socket is owned by root and readable by no one else, so the Collector reaches the daemon only when it runs as root. The Helm chart refuses to render the containerd integration without runAsRoot, rather than deploying a Collector that can’t open the socket.
3. Optional: Report a chosen set of containerd namespaces instead of every namespace the daemon has. For example, add the following to the Helm values file for your CXDOT Collector to report Kubernetes containers alone:
   config:
     integrations:
       containerd:
         enabled: true
         namespaces:
           - k8s.io
   
   A non-empty namespaces list reports exactly the namespaces you name, and exclude_namespaces no longer applies.
4. Optional: Stop collecting open file descriptor counts. For example, add the following to the Helm values file for your CXDOT Collector:
   config:
     integrations:
       containerd:
         enabled: true
         collect_open_file_descriptors: false
   
   Counting descriptors costs one extra call to the daemon for each container, plus a read of the node’s process table. The work therefore grows with the number of containers on a node and the number of processes each one runs. Turning it off leaves every other metric collecting normally.

​

Validate

To validate the containerd integration, follow these steps:

1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health and cxdot.integration.name=containerd. Confirm that the metric reports 1, meaning the Collector reached the daemon and listed its namespaces, containers, and images. A value of 0 means that one of those listings failed. The host.name attribute identifies which node’s daemon a reading describes.
2. In Metrics Explorer, run the following query:
   count by ("host.name") ({"container.memory.usage", "cxdot.integration.name"="containerd"})
   
   Confirm that the query returns one time series per node, each counting the containers running on that node. The cxdot.integration.name filter keeps the count to this integration, because the Kubernetes integration reports the same metric name for the containers the kubelet runs.

​

Troubleshooting

• Containers report their uptime and CPU limit, but no CPU time, memory, block I/O, or process counts: The node runs cgroups v1, whose readings the daemon reports in a form the integration doesn’t read. Only the fields the integration takes from each container’s own record remain. Run the node on cgroups v2 to collect the rest.
• Every other metric arrives, but container.open_file_descriptors is missing: Descriptor counts come from the node’s own process table, which the Collector reads under root_path. A Collector that doesn’t run as root reads no other user’s descriptors and reports no count for any container. Outside the Helm chart, nothing mounts the node file system or sets root_path in node mode, and the count is skipped for every container. The count is also skipped for an individual container whose processes the Collector can’t all read, because a partial tally can’t be told apart from a real one.
• Each container on a node is reported twice, once by the containerd integration and once by the Docker integration: The node runs Docker Engine on top of containerd, which keeps its containers in the moby containerd namespace, and that namespace was removed from exclude_namespaces. Restore moby to exclude_namespaces and let the Docker integration report those containers.

​

Configuration reference

Configure one containerd integration instance with the following settings. In Helm values, place these settings under config.integrations.containerd. In a Collector configuration file, place them under cxdot.integrations.containerd.

​

Optional settings

• enabled Type: boolean. Optional. Default: true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it.
• endpoint Type: string. Optional. Default: unix:///run/containerd/containerd.sock. The containerd socket the Collector reads container metrics from, as a unix:// URL or a bare path. The daemon serves its API over a Unix socket only. The default is the socket on the node the Collector runs on: directly in host mode, and in Kubernetes because the Helm chart mounts that node’s socket directory into the Collector.
• namespaces Type: array of string. Optional. Default: []. The containerd namespaces to report on. An empty list reports every namespace the daemon has except those in exclude_namespaces. Kubernetes containers live in k8s.io; containers created with nerdctl or ctr live in default unless told otherwise.
• exclude_namespaces Type: array of string. Optional. Default: ["moby"]. The containerd namespaces to leave out when namespaces is empty. A namespace listed in namespaces is always reported, regardless of this setting. The default leaves out moby, where Docker Engine runs its containers when containerd is its runtime: the docker integration reports those, and reporting them here too would describe each container twice, once per runtime.
• exclude_pause_containers Type: boolean. Optional. Default: true. Whether to leave out the sandbox container, also called the pause container, that Kubernetes runs for every pod. It holds the pod’s network and nothing else, so its readings are a flat near-zero per pod. Set to false to report it like any other container.
• collection_interval Type: duration. Optional. Default: 10s. How often the Collector collects metrics from the containerd daemon.
• timeout Type: duration. Optional. Default: 10s. Maximum time the Collector waits for the containerd daemon to respond during one collection. This value must not exceed collection_interval. If the deadline expires, that collection fails.
• collect_open_file_descriptors Type: boolean. Optional. Default: true. Whether to collect container.open_file_descriptors, the number of file descriptors open across all of a container’s processes. If true, the Collector reports the metric. If false, it reports everything else and omits this one. The count is read from the node’s process table under root_path, with one daemon call per container to list its processes, so its cost grows with the number of containers and the processes each runs. The Collector can read descriptor counts only when it runs as root, because another user’s open descriptors are readable only by root. In Kubernetes the Helm chart already requires nodeCollector.securityContext.runAsRoot for this integration, because the containerd socket itself is readable only by root.
• root_path Type: string. Optional. Root of the node file system when the Collector runs in a container with the node root bind-mounted, for example /hostfs. In Kubernetes the Helm chart mounts it and sets this value, so it rarely needs setting by hand. The Collector reads it for two per-container fields the daemon doesn’t report: the open file descriptor count, and the kernel memory total the kernel prints in each container’s memory.stat. Without it, the descriptor count isn’t reported and kernel memory is the sum of the kernel stack and slab readings the daemon does report, which is lower than the true total. In host mode the Collector runs on the node it reports on, so this value defaults to /. It stays unset in node mode outside the Helm chart, where nothing mounts the node root or sets this value for you. Set it to an empty string to turn off the node file system reads where the default would otherwise apply.
• containers Type: object. Optional. Settings for containerd container metrics collection.
• containers.enabled Type: boolean. Optional. Default: true. Whether this integration instance collects container metrics. If true, the Collector reports per-container readings and image sizes from the daemon this instance reads. If false, the instance collects nothing, the same result as turning the whole instance off with its own enabled setting, because container metrics are all this integration collects.