Discover Kubernetes scrape targets with the CXDOT Collector

If the CXDOT Collector is deployed in a Kubernetes cluster, it uses autodiscovery to find supported scrape targets for pull-based metric sources, even as workloads change. This lets the CXDOT Collector begin collecting metrics when a matching target appears, and stop collecting metrics when Kubernetes no longer reports that target.

Kubernetes autodiscovery doesn’t identify log or trace sources.

Prerequisites

Before using Kubernetes autodiscovery, you must meet these prerequisites:

The config.integrations.discovery.enabled setting in your Helm values file must be enabled. If no value is specified, this setting is enabled by default. This setting lets the CXDOT Collector discover targets.
The apiServer.metadataServer.enabled setting in your Helm values file must be enabled. If no value is specified, this setting is enabled by default. This setting lets the CXDOT Collector retrieve Kubernetes metadata, including metadata used for autodiscovery.
Configure an integration that supports Kubernetes autodiscovery.
Ensure that the CXDOT Collector can connect to the workload or service associated with that integration.

Use autodiscovery

Autodiscovery is enabled by default, but the CXDOT Collector discovers targets only if a corresponding integration is enabled. When an integration is enabled, the Collector uses two methods to discover targets associated with that integration: label-based discovery, and discovery through Datadog annotations.

If a Pod or container includes both labels and Datadog annotations for the same integration, annotation-based targets replace label-based targets. However, if none of the matching annotations resolve successfully, the CXDOT Collector uses an eligible label-based target instead.

Use labels to discover targets

The CXDOT Collector uses labels to discover targets when a Pod’s labels match the criteria defined by an enabled integration. For example, the NGINX integration matches Pods with the label app.kubernetes.io/name: nginx. When a Pod includes this label, the Collector discovers that Pod as an NGINX scrape target. Label discovery criteria vary by integration. For more information, consult the documentation for each integration.

Use Datadog annotations to discover targets

The CXDOT Collector uses Datadog annotations to discover targets when a Pod uses Datadog Autodiscovery annotations that use the v2 .checks format and the corresponding CXDOT Collector integration is enabled. Use this method for workloads that already have Datadog annotations or need target-specific settings that label-based discovery can’t provide.

Legacy v1 annotations and Datadog exclusion annotations aren’t supported.

Configure Pod annotations

For a Deployment, DaemonSet, or StatefulSet, add an annotation to the Pod template in spec.template.metadata.annotations. For a standalone Pod, add an annotation to metadata.annotations. The annotation key must follow this format:

ad.datadoghq.com/<container>.checks

Replace <container> with the name of the monitored container. In a workload controller, find this value under spec.template.spec.containers. The value of this annotation must contain a JSON object that defines one or more Datadog checks. Each top-level key identifies a check, and its instances array provides configuration for one or more endpoints. Each valid instance becomes a discovered scrape target.

The CXDOT Collector supports a subset of Datadog Autodiscovery configuration. It reads the instances list and ignores init_config, and it processes only the check names and instance fields supported by the corresponding CXDOT Collector integration. Consult each integration’s documentation before reusing an existing Datadog annotation.

For example, the following Deployment fragment defines an NGINX target:

spec:
  template:
    metadata:
      annotations:
        ad.datadoghq.com/web.checks: |
          {
            "nginx": {
              "instances": [
                {
                  "nginx_status_url": "http://%%host%%:%%port%%/status"
                }
              ]
            }
          }
    spec:
      containers:
        - name: web
          ports:
            - containerPort: 80
              protocol: TCP

In this example, the value web in the annotation key identifies the monitored container and matches name: web in the container list. The top-level nginx key in the JSON identifies the Datadog check that the CXDOT Collector maps to the NGINX integration, and nginx_status_url defines the endpoint to monitor. The %%host%% variable resolves to the Pod IP address, and %%port%% resolves to the highest-numbered TCP port declared by the web container.

Use template variables

The CXDOT Collector supports the following template variables inside instances entries. The Collector replaces these variables with their associated value before using the entry to define a scrape target.

The CXDOT Collector resolves these variables relative to the container specified in the annotation key.

Variable	Value
`%%host%%`	The IP address of the Pod that contains the specified container.
`%%port%%`	The highest-numbered TCP port declared by the specified container.
`%%port_NAME%%`	The specified container’s TCP port whose `name` field matches `NAME`.
`%%port_INDEX%%`	The TCP port at the zero-based position specified by `INDEX`, after the Collector sorts the specified container’s TCP ports from lowest to highest.
`%%kube_namespace%%`	The Kubernetes namespace that contains the annotated Pod.
`%%env_VAR%%`	The value of the environment variable specified by `VAR`. This must be an environment variable available to the CXDOT Collector process. A variable that exists only in the Pod or container being monitored won’t resolve. For more information, see Use environment variables.

Use environment variables

Use the %%env_VAR%% template variable inside instances entries to reference an environment variable available to the CXDOT Collector process. No environment variables are allowed by default. To use %%env_VAR%% in annotations, ensure that the variable exists in the environment where the Collector is deployed, then add its name to allowed_env_vars in your Helm values file:

values.yaml

config:
  integrations:
    discovery:
      kubernetes:
        allowed_env_vars:
          - NAME_OF_ENVIRONMENT_VARIABLE

Environment variable names are case-sensitive. If a referenced variable is unavailable or isn’t listed in allowed_env_vars, the CXDOT Collector skips the affected instance.

Use Kubernetes Secrets

Use the following syntax inside instances entries to retrieve the value of a Kubernetes Secret:

ENC[k8s_secret@NAMESPACE/SECRET/KEY]

Replace the following:

NAMESPACE: The namespace that contains the Secret.
SECRET: The name of the Secret.
KEY: The key that contains the value to retrieve.

No Secrets are allowed by default. To use Kubernetes Secrets in annotations, add each required Secret to secretAllowlist in your Helm values file:

values.yaml

apiServer:
  metadataServer:
    secretAllowlist:
      - namespace: default
        names:
          - SECRET

Secrets must be in the same namespace as the annotated Pod.

Configure autodiscovery behavior

Use the information in this section to configure the CXDOT Collector’s autodiscovery behavior.

Set the Kubernetes polling interval

By default, the CXDOT Collector polls for changes to targets every 30 seconds. To change this polling interval, set the value of node_poll_interval:

values.yaml

config:
  integrations:
    discovery:
      kubernetes:
        node_poll_interval: 15s

This must be a positive, non-zero value. Shorter intervals let the Collector detect target changes sooner, but also increase request load on the CXDOT API server.

Filter discovered targets by namespace

By default, the CXDOT Collector can discover supported targets in any namespace. To limit autodiscovery to specific namespaces, configure include_metrics.kube_namespace and exclude_metrics.kube_namespace. These settings accept RE2 regular expressions. The Collector matches each expression against the entire namespace name. When include_metrics.kube_namespace is set, the Collector discovers targets only in matching namespaces. When exclude_metrics.kube_namespace is set, the Collector ignores namespaces that match this value. Exclusion expressions take precedence over inclusion expressions. For example, the following configuration discovers targets only in namespaces that begin with prod- or staging-, except for staging-canary:

values.yaml

config:
  integrations:
    discovery:
      include_metrics:
        kube_namespace: "(prod|staging)-.*"
      exclude_metrics:
        kube_namespace: "staging-canary"

These settings don’t affect static targets or other collection paths.

Use both discovered targets and static targets for the same integration type

Static targets are endpoints that are explicitly listed in an integration’s configuration settings. For a given instance of an integration, setting static targets disables autodiscovery for that instance. To use both discovered targets and static targets for the same integration type, define two instances of the integration, then configure static targets for only one instance of that integration:

values.yaml

config:
  integrations:
    nginx/first-instance:
      enabled: true
    nginx/second-instance:
      enabled: true
      endpoints:
        - endpoint: nginx1.example.com:80
        - endpoint: nginx2.example.com:80

Disable autodiscovery

To disable autodiscovery, set config.integrations.discovery.enabled to false:

values.yaml

config:
  integrations:
    discovery:
      enabled: false

Don’t set apiServer.metadataServer.enabled to false solely to disable autodiscovery. Disabling the metadata server also disables other features of the CXDOT Collector.

Cortex XCOR

Palo Alto Networks Telemetry Pipeline

More information

Discover Kubernetes scrape targets with the CXDOT Collector

Prerequisites

Use autodiscovery

Use labels to discover targets

Use Datadog annotations to discover targets

Configure Pod annotations

Use template variables

Use environment variables

Use Kubernetes Secrets

Configure autodiscovery behavior

Set the Kubernetes polling interval

Filter discovered targets by namespace

Use both discovered targets and static targets for the same integration type

Disable autodiscovery

​Prerequisites

​Use autodiscovery

​Use labels to discover targets

​Use Datadog annotations to discover targets

​Configure Pod annotations

​Use template variables

​Use environment variables

​Use Kubernetes Secrets

​Configure autodiscovery behavior

​Set the Kubernetes polling interval

​Filter discovered targets by namespace

​Use both discovered targets and static targets for the same integration type

​Disable autodiscovery

Prerequisites

Use autodiscovery

Use labels to discover targets

Use Datadog annotations to discover targets

Configure Pod annotations

Use template variables

Use environment variables

Use Kubernetes Secrets

Configure autodiscovery behavior

Set the Kubernetes polling interval

Filter discovered targets by namespace

Use both discovered targets and static targets for the same integration type

Disable autodiscovery