Skip to main content
Requires CXDOT Collector version 1.5.0 or later.
You can build custom pipelines to modify how the CXDOT Collector processes and routes telemetry data. Custom pipelines affect telemetry data after an integration collects that data, but before the CXDOT Collector sends it to Palo Alto Networks Cortex XCOR. For example, you can create a custom pipeline that removes sensitive information from logs before the Collector sends those logs to Cortex XCOR, or a custom pipeline that sends one copy of metrics to Cortex XCOR and another copy to a different destination. Each custom pipeline applies to either metrics, logs, or traces, and by default affects data of that telemetry type from all enabled integrations. For example, a pipeline for traces affects trace data from every integration that collects traces, unless the pipeline is configured to only affect certain integrations.

​
Components

Custom pipelines use a subset of OpenTelemetry components. Each custom pipeline consists of the following components: The CXDOT Collector provides two components that can be used as receivers and exporters. Use the forward/cxdot/from-integrations component as a receiver to receive telemetry data from CXDOT Collector integrations. Use the forward/cxdot/to-egress component as an exporter to send telemetry data to Cortex XCOR.
When a custom pipeline receives telemetry data from forward/cxdot/from-integrations, that custom pipeline replaces the default path for that telemetry type. To continue sending all telemetry data of that type to Cortex XCOR, every processing or routing path must eventually export to forward/cxdot/to-egress.

​
Connect multiple pipelines

If you’re using multiple custom pipelines for the same telemetry type, use forward/cxdot/from-integrations as a receiver in the first pipeline of your sequence, then use OpenTelemetry connectors to allow telemetry data to flow from one pipeline to the next. Use forward/cxdot/to-egress as an exporter in any pipelines that should send telemetry data to Cortex XCOR. For example, the following snippet uses the forward/next connector to send metrics from one custom pipeline to another. The forward/cxdot/from-integrations receiver in the first pipeline receives telemetry data, and the forward/cxdot/to-egress exporter in the second pipeline sends that data to Cortex XCOR.
If multiple pipelines are joined by a routing connector, those pipelines must all handle the same telemetry type as that routing connector. For example, a routing connector that handles metrics must join only pipelines for metrics.

​
List compatible OpenTelemetry components

To list the OpenTelemetry components that are compatible with your version of the CXDOT Collector, use the following command:
Replace POD with the name of any running CXDOT Collector Pod.
The forward component used by forward/cxdot/from-integrations and forward/cxdot/to-egress appears in the command’s connectors list.

​
Create a custom pipeline

To create a custom pipeline, add a service.pipelines block to your configuration file that defines receivers and exporters. Optionally, add a processors block that defines processors.
Before you create a custom pipeline for logs or traces, make sure to enable telemetry egress for that telemetry type.

​
Process telemetry data

The following example creates a custom pipeline named metrics/add-env that adds the env = prod resource attribute to metrics. Logs and traces aren’t affected.
Add processors and service.pipelines blocks to both the nodeCollector.configOverlay and clusterCollector.configOverlay sections of your Helm values file.
values.yaml

​
Send telemetry data to an additional destination

The following example creates a custom pipeline named logs/copy that sends a copy of log data to an OTLP HTTP endpoint while continuing to send logs to Cortex XCOR. Metrics and traces aren’t affected.
Add exporters and service.pipelines blocks to both the nodeCollector.configOverlay and clusterCollector.configOverlay sections of your Helm values file.
values.yaml

​
Send batched telemetry data to an additional destination

Telemetry data exported through forward/cxdot/to-egress is batched before being sent to Cortex XCOR, but other exporters don’t batch data by default. To batch telemetry data sent through another exporter, add a batch processor to the pipeline for that destination. The following example creates two custom pipelines. The first custom pipeline, named logs/copy, sends logs to Cortex XCOR using the forward/cxdot/to-egress exporter, and also routes a copy of that log data to the second pipeline. The second custom pipeline, named logs/additional, uses a batch processor to batch these logs before sending them to an OTLP HTTP endpoint. Metrics and traces aren’t affected.
Add connectors, exporters, processors, and service.pipelines blocks to both the nodeCollector.configOverlay and clusterCollector.configOverlay sections of your Helm values file.
values.yaml

​
Configure per-integration behavior

Each custom pipeline applies to either metrics, logs, or traces, and by default affects data of that telemetry type from all enabled integrations. For example, a pipeline for traces affects trace data from every integration that collects traces. To change this behavior, use the xcor.integration.name attribute and cxdot.integration.instance.name attribute to configure per-integration behavior for your custom pipelines. This lets custom pipelines selectively route and process telemetry data from specified integrations without affecting other integrations. The xcor.integration.name attribute identifies an integration type. For example, this condition block matches every configured MySQL integration instance. If multiple MySQL integration instances are configured, it will match them all.
The cxdot.integration.instance.name attribute identifies a named integration instance. Use this attribute in conjunction with the xcor.integration.name attribute. For example, this condition block matches a configured NGINX integration with the instance name bravo:

​
Process telemetry data from one integration type

The following example creates a custom pipeline named metrics/filter-mysql that drops the mysql.locks metric from all instances of the MySQL integration. Metrics from other integration types aren’t affected, and logs and traces aren’t affected.
Add processors and service.pipelines blocks to both the nodeCollector.configOverlay and clusterCollector.configOverlay sections of your Helm values file.
values.yaml

​
Route telemetry data from one integration type

The following example creates three custom pipelines. The first custom pipeline, named metrics/route-integrations, receives metrics from all integrations and sends them to a routing connector, which routes metrics to one of two branching pipelines. The metrics/mysql pipeline sends metrics from all instances of the MySQL integration to Cortex XCOR and an additional destination. The metrics/other pipeline sends metrics from all other integration types to Cortex XCOR. Metrics from all integrations are sent to Cortex XCOR, but only metrics from instances of the MySQL integration are sent to the additional destination. Logs and traces aren’t affected.
Add connectors, exporters, and service.pipelines blocks to both the nodeCollector.configOverlay and clusterCollector.configOverlay sections of your Helm values file.
values.yaml