Skip to main content
Palo Alto Networks Cortex XCOR receives Amazon CloudWatch metrics through CloudWatch Metrics Streams. Configure Amazon Web Services (AWS) to continuously stream metrics to Cortex XCOR through the AWS Management Console or Terraform.

Before you begin

Some AWS services publish metrics at intervals longer than the two-hour ingestion age limit in Cortex XCOR. For example, Amazon S3 daily storage metrics can have timestamps older than two hours when CloudWatch delivers them. Before streaming a metric, verify that its publishing interval is shorter than the ingestion age limit. Cortex XCOR can read S3 daily storage metrics from CloudWatch instead: see S3 daily storage metrics.

Control metric volume

CloudWatch Metric Streams generate metrics that consume Standard Metrics License capacity. Before configuring ingestion, optionally create a drop rule for unwanted metrics. The following Chronoctl YAML resource definition creates a drop rule that drops all metrics from CloudWatch Metric Streams except the AWS/CloudWatch/MetricStreams and AWS/Firehose namespaces required by the validation dashboards.
Chronoctl example
Modify the rule to allow more metrics from other AWS namespaces. This example also allows metrics from the AWS ECS namespace.
Chronoctl example
To reduce the AWS cost of streaming unwanted metrics, configure CloudWatch Metric Streams to include or exclude specific namespaces:
  1. In the AWS Management Console, go to Streams.
  2. Edit the Metric Stream.
  3. Under Metrics to be streamed, include or exclude namespaces.

Configure CloudWatch Metric Streams

In each AWS account and region you want to collect metrics from, a CloudWatch Metric Stream sends data to Amazon Data Firehose. The Firehose stream forwards the data to the CloudWatch ingest endpoint in your Cortex XCOR tenant. Cortex XCOR processes the metrics and makes them available for queries, monitors, and dashboards.

CloudWatch roles and permissions

To use CloudWatch Metric Streams in Cortex XCOR, you must configure a CloudWatch Metric Stream in each AWS account and region. The account you use to set up the CloudWatch Metric Stream must either have the CloudWatchFullAccess policy and iam:PassRole permission, or it must have the following list of permissions:
  • iam:PassRole
  • cloudwatch:PutMetricStream
  • cloudwatch:DeleteMetricStream
  • cloudwatch:GetMetricStream
  • cloudwatch:ListMetricStreams
  • cloudwatch:StartMetricStreams
  • cloudwatch:StopMetricStreams
  • iam:CreateRole
  • iam:PutRolePolicy

Cortex XCOR authentication

Create or use an API token for a Cortex XCOR restricted service account with write-only permission. For more information, see Create a restricted service account. You must also provide your Cortex XCOR organization name. This is the <ORG_NAME> segment in https://<ORG_NAME>.chronosphere.io.

AWS resources and IAM roles

The setup creates the following resources and Identity and Access Management (IAM) roles:

AWS resources

  • Amazon S3 bucket: Stores eligible failed deliveries from Amazon Data Firehose.
  • CloudWatch log group: Stores logs from the Firehose stream.
  • Amazon Data Firehose stream: Sends metrics to Cortex XCOR through an HTTP endpoint and stores eligible failed deliveries in the S3 bucket.
  • IAM role for Amazon Data Firehose: Grants the Firehose stream these permissions to access the S3 bucket and CloudWatch Logs:
    • s3:AbortMultipartUpload
    • s3:GetBucketLocation
    • s3:GetObject
    • s3:ListBucket
    • s3:ListBucketMultipartUploads
    • s3:PutObject
    • logs:PutLogEvents

IAM roles

The setup also creates an IAM role with these permissions so CloudWatch Metric Streams can publish data to the Firehose stream:
  • firehose:PutRecord
  • firehose:PutRecordBatch

Apply the configuration

Choose the Terraform or AWS Management Console setup.
  1. Set values for the following Terraform variables, and then apply the configuration.
    Terraform example

Validate CloudWatch Metric Stream ingestion

After setup, wait for data to arrive in Cortex XCOR. Validate ingestion with the operational dashboards in Cortex XCOR and the AWS Management Console.
The included dashboards query the default CloudWatch metric names. For tenants configured for Datadog-compatible naming, validate the snake_case metric names in Metrics Explorer.
The integration also includes a Metrics Overview dashboard that shows CloudWatch Metric Streams delivery volume and publish errors by account, region, and stream. It’s available from the Amazon Web Services integration in the Integrations Hub.

Validate the CloudWatch Metrics Ingestion & Health dashboard

The CloudWatch Metrics Ingestion & Health dashboard displays operational information about the health of your CloudWatch Metrics Streams integration with Cortex XCOR. The CloudWatch Metric Streams and Data Firehose panel groups rely on CloudWatch metrics sent from those services to Cortex XCOR. To populate these charts, include metrics from the AWS/Firehose and AWS/CloudWatch/MetricStreams namespaces in your CloudWatch Metrics Streams configuration.
  1. In Cortex XCOR, go to Dashboards.
  2. In the search bar, enter CloudWatch, and then select the CloudWatch Metrics Ingestion & Health dashboard. The metrics ingestion panel group displays information about the CloudWatch metrics Cortex XCOR received.
  3. Use the Data Firehose records received by Amazon Resource Name chart to confirm that the ingestion API received Data Firehose records.
  4. Use the CloudWatch metric updates received chart to confirm the number of CloudWatch metric updates Cortex XCOR extracts from the Data Firehose records.
  5. Confirm that the Transformed metrics chart shows no Rejected Data Points.
  6. Use the Unique time series by AWS metric namespace chart to confirm that metrics from the AWS namespaces you want are in the stream that Cortex XCOR received.

Validate the Metric Stream dashboard

In the AWS Management Console, use the Metric Updates chart for specific metrics to validate that metrics are streaming.
  1. In the AWS Management Console, go to CloudWatch > Metric Streams.
  2. Select chronosphere-cloudwatch-metric-stream to view the status and operational statistics.
  3. Verify that the Status is Running.
  4. Verify how many updates have been sent in the Metric Updates chart. If the stream is working, the chart should report a non-zero number of updates.
  5. Verify whether any errors were reported in the Errors chart. The value should be 0.

Validate the Amazon Data Firehose status

In the AWS Management Console, use the Firehose charts to validate that metrics are streaming.
  1. In the AWS Management Console, go to Amazon Data Firehose > Firehose Streams.
  2. Select chronosphere-cloudwatch-firehose to view status and operational statistics for the Amazon Data Firehose.
  3. The Incoming bytes, Incoming put requests, and Incoming records charts should all report non-zero values.
  4. The HTTP endpoint delivery success chart should report 100%.
  5. The Records delivered to HTTP endpoint chart should report a non-zero value.

Query metrics about AWS CloudWatch Metric Streams

Query metrics about AWS CloudWatch Metric Streams in Cortex XCOR. These metrics won’t appear in the Cortex XCOR Metrics Explorer if you’ve defined a drop rule to drop all AWS metrics. Modify the rule to allow some metrics, such as all metrics from the AWS/CloudWatch/MetricStreams namespace. For examples, see Control metric volume.
  1. In Cortex XCOR, go to Explorers > Metrics Explorer.
  2. In the query box, enter aws_cloudwatch_metricstreams to view a list of AWS metrics received from the CloudWatch Metric Stream.
  3. Select the metric you want to query to add it to the query prompt.
Write a query around the selected metric. For default metric naming, run this query to report the rate of CloudWatch metric updates:
For Datadog-compatible metric naming, run:

Metric semantics

Metric naming conventions

Metric names of ingested CloudWatch metrics in Cortex XCOR follow the prefix naming pattern:
  • <namespace>: The namespace is lowercased, and Cortex XCOR replaces all forward slash (/) and period (.) characters in the CloudWatch namespace with underscores (_). All AWS service namespaces follow the naming convention AWS/<ServiceName>, where <ServiceName> is replaced with the service name. In Cortex XCOR, the ingested metrics therefore begin with aws_<servicename>. For a list of AWS services and their respective namespaces, see AWS services that publish CloudWatch metrics. If you create custom metrics, the namespace you set for the metric becomes the metric name prefix in Cortex XCOR.
  • <MetricName>: By default, Cortex XCOR preserves the CloudWatch metric name’s case. Tenants configured for Datadog-compatible naming receive the metric name in snake_case.
  • <statistic>: Cortex XCOR appends the CloudWatch statistic’s name (count, sum, maximum, minimum, average). If you define additional statistics for a metric, Cortex XCOR appends the corresponding CloudWatch metric statistic name (pXX).
For examples, see Example metric names.

Metric labeling conventions

Cortex XCOR adds CloudWatch metric dimensions as labels to the time series following the pattern dimension_<DimensionName>. For examples, see Example metric names.

Stream resource attributes

CloudWatch Metric Streams include OpenTelemetry Protocol (OTLP) resource attributes, which Cortex XCOR merges into the time series. Cortex XCOR replaces periods (.) with underscores (_) in attribute key names. Amazon Data Firehose includes the following resource attributes in every post:
  • aws_exporter_arn: The Amazon Resource Name (ARN) of the CloudWatch Metric Stream, which uniquely identifies the metric writer instance.
  • cloud_account_id: The AWS account associated with the CloudWatch Metric Stream, such as 123456789012.
  • cloud_provider: The value is always aws.
  • cloud_region: The AWS region of the Amazon Data Firehose sending the stream, such as us-east-2.
For examples, see Example metric names.

Add custom resource attributes using stream parameters

Define custom key-value pairs as parameters for Amazon Data Firehose to include in each HTTP request. Cortex XCOR treats all additional parameters as resource attributes and merges them into the time series.
Your custom parameters take precedence over the default CloudWatch metrics resource attributes. Avoid parameter names that conflict with default CloudWatch attribute names.
Configure additional parameters through Terraform or the AWS Management Console.
To configure parameters using Terraform:
  1. Edit the Terraform module.
  2. In the aws_kinesis_firehose_delivery_stream resource definition, modify the request_configuration block to define additional common_attributes. The AWS Management Console names this setting Parameters, while the API name is common_attributes. The following example configures the content_encoding parameter as GZIP and defines two common_attributes, testname and testname2:
    Terraform example

Example metric names

Given a CloudWatch metric with the following attributes:
  • The namespace AWS/EBS (AWS Service)
  • The metric name VolumeReadBytes
  • The dimension VolumeId
  • The custom Firehose destination parameter environment
  • The CloudWatch metric resource attributes aws_exporter_arn, cloud_account_id, cloud_provider, and cloud_region
Cortex XCOR creates metrics with these names and labels:
Given a CloudWatch metric with the following attributes:
  • The namespace Buildkite (custom metrics)
  • The metric name RunningJobsCount
  • The custom Firehose destination parameter environment
  • No dimension
  • CloudWatch resource metric attributes aws_exporter_arn, cloud_account_id, cloud_provider, and cloud_region
Cortex XCOR creates metrics with these names and labels:

S3 daily storage metrics

Amazon S3 publishes its daily storage metrics, BucketSizeBytes and NumberOfObjects, to CloudWatch once a day, and they don’t arrive in Cortex XCOR through CloudWatch Metric Streams. Cortex XCOR can read them from CloudWatch in your AWS accounts instead, with the same AWS read role it uses to read your resource tags.
Cortex XCOR Support turns on these reads for your tenant. To request them, contact Cortex XCOR Support.
When Cortex XCOR Support turns on these reads, it also excludes both metrics from the aggregation recommendations for your tenant, because their values arrive after the window aggregation uses.

Prerequisites for S3 daily storage metrics

Cortex XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the Cortex XCOR AWS read role at template version 1.1.0 or later, which grants the read-only cloudwatch:ListMetrics and cloudwatch:GetMetricData actions these reads use.
  • AWS bills GetMetricData requests to your account per metric requested. Cortex XCOR requests each bucket’s metrics once or twice a day.
  • S3 publishes a bucket’s storage metrics only in the bucket’s own region. Cortex XCOR reads no buckets in a region it doesn’t read your resource tags in.

S3 daily storage metric names

Each time series has these labels:
  • bucketname: The bucket’s name.
  • storagetype: The storage class, such as StandardStorage or GlacierStorage. For aws_s3_number_of_objects_average, it’s always AllStorageTypes.
  • Namespace (always AWS/S3), cloud_provider (always aws), cloud_account_id and cloud_region.
  • aws_account_alias: The alias Cortex XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowed iam:ListAccountAliases, which read role template 1.2.0 grants.
  • The bucket’s resource tags, where Cortex XCOR reads them.
These time series don’t have dimension_* labels or the stream resource attributes that streamed metrics have.

S3 daily storage timestamps

Cortex XCOR writes each UTC day’s value with a timestamp one hour after the day ends. For example, the value for November 16 has the timestamp November 17, 01:00 UTC. CloudWatch timestamps the same value at the start of its day, so the CloudWatch console shows it a day earlier than Cortex XCOR does. Cortex XCOR reads each day once or twice between about 01:05 and 03:00 UTC the next day. After Cortex XCOR Support turns on these reads, the first value arrives between about 01:05 and 02:05 UTC the next day, for the day that was in progress when they were turned on. If they’re turned on before about 03:00 UTC, Cortex XCOR might also write the value for the day before. It doesn’t write values for earlier days. If AWS publishes a day’s value later than about 01:55 UTC the next day, the day might be missing. If AWS changes a day’s value after that, Cortex XCOR keeps the value it read.

Query S3 daily storage metrics

Each value has one data point a day, so queries need a lookback of more than a day:
  • To show each day’s value at the end of its day in a time series panel, and to line it up with other metrics, add offset -1h to the selector:
  • To show the latest value in a stat panel, a table, or a monitor, use last_over_time over 27 hours, with no offset. A day’s value can arrive up to about two hours after its timestamp, so a window of one day returns no data each day from 01:00 UTC until that day’s value arrives. An instant query without last_over_time returns no data for most of each day.

Validate S3 daily storage ingestion

After 03:10 UTC on the day after Cortex XCOR Support turns on these reads, run this query. It returns the number of buckets with a value in the last 27 hours:

Monitor drop rules

To view the number of data points per second that Cortex XCOR drops with the example CloudWatch Metric Streams drop rule, use the following PromQL query: