Before you begin
Some AWS services publish metrics at intervals longer than the two-hour ingestion age limit in Cortex XCOR. For example, Amazon S3 daily storage metrics can have timestamps older than two hours when CloudWatch delivers them. Before streaming a metric, verify that its publishing interval is shorter than the ingestion age limit. Cortex XCOR can read S3 daily storage metrics from CloudWatch instead: see S3 daily storage metrics.Control metric volume
CloudWatch Metric Streams generate metrics that consume Standard Metrics License capacity. Before configuring ingestion, optionally create a drop rule for unwanted metrics. The following Chronoctl YAML resource definition creates a drop rule that drops all metrics from CloudWatch Metric Streams except theAWS/CloudWatch/MetricStreams and AWS/Firehose namespaces required by
the validation dashboards.
Chronoctl example
Chronoctl example
- In the AWS Management Console, go to Streams.
- Edit the Metric Stream.
- Under Metrics to be streamed, include or exclude namespaces.
Configure CloudWatch Metric Streams
In each AWS account and region you want to collect metrics from, a CloudWatch Metric Stream sends data to Amazon Data Firehose. The Firehose stream forwards the data to the CloudWatch ingest endpoint in your Cortex XCOR tenant. Cortex XCOR processes the metrics and makes them available for queries, monitors, and dashboards.CloudWatch roles and permissions
To use CloudWatch Metric Streams in Cortex XCOR, you must configure a CloudWatch Metric Stream in each AWS account and region. The account you use to set up the CloudWatch Metric Stream must either have theCloudWatchFullAccess policy
and iam:PassRole permission, or it must have the following list of permissions:
iam:PassRolecloudwatch:PutMetricStreamcloudwatch:DeleteMetricStreamcloudwatch:GetMetricStreamcloudwatch:ListMetricStreamscloudwatch:StartMetricStreamscloudwatch:StopMetricStreamsiam:CreateRoleiam:PutRolePolicy
Cortex XCOR authentication
Create or use an API token for a Cortex XCOR restricted service account with write-only permission. For more information, see Create a restricted service account. You must also provide your Cortex XCOR organization name. This is the<ORG_NAME>
segment in https://<ORG_NAME>.chronosphere.io.
AWS resources and IAM roles
The setup creates the following resources and Identity and Access Management (IAM) roles:AWS resources
- Amazon S3 bucket: Stores eligible failed deliveries from Amazon Data Firehose.
- CloudWatch log group: Stores logs from the Firehose stream.
- Amazon Data Firehose stream: Sends metrics to Cortex XCOR through an HTTP endpoint and stores eligible failed deliveries in the S3 bucket.
- IAM role for Amazon Data Firehose: Grants the Firehose stream these permissions
to access the S3 bucket and CloudWatch Logs:
s3:AbortMultipartUploads3:GetBucketLocations3:GetObjects3:ListBuckets3:ListBucketMultipartUploadss3:PutObjectlogs:PutLogEvents
IAM roles
The setup also creates an IAM role with these permissions so CloudWatch Metric Streams can publish data to the Firehose stream:firehose:PutRecordfirehose:PutRecordBatch
Apply the configuration
Choose the Terraform or AWS Management Console setup.- Terraform
- AWS Management Console
-
Set values for the following
Terraform
variables, and then apply the configuration.
Terraform example
Validate CloudWatch Metric Stream ingestion
After setup, wait for data to arrive in Cortex XCOR. Validate ingestion with the operational dashboards in Cortex XCOR and the AWS Management Console.The included dashboards query the default CloudWatch metric names. For tenants
configured for Datadog-compatible naming, validate the
snake_case metric names in
Metrics Explorer.Validate the CloudWatch Metrics Ingestion & Health dashboard
The CloudWatch Metrics Ingestion & Health dashboard displays operational information about the health of your CloudWatch Metrics Streams integration with Cortex XCOR. The CloudWatch Metric Streams and Data Firehose panel groups rely on CloudWatch metrics sent from those services to Cortex XCOR. To populate these charts, include metrics from theAWS/Firehose and
AWS/CloudWatch/MetricStreams namespaces in your CloudWatch Metrics Streams
configuration.
- In Cortex XCOR, go to Dashboards.
- In the search bar, enter CloudWatch, and then select the CloudWatch Metrics Ingestion & Health dashboard. The metrics ingestion panel group displays information about the CloudWatch metrics Cortex XCOR received.
- Use the Data Firehose records received by Amazon Resource Name chart to confirm that the ingestion API received Data Firehose records.
- Use the CloudWatch metric updates received chart to confirm the number of CloudWatch metric updates Cortex XCOR extracts from the Data Firehose records.
- Confirm that the Transformed metrics chart shows no Rejected Data Points.
- Use the Unique time series by AWS metric namespace chart to confirm that metrics from the AWS namespaces you want are in the stream that Cortex XCOR received.
Validate the Metric Stream dashboard
In the AWS Management Console, use the Metric Updates chart for specific metrics to validate that metrics are streaming.- In the AWS Management Console, go to CloudWatch > Metric Streams.
- Select
chronosphere-cloudwatch-metric-streamto view the status and operational statistics. - Verify that the Status is
Running. - Verify how many updates have been sent in the Metric Updates chart. If the stream is working, the chart should report a non-zero number of updates.
- Verify whether any errors were reported in the Errors chart. The value should be 0.
Validate the Amazon Data Firehose status
In the AWS Management Console, use the Firehose charts to validate that metrics are streaming.- In the AWS Management Console, go to Amazon Data Firehose > Firehose Streams.
- Select
chronosphere-cloudwatch-firehoseto view status and operational statistics for the Amazon Data Firehose. - The Incoming bytes, Incoming put requests, and Incoming records charts should all report non-zero values.
- The HTTP endpoint delivery success chart should report 100%.
- The Records delivered to HTTP endpoint chart should report a non-zero value.
Query metrics about AWS CloudWatch Metric Streams
Query metrics about AWS CloudWatch Metric Streams in Cortex XCOR. These metrics won’t appear in the Cortex XCOR Metrics Explorer if you’ve defined a drop rule to drop all AWS metrics. Modify the rule to allow some metrics, such as all metrics from theAWS/CloudWatch/MetricStreams namespace. For
examples, see Control metric volume.
- In Cortex XCOR, go to Explorers > Metrics Explorer.
- In the query box, enter
aws_cloudwatch_metricstreamsto view a list of AWS metrics received from the CloudWatch Metric Stream. - Select the metric you want to query to add it to the query prompt.
Metric semantics
Metric naming conventions
Metric names of ingested CloudWatch metrics in Cortex XCOR follow the prefix naming pattern:-
<namespace>: The namespace is lowercased, and Cortex XCOR replaces all forward slash (/) and period (.) characters in the CloudWatch namespace with underscores (_). All AWS service namespaces follow the naming conventionAWS/<ServiceName>, where<ServiceName>is replaced with the service name. In Cortex XCOR, the ingested metrics therefore begin withaws_<servicename>. For a list of AWS services and their respective namespaces, see AWS services that publish CloudWatch metrics. If you create custom metrics, the namespace you set for the metric becomes the metric name prefix in Cortex XCOR. -
<MetricName>: By default, Cortex XCOR preserves the CloudWatch metric name’s case. Tenants configured for Datadog-compatible naming receive the metric name insnake_case. -
<statistic>: Cortex XCOR appends the CloudWatch statistic’s name (count,sum,maximum,minimum,average). If you define additional statistics for a metric, Cortex XCOR appends the corresponding CloudWatch metric statistic name (pXX).
Metric labeling conventions
Cortex XCOR adds CloudWatch metric dimensions as labels to the time series following the patterndimension_<DimensionName>. For examples, see
Example metric names.
Stream resource attributes
CloudWatch Metric Streams include OpenTelemetry Protocol (OTLP) resource attributes, which Cortex XCOR merges into the time series. Cortex XCOR replaces periods (.) with underscores (_) in attribute key names.
Amazon Data Firehose includes the following resource attributes in every post:
aws_exporter_arn: The Amazon Resource Name (ARN) of the CloudWatch Metric Stream, which uniquely identifies the metric writer instance.cloud_account_id: The AWS account associated with the CloudWatch Metric Stream, such as123456789012.cloud_provider: The value is alwaysaws.cloud_region: The AWS region of the Amazon Data Firehose sending the stream, such asus-east-2.
Add custom resource attributes using stream parameters
Define custom key-value pairs as parameters for Amazon Data Firehose to include in each HTTP request. Cortex XCOR treats all additional parameters as resource attributes and merges them into the time series.Your custom parameters take precedence over the default CloudWatch metrics resource
attributes. Avoid parameter names that conflict with default CloudWatch attribute
names.
- Terraform
- AWS Management Console
To configure parameters using Terraform:
- Edit the Terraform module.
-
In the
aws_kinesis_firehose_delivery_streamresource definition, modify therequest_configurationblock to define additionalcommon_attributes. The AWS Management Console names this setting Parameters, while the API name iscommon_attributes. The following example configures thecontent_encodingparameter asGZIPand defines twocommon_attributes,testnameandtestname2:Terraform example
Example metric names
Given a CloudWatch metric with the following attributes:- The namespace
AWS/EBS(AWS Service) - The metric name
VolumeReadBytes - The dimension
VolumeId - The custom Firehose destination parameter
environment - The CloudWatch metric resource attributes
aws_exporter_arn,cloud_account_id,cloud_provider, andcloud_region
- The namespace
Buildkite(custom metrics) - The metric name
RunningJobsCount - The custom Firehose destination parameter
environment - No dimension
- CloudWatch resource metric attributes
aws_exporter_arn,cloud_account_id,cloud_provider, andcloud_region
S3 daily storage metrics
Amazon S3 publishes its daily storage metrics,BucketSizeBytes and
NumberOfObjects, to CloudWatch once a day, and they don’t arrive in Cortex
XCOR through CloudWatch Metric Streams. Cortex XCOR can read them from
CloudWatch in your AWS accounts instead, with the same AWS read role it uses to
read your resource tags.
Cortex XCOR Support turns on these reads for your tenant. To request them, contact
Cortex XCOR Support.
Prerequisites for S3 daily storage metrics
Cortex XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the Cortex XCOR AWS read role at template version 1.1.0 or later, which grants the read-onlycloudwatch:ListMetrics and cloudwatch:GetMetricData actions these reads use.
- AWS bills
GetMetricDatarequests to your account per metric requested. Cortex XCOR requests each bucket’s metrics once or twice a day. - S3 publishes a bucket’s storage metrics only in the bucket’s own region. Cortex XCOR reads no buckets in a region it doesn’t read your resource tags in.
S3 daily storage metric names
Each time series has these labels:
bucketname: The bucket’s name.storagetype: The storage class, such asStandardStorageorGlacierStorage. Foraws_s3_number_of_objects_average, it’s alwaysAllStorageTypes.Namespace(alwaysAWS/S3),cloud_provider(alwaysaws),cloud_account_idandcloud_region.aws_account_alias: The alias Cortex XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowediam:ListAccountAliases, which read role template 1.2.0 grants.- The bucket’s resource tags, where Cortex XCOR reads them.
dimension_* labels or the stream resource
attributes that streamed metrics have.
S3 daily storage timestamps
Cortex XCOR writes each UTC day’s value with a timestamp one hour after the day ends. For example, the value for November 16 has the timestamp November 17, 01:00 UTC. CloudWatch timestamps the same value at the start of its day, so the CloudWatch console shows it a day earlier than Cortex XCOR does. Cortex XCOR reads each day once or twice between about 01:05 and 03:00 UTC the next day. After Cortex XCOR Support turns on these reads, the first value arrives between about 01:05 and 02:05 UTC the next day, for the day that was in progress when they were turned on. If they’re turned on before about 03:00 UTC, Cortex XCOR might also write the value for the day before. It doesn’t write values for earlier days. If AWS publishes a day’s value later than about 01:55 UTC the next day, the day might be missing. If AWS changes a day’s value after that, Cortex XCOR keeps the value it read.Query S3 daily storage metrics
Each value has one data point a day, so queries need a lookback of more than a day:-
To show each day’s value at the end of its day in a time series panel, and to
line it up with other metrics, add
offset -1hto the selector: -
To show the latest value in a stat panel, a table, or a monitor, use
last_over_timeover 27 hours, with no offset. A day’s value can arrive up to about two hours after its timestamp, so a window of one day returns no data each day from 01:00 UTC until that day’s value arrives. An instant query withoutlast_over_timereturns no data for most of each day.