> ## Documentation Index > Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Ingest AWS logs > Learn how to send AWS logs to Palo Alto Networks Cortex XCOR. Palo Alto Networks Cortex XCOR receives Amazon Web Services (AWS) logs through [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html). In each AWS account and region, a Firehose stream sends logs to the AWS logs ingest endpoint in your Cortex XCOR tenant. CloudWatch Logs reach the stream through a subscription, and some AWS services send their logs to the stream directly. ## What it collects * CloudWatch Logs: Logs that AWS services and your applications write to CloudWatch log groups. Use an account-level subscription policy to cover every log group in a region, including new ones, or add a subscription filter to each log group. Subscriptions cover only log groups in the Standard log class. CloudTrail can also send its events to a log group. * Logs sent straight to Firehose: Services such as Virtual Private Cloud (VPC) Flow Logs and Web Application Firewall (WAF) can deliver logs to a Firehose stream without a log group. * Not collected: Logs that AWS writes only to Amazon S3, such as Classic Load Balancer access logs. ## How it works The following diagram shows how AWS log sources reach Cortex XCOR and where Firehose stores eligible failed deliveries. ```mermaid actions={false} theme={null} graph LR accTitle: AWS log delivery accDescr: CloudWatch Logs, VPC Flow Logs, and AWS WAF send logs through Amazon Data Firehose to the Cortex XCOR AWS logs ingest endpoint. Eligible failed deliveries go to Amazon S3. subgraph AWS-account subgraph Region lg[CloudWatch log groups] vl[VPC Flow Logs, WAF] adf[Amazon Data Firehose] s3[S3 bucket for eligible failed data] lg -- subscription --> adf vl --> adf adf -. retryable failures .-> s3 end end subgraph op[Cortex XCOR] subgraph Tenant ep[AWS logs ingest endpoint] end end adf --> ep classDef greenFill stroke:#60E684,fill:#2FBF71,stroke-width:2px,color:#FFFFFF class ep,op greenFill; classDef blueFill fill:#000243,stroke-width:2px,color:#FFFFFF class lg,vl,adf,s3,AWS-account blueFill; classDef orange stroke-width:2px,color:#FFFFFF,stroke:#FF9D1C class lg,vl,adf,s3,AWS-account orange ``` ## Before you begin * AWS permissions: The identity that runs the setup needs the permissions listed in [Required permissions](#required-permissions). * A Cortex XCOR API token: Create or use an API token for a restricted service account with write-only permission. For more information, see [Create a restricted service account](https://docs-xcor.paloaltonetworks.com/administer/accounts-teams/service-accounts.md#create-a-restricted-service-account). - Your organization name: The `` segment in `https://.chronosphere.io`. * Existing subscriptions: Each account and region allows one account-level subscription policy, and each log group allows five subscription filters of its own, which the account-level policy doesn't count against. If another tool already uses the account-level policy, subscribe log groups individually. To inspect the account policies, run: ```shell theme={null} aws logs describe-account-policies --policy-type SUBSCRIPTION_FILTER_POLICY --region ``` ### Required permissions To run the Terraform or CloudFormation setup in one account, the identity that runs it needs the following permissions. These permissions create and remove the setup resources. StackSets and the console need additional permissions. * Amazon Data Firehose * Actions: `CreateDeliveryStream`, `DeleteDeliveryStream`, `DescribeDeliveryStream`, `UpdateDestination`, `ListTagsForDeliveryStream`, `TagDeliveryStream`, `UntagDeliveryStream` * Resource: The stream * Amazon S3 * Actions: `CreateBucket`, `DeleteBucket`, `PutLifecycleConfiguration`, `PutBucketTagging`, `ListBucket`, `Get*`, and `DeleteObject` to empty the bucket when you remove it * Resource: The failed-data bucket * CloudWatch Logs * Actions: `CreateLogGroup`, `DeleteLogGroup`, `PutRetentionPolicy`, `CreateLogStream`, `DeleteLogStream`, `DescribeLogGroups`, `DescribeLogStreams`, `ListTagsForResource`, `TagResource`, `PutAccountPolicy`, `DeleteAccountPolicy`, `DescribeAccountPolicies`, `PutSubscriptionFilter`, `DeleteSubscriptionFilter`, `DescribeSubscriptionFilters` * Resource: All resources, because account policies have no resource ARN * IAM * Actions: `CreateRole`, `DeleteRole`, `GetRole`, `PutRolePolicy`, `GetRolePolicy`, `DeleteRolePolicy`, `ListRolePolicies`, `ListAttachedRolePolicies`, `ListInstanceProfilesForRole`, `PassRole` * Resources: The two roles the setup creates * AWS Secrets Manager * Actions: `CreateSecret`, `DeleteSecret`, `DescribeSecret`, `UpdateSecret`, `GetSecretValue`, `TagResource`, `UntagResource` * Resource: The token secret * CloudFormation, for the CloudFormation tab only * Actions: `CreateChangeSet`, `DescribeChangeSet`, `ExecuteChangeSet`, `DeleteChangeSet`, `DescribeStacks`, `DescribeStackEvents`, `DeleteStack`, `GetTemplateSummary` * Resource: The stack The setup creates two roles: * Firehose role: Trusted by Firehose in your account. It can write to the failed-data bucket, read the token secret, and write the stream's error log. * Subscription role: Trusted by CloudWatch Logs in your account. It can put records into the stream. The setup has no access to your existing buckets or other resources, and it doesn't send logs written before it's in place. ## Firehose settings Use these settings regardless of how you create the stream: Firehose bills each CloudWatch Logs record at its compressed size, rounded up to 5 KB. Log groups that send few events per batch cost more per log. Firehose also bills data transfer for every delivery attempt, including retries. While deliveries fail, data transfer grows with each retry until the retry window ends. * Source: `Direct PUT`. * Destination: `HTTP Endpoint`. * HTTP endpoint URL: `https://.chronosphere.io/api/v1/data/logs/ingest/integrations/aws`. * Authentication: An AWS Secrets Manager secret containing `{"api_key":""}`. Firehose picks up a rotated token within about 10 minutes. Using the API token as the endpoint access key also works, but stores it in the stream settings. * Content encoding: `GZIP` for smaller requests and lower data transfer costs. * Retry duration: `7200` seconds, the longest Firehose allows. Records that still fail go to the S3 bucket. * Buffer size: `1` MiB. Keep this value at 4 MiB or less. Firehose treats the size as a hint and encodes each record in a text representation larger than the source record. Cortex XCOR rejects requests over 16 MiB with a `413`, and Firehose drops a batch that gets a `413` without retrying it or backing it up. * Buffer interval: `60` seconds. * Decompression and message extraction: Off. Cortex XCOR decodes CloudWatch Logs records. Message extraction removes the log group and log stream used to identify the service. * Source record backup in Amazon S3: `Failed data only`. * Backup bucket lifecycle: Expire objects after `14` days. Cortex XCOR drops retried logs more than 24 hours old. * Parameters: None. For the exception for VPC Flow Logs in a custom format, see [Send logs straight to Firehose](#send-logs-straight-to-firehose). ## Set up log delivery Set up delivery in each account and region you want logs from. > [tab] Terraform > > 1. Store your API token in AWS Secrets Manager in the region you're setting up. Note > the secret's ARN. > > Read the token without echoing it, so it stays out of your shell history. Run the > `read` command on its own: > > ```shell theme={null} > printf 'API token: '; read -rs API_TOKEN; echo > ``` > > Then create the secret: > > ```shell theme={null} > if [ -n "$API_TOKEN" ]; then > printf '{"api_key":"%s"}' "$API_TOKEN" | > aws secretsmanager create-secret --region \ > --name cortex-xcor/aws-logs-api-token \ > --secret-string file:///dev/stdin > else > echo 'API_TOKEN is empty: run the read command first' > fi > unset API_TOKEN > ``` > > 2. The configuration was tested with version 6 of the `aws` provider. It also needs > the `time` provider. Set the region on the `aws` provider: > > ```terraform theme={null} > terraform { > required_providers { > aws = { source = "hashicorp/aws", version = ">= 6.0" } > time = { source = "hashicorp/time" } > } > } > > provider "aws" { > region = "" > } > ``` > > 3. Set values for these variables, and apply the configuration in each account and > region. It creates the Firehose stream, S3 bucket for failed data with a lifecycle > rule, and IAM roles. Unless you turn it off, it also creates an account-level > subscription policy that sends every log group in the region except the stream's > own error logs. > > ```terraform expandable Terraform example icon="square-terminal" theme={null} > # Variables > > variable "xcor_org_name" { > type = string > description = "The name of your Cortex XCOR organization, which is the subdomain name before .chronosphere.io." > } > > variable "api_token_secret_arn" { > type = string > description = "The ARN of the Secrets Manager secret holding {\"api_key\":\"\"}, in the stream's region." > } > > variable "stream_name" { > type = string > description = "The name of the Firehose stream." > default = "cortex-xcor-logs" > } > > variable "create_account_policy" { > type = bool > description = "Set to false if this account and region already has an account-level subscription policy." > default = true > } > > variable "failed_data_expiration_days" { > type = number > description = "Days to keep records Firehose couldn't deliver." > default = 14 > } > > locals { > tags = { > "cortex-xcor:template" = "aws-logs-firehose" > "cortex-xcor:template-version" = "1.0.0" > } > } > > data "aws_caller_identity" "current" {} > > # S3 bucket and log group for failed deliveries > > resource "aws_s3_bucket" "failed_data" { > bucket_prefix = "cortex-xcor-logs-failed-" > tags = local.tags > } > > resource "aws_s3_bucket_lifecycle_configuration" "failed_data" { > bucket = aws_s3_bucket.failed_data.id > rule { > id = "expire-failed-data" > status = "Enabled" > filter {} > expiration { > days = var.failed_data_expiration_days > } > } > } > > resource "aws_cloudwatch_log_group" "firehose" { > name = "/aws/kinesisfirehose/${var.stream_name}" > retention_in_days = 14 > } > > resource "aws_cloudwatch_log_stream" "firehose" { > name = "DestinationDelivery" > log_group_name = aws_cloudwatch_log_group.firehose.name > } > > # Firehose stream > > data "aws_iam_policy_document" "firehose_trust" { > statement { > actions = ["sts:AssumeRole"] > principals { > type = "Service" > identifiers = ["firehose.amazonaws.com"] > } > condition { > test = "StringEquals" > variable = "aws:SourceAccount" > values = [data.aws_caller_identity.current.account_id] > } > } > } > > data "aws_iam_policy_document" "firehose_access" { > statement { > actions = [ > "s3:AbortMultipartUpload", > "s3:GetBucketLocation", > "s3:GetObject", > "s3:ListBucket", > "s3:ListBucketMultipartUploads", > "s3:PutObject", > ] > resources = [aws_s3_bucket.failed_data.arn, "${aws_s3_bucket.failed_data.arn}/*"] > } > statement { > actions = ["secretsmanager:GetSecretValue"] > resources = [var.api_token_secret_arn] > } > statement { > actions = ["logs:PutLogEvents"] > resources = ["${aws_cloudwatch_log_group.firehose.arn}:*"] > } > } > > resource "aws_iam_role" "firehose" { > name_prefix = "cortex-xcor-logs-firehose-" > assume_role_policy = data.aws_iam_policy_document.firehose_trust.json > } > > resource "aws_iam_role_policy" "firehose" { > role = aws_iam_role.firehose.id > policy = data.aws_iam_policy_document.firehose_access.json > } > > resource "aws_kinesis_firehose_delivery_stream" "logs" { > name = var.stream_name > destination = "http_endpoint" > tags = local.tags > > http_endpoint_configuration { > name = "Cortex XCOR" > url = "https://${var.xcor_org_name}.chronosphere.io/api/v1/data/logs/ingest/integrations/aws" > buffering_size = 1 # MiB > buffering_interval = 60 # seconds > retry_duration = 7200 # seconds > role_arn = aws_iam_role.firehose.arn > s3_backup_mode = "FailedDataOnly" > > secrets_manager_configuration { > enabled = true > secret_arn = var.api_token_secret_arn > role_arn = aws_iam_role.firehose.arn > } > > s3_configuration { > role_arn = aws_iam_role.firehose.arn > bucket_arn = aws_s3_bucket.failed_data.arn > buffering_size = 10 # MiB > buffering_interval = 300 # seconds > compression_format = "GZIP" > } > > request_configuration { > content_encoding = "GZIP" > } > > cloudwatch_logging_options { > enabled = true > log_group_name = aws_cloudwatch_log_group.firehose.name > log_stream_name = aws_cloudwatch_log_stream.firehose.name > } > } > } > > # CloudWatch Logs subscription > > data "aws_iam_policy_document" "subscription_trust" { > statement { > actions = ["sts:AssumeRole"] > principals { > type = "Service" > identifiers = ["logs.amazonaws.com"] > } > condition { > test = "StringLike" > variable = "aws:SourceArn" > values = ["arn:aws:logs:*:${data.aws_caller_identity.current.account_id}:*"] > } > } > } > > data "aws_iam_policy_document" "subscription_access" { > statement { > actions = ["firehose:PutRecord", "firehose:PutRecordBatch"] > resources = [aws_kinesis_firehose_delivery_stream.logs.arn] > } > } > > resource "aws_iam_role" "subscription" { > name_prefix = "cortex-xcor-logs-subscription-" > assume_role_policy = data.aws_iam_policy_document.subscription_trust.json > } > > resource "aws_iam_role_policy" "subscription" { > role = aws_iam_role.subscription.id > policy = data.aws_iam_policy_document.subscription_access.json > } > > # CloudWatch Logs sends a test record when the policy is created, and a new > # role's permissions take a few seconds to take effect. > resource "time_sleep" "subscription_role" { > depends_on = [aws_iam_role_policy.subscription] > create_duration = "30s" > } > > resource "aws_cloudwatch_log_account_policy" "logs" { > count = var.create_account_policy ? 1 : 0 > depends_on = [time_sleep.subscription_role] > > policy_name = "cortex-xcor-logs" > policy_type = "SUBSCRIPTION_FILTER_POLICY" > scope = "ALL" > # The stream's own error logs are excluded, or they'd loop back into it. > selection_criteria = "LogGroupName NOT IN [\"${aws_cloudwatch_log_group.firehose.name}\"]" > policy_document = jsonencode({ > DestinationArn = aws_kinesis_firehose_delivery_stream.logs.arn > RoleArn = aws_iam_role.subscription.arn > FilterPattern = "" > }) > } > > # Used to subscribe log groups individually > > output "stream_arn" { > value = aws_kinesis_firehose_delivery_stream.logs.arn > } > > output "subscription_role_arn" { > value = aws_iam_role.subscription.arn > } > ``` > > 4. If you set `create_account_policy = false`, follow > [Subscribe individual log groups](#subscribe-individual-log-groups) with the > `stream_arn` and `subscription_role_arn` outputs. > [tab] AWS Management Console > > 1. In [AWS Secrets Manager](https://console.aws.amazon.com/secretsmanager/), store a > secret of type **Other type of secret** with the key `api_key` and your API token > as the value. Name it `cortex-xcor/aws-logs-api-token`. > 2. Open the > [Amazon Data Firehose console](https://console.aws.amazon.com/firehose/) and select > **Create Firehose stream**. > 3. Set the following options: > * **Source:** `Direct PUT` > * **Destination:** `HTTP Endpoint` > * **Firehose stream name:** `cortex-xcor-logs` > * **Transform records:** If the console offers **Decompress source records from > Amazon CloudWatch Logs**, leave decompression and message extraction off. > * **Destination settings:** > * **HTTP endpoint name:** `Cortex XCOR` > * **HTTP endpoint URL:** > `https://.chronosphere.io/api/v1/data/logs/ingest/integrations/aws`. > Replace `` with your organization name. > * **Authentication:** `Use AWS Secrets Manager`, and select the secret you created. > * **Content encoding:** `GZIP` > * **Retry duration:** `7200 seconds` > * Expand **Buffer hints**. > * **Buffer size:** `1 MiB` > * **Buffer interval:** `60 seconds` > * **Backup settings:** > * **Source record backup in Amazon S3:** `Failed data only` > * **S3 backup bucket:** Create a bucket for this stream with default settings. In > the bucket's **Management** tab, add a lifecycle rule that expires objects > after 14 days. Don't add that rule to a bucket that holds other data. > * Expand **Buffer hints, compression and encryption**, and set > **Compression for data records** to `GZIP`. > * **Advanced settings:** Under **Service access**, let the console create an IAM > role. Confirm that the role allows `secretsmanager:GetSecretValue` on the secret. > 4. Select **Create Firehose stream**. > 5. In the [IAM console](https://console.aws.amazon.com/iam/), create a role that > CloudWatch Logs can assume. Set the trusted entity to `logs.amazonaws.com`, with > the condition that `aws:SourceArn` is like > `arn:aws:logs:::*`. Grant the role `firehose:PutRecord` and > `firehose:PutRecordBatch` on the stream. > 6. Use the AWS Command Line Interface to subscribe log groups with an > account-level subscription policy. Exclude the stream's own error log group to > prevent its logs from looping back into the stream: > > ```shell theme={null} > aws logs put-account-policy --region \ > --policy-name cortex-xcor-logs \ > --policy-type SUBSCRIPTION_FILTER_POLICY \ > --scope ALL \ > --selection-criteria 'LogGroupName NOT IN ["/aws/kinesisfirehose/cortex-xcor-logs"]' \ > --policy-document '{"DestinationArn":"","RoleArn":"","FilterPattern":""}' > ``` > > If the account already has an account-level subscription policy, subscribe log > groups individually. > [tab] CloudFormation > > 1. Store your API token in AWS Secrets Manager in the region you're setting up. Note > the secret's ARN. > > Read the token without echoing it, so it stays out of your shell history. Run the > `read` command on its own: > > ```shell theme={null} > printf 'API token: '; read -rs API_TOKEN; echo > ``` > > Then create the secret: > > ```shell theme={null} > if [ -n "$API_TOKEN" ]; then > printf '{"api_key":"%s"}' "$API_TOKEN" | > aws secretsmanager create-secret --region \ > --name cortex-xcor/aws-logs-api-token \ > --secret-string file:///dev/stdin > else > echo 'API_TOKEN is empty: run the read command first' > fi > unset API_TOKEN > ``` > > 2. Save this template as `cortex-xcor-aws-logs.yaml`. It creates the Firehose stream, > S3 bucket for failed data with a lifecycle rule, and IAM roles. Unless you turn it > off, it also creates an account-level subscription policy that sends every log > group in the region except the stream's own error logs. > > ```yaml expandable CloudFormation template icon="file-code" theme={null} > AWSTemplateFormatVersion: "2010-09-09" > Description: Sends CloudWatch Logs in this account and region to Cortex XCOR through Amazon Data Firehose. Template version 1.1.0. > > Parameters: > OrgName: > Type: String > Description: Your Cortex XCOR organization name, the subdomain before .chronosphere.io. > ApiTokenSecretArn: > Type: String > Default: "" > Description: ARN of the Secrets Manager secret holding {"api_key":""}, in this stack's region. Leave empty to set ApiToken instead. > ApiToken: > Type: String > Default: "" > NoEcho: true > Description: The API token, for a stack that creates its own secret, such as a StackSet instance. Leave empty to set ApiTokenSecretArn instead. > StreamName: > Type: String > Default: cortex-xcor-logs > CreateAccountPolicy: > Type: String > Default: "true" > AllowedValues: ["true", "false"] > Description: Set to false if this account and region already has an account-level subscription policy. > FailedDataExpirationDays: > Type: Number > Default: 14 > MinValue: 1 > Description: Days to keep records Firehose couldn't deliver. > > Rules: > OneTokenSource: > Assertions: > - Assert: !Or [!Not [!Equals [!Ref ApiTokenSecretArn, ""]], !Not [!Equals [!Ref ApiToken, ""]]] > AssertDescription: Set ApiTokenSecretArn or ApiToken. > - Assert: !Or [!Equals [!Ref ApiTokenSecretArn, ""], !Equals [!Ref ApiToken, ""]] > AssertDescription: Set only one of ApiTokenSecretArn and ApiToken. > > Conditions: > CreatePolicy: !Equals [!Ref CreateAccountPolicy, "true"] > CreateSecret: !Equals [!Ref ApiTokenSecretArn, ""] > > Resources: > ApiTokenSecret: > Type: AWS::SecretsManager::Secret > Condition: CreateSecret > Properties: > Description: Cortex XCOR API token for the AWS logs Firehose stream > SecretString: !Sub '{"api_key":"${ApiToken}"}' > > FailedDataBucket: > Type: AWS::S3::Bucket > DeletionPolicy: RetainExceptOnCreate > UpdateReplacePolicy: Retain > Properties: > LifecycleConfiguration: > Rules: > - Id: expire-failed-data > Status: Enabled > ExpirationInDays: !Ref FailedDataExpirationDays > Tags: > - { Key: "cortex-xcor:template", Value: aws-logs-firehose } > - { Key: "cortex-xcor:template-version", Value: "1.1.0" } > > FirehoseLogGroup: > Type: AWS::Logs::LogGroup > Properties: > LogGroupName: !Sub /aws/kinesisfirehose/${StreamName} > RetentionInDays: 14 > > FirehoseLogStream: > Type: AWS::Logs::LogStream > Properties: > LogGroupName: !Ref FirehoseLogGroup > LogStreamName: DestinationDelivery > > FirehoseRole: > Type: AWS::IAM::Role > Properties: > AssumeRolePolicyDocument: > Version: "2012-10-17" > Statement: > - Effect: Allow > Principal: { Service: firehose.amazonaws.com } > Action: sts:AssumeRole > Condition: > StringEquals: > aws:SourceAccount: !Ref AWS::AccountId > Policies: > - PolicyName: delivery > PolicyDocument: > Version: "2012-10-17" > Statement: > - Effect: Allow > Action: [s3:AbortMultipartUpload, s3:GetBucketLocation, s3:GetObject, s3:ListBucket, s3:ListBucketMultipartUploads, s3:PutObject] > Resource: [!GetAtt FailedDataBucket.Arn, !Sub "${FailedDataBucket.Arn}/*"] > - Effect: Allow > Action: secretsmanager:GetSecretValue > Resource: !If [CreateSecret, !Ref ApiTokenSecret, !Ref ApiTokenSecretArn] > - Effect: Allow > Action: logs:PutLogEvents > Resource: !GetAtt FirehoseLogGroup.Arn > > DeliveryStream: > Type: AWS::KinesisFirehose::DeliveryStream > Properties: > DeliveryStreamName: !Ref StreamName > DeliveryStreamType: DirectPut > Tags: > - { Key: "cortex-xcor:template", Value: aws-logs-firehose } > - { Key: "cortex-xcor:template-version", Value: "1.1.0" } > HttpEndpointDestinationConfiguration: > EndpointConfiguration: > Name: Cortex XCOR > Url: !Sub https://${OrgName}.chronosphere.io/api/v1/data/logs/ingest/integrations/aws > SecretsManagerConfiguration: > Enabled: true > SecretARN: !If [CreateSecret, !Ref ApiTokenSecret, !Ref ApiTokenSecretArn] > RoleARN: !GetAtt FirehoseRole.Arn > RequestConfiguration: > ContentEncoding: GZIP > BufferingHints: > SizeInMBs: 1 > IntervalInSeconds: 60 > RetryOptions: > DurationInSeconds: 7200 > RoleARN: !GetAtt FirehoseRole.Arn > S3BackupMode: FailedDataOnly > S3Configuration: > BucketARN: !GetAtt FailedDataBucket.Arn > RoleARN: !GetAtt FirehoseRole.Arn > CompressionFormat: GZIP > BufferingHints: > SizeInMBs: 10 > IntervalInSeconds: 300 > CloudWatchLoggingOptions: > Enabled: true > LogGroupName: !Ref FirehoseLogGroup > LogStreamName: !Ref FirehoseLogStream > > SubscriptionRole: > Type: AWS::IAM::Role > Properties: > AssumeRolePolicyDocument: > Version: "2012-10-17" > Statement: > - Effect: Allow > Principal: { Service: logs.amazonaws.com } > Action: sts:AssumeRole > Condition: > StringLike: { "aws:SourceArn": !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*" } > Policies: > - PolicyName: put-to-firehose > PolicyDocument: > Version: "2012-10-17" > Statement: > - Effect: Allow > Action: [firehose:PutRecord, firehose:PutRecordBatch] > Resource: !GetAtt DeliveryStream.Arn > > AccountPolicy: > Type: AWS::Logs::AccountPolicy > Condition: CreatePolicy > Properties: > PolicyName: cortex-xcor-logs > PolicyType: SUBSCRIPTION_FILTER_POLICY > Scope: ALL > # The stream's own error logs are excluded, or they'd loop back into it. > SelectionCriteria: !Sub 'LogGroupName NOT IN ["/aws/kinesisfirehose/${StreamName}"]' > PolicyDocument: !Sub '{"DestinationArn":"${DeliveryStream.Arn}","RoleArn":"${SubscriptionRole.Arn}","FilterPattern":""}' > > Outputs: > DeliveryStreamArn: > Value: !GetAtt DeliveryStream.Arn > SubscriptionRoleArn: > Value: !GetAtt SubscriptionRole.Arn > ``` > > 3. Deploy the stack: > > ```shell theme={null} > aws cloudformation deploy --region \ > --template-file cortex-xcor-aws-logs.yaml \ > --stack-name cortex-xcor-aws-logs \ > --capabilities CAPABILITY_IAM \ > --parameter-overrides OrgName= ApiTokenSecretArn= > ``` > > `deploy` keeps a parameter's previous value when you leave it out. To switch an > existing stack between `ApiToken` and `ApiTokenSecretArn`, pass the one you stop > using as empty, such as `ApiToken=""`. > > 4. If you set `CreateAccountPolicy=false`, follow > [Subscribe individual log groups](#subscribe-individual-log-groups) with the > `DeliveryStreamArn` and `SubscriptionRoleArn` stack outputs. ### Set up many accounts and regions Set up one stream in each account and region you want logs from. Each stream reports its own deliveries, so a problem appears in the account and region it affects. To deploy to many accounts in an organization in AWS Organizations, run the CloudFormation template as a StackSet with service-managed permissions. Pass the API token as `ApiToken` instead of `ApiTokenSecretArn`. Each stack instance then creates its own secret, so accounts that automatic deployment adds later need no extra setup, unless they already have an account-level subscription policy. CloudFormation shows `ApiToken` as `****`, but keeps it as a StackSet parameter. If the token should live only in Secrets Manager, set up each account with `ApiTokenSecretArn` instead. A StackSet exists only in the region where you create it. Choose one region to administer it from, and pass that region as `--region ` on every StackSet command, including later rotations and removal. With a delegated administrator, pick one of the [regions AWS supports for delegated administrators](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-delegated-admin.html). Before you start, from the organization's management account, [enable all features](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_org_support-all-features.html) in AWS Organizations and [activate trusted access](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-activate-trusted-access.html) for StackSets: ```shell theme={null} aws cloudformation activate-organizations-access ``` If an account and region the StackSet targets already has the single-account setup from this page, remove it first, as described in [Remove log delivery](#remove-log-delivery). The StackSet's stacks use the same stream and log group names, so their instances fail there. The management account isn't targeted, so keep its setup. To administer the StackSet from a [delegated administrator](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-delegated-admin.html) account instead, register that account from the management account: ```shell theme={null} aws organizations register-delegated-administrator \ --service-principal member.org.stacksets.cloudformation.amazonaws.com \ --account-id ``` From a delegated administrator, add `--call-as DELEGATED_ADMIN` to every StackSet command on this page: `create-stack-set`, `create-stack-instances`, `update-stack-instances`, `update-stack-set`, `describe-stack-set-operation`, `list-stack-instances`, `delete-stack-instances`, and `delete-stack-set`. The delegated administrator also needs `organizations:ListDelegatedAdministrators`. The commands below read the API token with `read -rs`, which works in `bash` and `zsh` and doesn't show the token as you paste it. Run the `read` command on its own, before the commands that use the token. 1. Read the API token: ```shell theme={null} printf 'API token: '; read -rs API_TOKEN; echo ``` 2. Write the parameters to a file only you can read, so the token stays out of your shell history: ```shell theme={null} rm -f stackset-params.json if [ -n "$API_TOKEN" ]; then (umask 077 && cat > stackset-params.json) <"}, {"ParameterKey": "ApiToken", "ParameterValue": "$API_TOKEN"} ] EOF else echo 'API_TOKEN is empty: run the read command first' fi unset API_TOKEN ``` 3. Create the StackSet from the template in the CloudFormation tab, saved as `cortex-xcor-aws-logs.yaml`, and then delete the parameters file: ```shell theme={null} if [ -f stackset-params.json ]; then aws cloudformation create-stack-set --region \ --stack-set-name cortex-xcor-aws-logs \ --template-body file://cortex-xcor-aws-logs.yaml \ --permission-model SERVICE_MANAGED \ --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \ --capabilities CAPABILITY_IAM \ --parameters file://stackset-params.json rm -f stackset-params.json else echo 'stackset-params.json is missing: run the previous step first' fi ``` With automatic deployment, an account that joins a target organizational unit gets a stack, and an account that leaves has its stack deleted. The deleted stack's failed-data bucket is kept. 4. Deploy the StackSet to your organizational units and regions. `` can be the organization's root ID (`r-...`) to cover every account. ```shell theme={null} OP_ID=$(aws cloudformation create-stack-instances --region \ --stack-set-name cortex-xcor-aws-logs \ --deployment-targets OrganizationalUnitIds= \ --regions \ --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \ --query OperationId --output text) ``` These preferences deploy to every account and region at once and keep going past accounts that fail. Without them, CloudFormation deploys to one account at a time and stops at the first failure, which also skips every later region. The preferences don't apply to automatic deployments. 5. Wait for the operation to finish, and then list the stack instances that failed. With these preferences, the operation reports `SUCCEEDED` even when some instances failed. ```shell theme={null} while :; do STATUS=$(aws cloudformation describe-stack-set-operation --region \ --stack-set-name cortex-xcor-aws-logs --operation-id "$OP_ID" \ --query StackSetOperation.Status --output text) case "$STATUS" in RUNNING|QUEUED|STOPPING) sleep 30 ;; *) break ;; esac done echo "Operation $STATUS" aws cloudformation list-stack-instances --region \ --stack-set-name cortex-xcor-aws-logs \ --filters Name=DETAILED_STATUS,Values=FAILED ``` An instance fails in an account and region that already has an account-level subscription policy, because each account and region allows only one. For each of those, override `CreateAccountPolicy` to `false`, wait for the operation with the commands in step 5, and then subscribe log groups individually in that account with the `DeliveryStreamArn` and `SubscriptionRoleArn` outputs of its stack: ```shell theme={null} OP_ID=$(aws cloudformation update-stack-instances --region \ --stack-set-name cortex-xcor-aws-logs \ --deployment-targets Accounts= \ --regions \ --parameter-overrides ParameterKey=CreateAccountPolicy,ParameterValue=false \ --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \ --query OperationId --output text) ``` Target accounts rather than an organizational unit, because an override set on an organizational unit doesn't apply to accounts that join it later. Fix failed instances promptly. Every later update to the StackSet tries to create them again, which makes the update much slower. A StackSet with service-managed permissions doesn't deploy to the management account. To collect that account's logs, set it up on its own as described earlier on this page. To rotate the token, read the new token on its own: ```shell theme={null} printf 'API token: '; read -rs API_TOKEN; echo ``` Then update the StackSet's `ApiToken` parameter, keeping the other parameters: ```shell theme={null} rm -f stackset-params.json if [ -n "$API_TOKEN" ]; then (umask 077 && cat > stackset-params.json) < \ --stack-set-name cortex-xcor-aws-logs \ --use-previous-template \ --capabilities CAPABILITY_IAM \ --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \ --parameters file://stackset-params.json \ --query OperationId --output text) rm -f stackset-params.json else echo 'API_TOKEN is empty: run the read command first' fi unset API_TOKEN ``` Wait for the operation and list failed instances with the commands in step 5. If any instance is listed, fix its cause and run the update again, which retries the instances that didn't update. If CloudFormation reports that another operation is in progress, such as an automatic deployment, wait for it to finish and run the update again. If the management account's own setup uses the same token, update its secret in each region where you set it up. Read the new token again: ```shell theme={null} printf 'API token: '; read -rs API_TOKEN; echo ``` Then update each region: ```shell theme={null} if [ -n "$API_TOKEN" ]; then for region in ; do printf '{"api_key":"%s"}' "$API_TOKEN" | aws secretsmanager put-secret-value --region "$region" \ --secret-id cortex-xcor/aws-logs-api-token \ --secret-string file:///dev/stdin done else echo 'API_TOKEN is empty: run the read command first' fi unset API_TOKEN ``` Each Firehose stream starts using the new token within 10 minutes of its secret changing. Don't revoke the old token until every stack and the management account's secret have updated, and 10 more minutes have passed. This page doesn't cover sending logs from many accounts to one stream. ### Subscribe individual log groups To send only some log groups, or when another tool holds the account-level policy, add a subscription filter to each log group: ```shell theme={null} aws logs put-subscription-filter --region \ --log-group-name \ --filter-name cortex-xcor-logs \ --filter-pattern "" \ --destination-arn \ --role-arn ``` Don't include these log groups in the account-level policy, or their logs arrive twice. ### Send logs straight to Firehose Some services deliver logs to a Firehose stream without a log group. Give each service its own stream, created in the console with the same settings under a unique name. By default, the [Terraform](https://docs-xcor.paloaltonetworks.com/tooling/infrastructure/terraform) and CloudFormation configurations also create the account subscription policy, so don't run them again for these streams. The account-level policy sends each additional stream's error log group, `/aws/kinesisfirehose/`, to the stream targeted by the policy unless you add the error log group to the policy exclusion. Cortex XCOR recognizes these logs on its own: * AWS WAF: By the stream name, which AWS requires to start with `aws-waf-logs-`, and by the log format. * Amazon API Gateway: By a stream name that starts with `amazon-apigateway-`. * VPC Flow Logs: By the default log format. For a custom format, add the stream parameter `xcor_integration` with the value `aws_vpc`. For other logs sent straight to Firehose, set the `xcor_integration` parameter to a recognized AWS integration ID, such as `aws_cloudtrail`, `aws_eks`, `aws_lambda`, or `aws_rds`. Cortex XCOR ignores unknown values. Logs it can't otherwise recognize are stored with the label `aws_other`. ## Validate log ingestion After setup, wait for logs to arrive before validating ingestion. 1. In the Amazon Data Firehose console, open the stream. **HTTP endpoint delivery success** should report 100%, and **Records delivered to HTTP endpoint** should be non-zero. 2. In Cortex XCOR, go to **Explorers > Logs Explorer** and run: ```text theme={null} ["xcor.integration.name"] = "aws" | summarize count() by xcor_integration ``` Each AWS service sending logs appears as a row. ## Log labels and fields Every log gets two labels: * `xcor.integration.name`: always `aws`, the integration's catalog ID. * `xcor_integration`: the AWS service that wrote the log, such as `aws_lambda`, `aws_rds`, `aws_eks`, `aws_vpc`, or `aws_cloudtrail`. Cortex XCOR determines the service from the log group name, log stream name, log format, or a stream parameter. Logs it can't attribute to an AWS service get `aws_other` and are stored like any other log. Keep the default AWS log group names. The exception is Amazon ECS: Cortex XCOR recognizes ECS logs only in log groups under `/aws/ecs/` or `/aws/vendedlogs/ecs/`, so the `/ecs/` group that the ECS console creates gets `aws_other`. Cortex XCOR can still recognize some services from custom log group names by using the log stream name or format. Other logs get `aws_other`. Every log also carries these fields, set from the delivery rather than the log message: | Field | Value | | :- | :- | | `aws.account_id` | The account that owns the log group, or the Firehose stream's account for direct delivery | | `aws.region` | The log group's region, when available, or the Firehose stream's region | | `aws.firehose.arn`, `aws.firehose.stream`, `aws.firehose.request_id` | The stream and request | | `aws.firehose.attributes.` | Each parameter you set on the stream | Logs delivered through a CloudWatch Logs subscription also carry these fields: | Field | Value | | :- | :- | | `aws.log_group`, `aws.log_stream` | Where CloudWatch Logs stored the log | | `aws.subscription_filters`, `aws.event_id` | From the CloudWatch Logs subscription | | `aws.timestamp` | When CloudWatch Logs stored the event | For an organization trail, `aws.account_id` is the management account. CloudTrail's `recipientAccountId` field identifies the account for each event, and its `awsRegion` field identifies the event's region. Cortex XCOR parses a JSON log message into fields and stores any other message in `message`. Fields set by the log take precedence over the following fields that Cortex XCOR sets: * Time: The log's `timestamp` field, then CloudTrail's `eventTime`, then the time CloudWatch Logs stored the event. A log sent straight to Firehose without a time gets the time Cortex XCOR receives it. * Service: The `service` field is the `xcor_integration` value without `aws_`, such as `cloudtrail`, `eks`, or `rds`. Lambda logs get the function name. Logs labeled `aws_other` get the log group, or `aws_other` when there's no log group. * Message: For CloudTrail, `message` is the event name, such as `AssumeRole`. * Severity: CloudTrail events with an `errorCode` get `ERROR`. Other CloudTrail events get `INFO`, and other logs without a severity get your organization's default. ## Troubleshooting Firehose records delivery errors in the log group `/aws/kinesisfirehose/`. * `401` errors, or a `content-type: application/json` error whose raw response starts with `401`: The token isn't an API token for a restricted service account with write-only permission in this organization. After a token change, Firehose can keep using the old token for up to 10 minutes. * Secrets Manager errors: The secret must be JSON with the key `api_key`, and the stream role needs `secretsmanager:GetSecretValue` on the secret. * `404` errors, or a `content-type: application/json` error whose raw response starts with `404`: The endpoint URL is incorrect. It must end in `/api/v1/data/logs/ingest/integrations/aws`. * Logs labeled `aws_other`: The log group has a custom name or is the ECS console's `/ecs/` group, or the logs came straight from a service that Cortex XCOR couldn't identify from the stream name or log format. Set the stream parameter `xcor_integration` to a recognized AWS integration ID. For example, use `aws_vpc` for VPC Flow Logs in a custom format. * Delivery succeeds but some logs are missing: Cortex XCOR drops logs more than 24 hours old or more than 24 hours in the future. CloudWatch Logs can deliver older events, such as after a long outage. * Duplicate logs: After an error, Firehose sends the entire batch again, so some records can arrive twice. Records that still fail after the retry window go to the S3 bucket. Cortex XCOR doesn't read the bucket, so those records aren't ingested unless they're sent again within 24 hours of their timestamps. ## Remove log delivery Until you remove it, the account-level policy keeps sending every log group in the region to the stream, and Firehose continues billing for it. * For Terraform, run `terraform destroy`. * For CloudFormation, note the failed-data bucket's name, and then delete the stack: ```shell theme={null} aws cloudformation describe-stack-resource --region \ --stack-name cortex-xcor-aws-logs --logical-resource-id FailedDataBucket \ --query StackResourceDetail.PhysicalResourceId --output text aws cloudformation delete-stack --region --stack-name cortex-xcor-aws-logs ``` From template 1.1.0, deleting the stack keeps the failed-data bucket, so leftover records can't make the delete fail. Its records still expire, and you can delete it with `aws s3 rb s3:// --force` when you no longer need them, where `` is the name the first command printed. If you passed `ApiToken`, deleting the stack also deletes its secret immediately, without a recovery window. * For a StackSet, delete the stack instances from the organizational units and regions you deployed to: ```shell theme={null} OP_ID=$(aws cloudformation delete-stack-instances --region \ --stack-set-name cortex-xcor-aws-logs \ --deployment-targets OrganizationalUnitIds= \ --regions \ --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \ --no-retain-stacks \ --query OperationId --output text) ``` Wait for the operation with the commands in step 5 of [Set up many accounts and regions](#set-up-many-accounts-and-regions), and then delete the StackSet. Each account keeps its failed-data bucket. ```shell theme={null} aws cloudformation delete-stack-set --region --stack-set-name cortex-xcor-aws-logs ``` * In the console, delete the account-level policy, then the Firehose stream, subscription role, failed-data bucket, and the stream's error log group, which the console creates: ```shell theme={null} aws logs delete-account-policy --region \ --policy-name cortex-xcor-logs --policy-type SUBSCRIPTION_FILTER_POLICY aws logs delete-log-group --region \ --log-group-name /aws/kinesisfirehose/cortex-xcor-logs ``` * For individual subscriptions, remove each subscription with: ```shell theme={null} aws logs delete-subscription-filter --region \ --log-group-name --filter-name cortex-xcor-logs ``` If you use Terraform, delete resources by hand, or delete a stack from a template before 1.1.0, empty the failed-data bucket first, or its deletion fails. If you created the token secret yourself and passed `ApiTokenSecretArn`, delete it too, because the setup doesn't own it. By default, Secrets Manager schedules deletion with a 30-day recovery window. During that window, another setup attempt with the same secret name fails. The `--force-delete-without-recovery` option skips the recovery window. Deletion is asynchronous, so wait and retry if the name remains unavailable: ```shell theme={null} aws secretsmanager delete-secret --region --secret-id cortex-xcor/aws-logs-api-token \ --force-delete-without-recovery ``` ## Related topics - [Ingest AWS CloudWatch metrics](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/server/aws/metrics.md) - [Ingest log data](https://docs-xcor.paloaltonetworks.com/ingest/logs.md) - [UpdateLogIngestConfig](https://docs-xcor.paloaltonetworks.com/tooling/api-info/definition/operations/UpdateLogIngestConfig.md) - [CreateLogIngestConfig](https://docs-xcor.paloaltonetworks.com/tooling/api-info/definition/operations/CreateLogIngestConfig.md) - [ReadLogIngestConfig](https://docs-xcor.paloaltonetworks.com/tooling/api-info/definition/operations/ReadLogIngestConfig.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.