Ingest AWS logs

Palo Alto Networks Cortex XCOR receives Amazon Web Services (AWS) logs through Amazon Data Firehose. In each AWS account and region, a Firehose stream sends logs to the AWS logs ingest endpoint in your Cortex XCOR tenant. CloudWatch Logs reach the stream through a subscription, and some AWS services send their logs to the stream directly.

What it collects

CloudWatch Logs: Logs that AWS services and your applications write to CloudWatch log groups. Use an account-level subscription policy to cover every log group in a region, including new ones, or add a subscription filter to each log group. Subscriptions cover only log groups in the Standard log class. CloudTrail can also send its events to a log group.
Logs sent straight to Firehose: Services such as Virtual Private Cloud (VPC) Flow Logs and Web Application Firewall (WAF) can deliver logs to a Firehose stream without a log group.
Not collected: Logs that AWS writes only to Amazon S3, such as Classic Load Balancer access logs.

How it works

The following diagram shows how AWS log sources reach Cortex XCOR and where Firehose stores eligible failed deliveries.

Before you begin

AWS permissions: The identity that runs the setup needs the permissions listed in Required permissions.
A Cortex XCOR API token: Create or use an API token for a restricted service account with write-only permission. For more information, see Create a restricted service account.

Your organization name: The <ORG_NAME> segment in https://<ORG_NAME>.chronosphere.io.

Existing subscriptions: Each account and region allows one account-level subscription policy, and each log group allows five subscription filters of its own, which the account-level policy doesn’t count against. If another tool already uses the account-level policy, subscribe log groups individually. To inspect the account policies, run:
aws logs describe-account-policies --policy-type SUBSCRIPTION_FILTER_POLICY --region <REGION>

Required permissions

To run the Terraform or CloudFormation setup in one account, the identity that runs it needs the following permissions. These permissions create and remove the setup resources. StackSets and the console need additional permissions.

Amazon Data Firehose
- Actions: CreateDeliveryStream, DeleteDeliveryStream, DescribeDeliveryStream, UpdateDestination, ListTagsForDeliveryStream, TagDeliveryStream, UntagDeliveryStream
- Resource: The stream
Amazon S3
- Actions: CreateBucket, DeleteBucket, PutLifecycleConfiguration, PutBucketTagging, ListBucket, Get*, and DeleteObject to empty the bucket when you remove it
- Resource: The failed-data bucket
CloudWatch Logs
- Actions: CreateLogGroup, DeleteLogGroup, PutRetentionPolicy, CreateLogStream, DeleteLogStream, DescribeLogGroups, DescribeLogStreams, ListTagsForResource, TagResource, PutAccountPolicy, DeleteAccountPolicy, DescribeAccountPolicies, PutSubscriptionFilter, DeleteSubscriptionFilter, DescribeSubscriptionFilters
- Resource: All resources, because account policies have no resource ARN
IAM
- Actions: CreateRole, DeleteRole, GetRole, PutRolePolicy, GetRolePolicy, DeleteRolePolicy, ListRolePolicies, ListAttachedRolePolicies, ListInstanceProfilesForRole, PassRole
- Resources: The two roles the setup creates
AWS Secrets Manager
- Actions: CreateSecret, DeleteSecret, DescribeSecret, UpdateSecret, GetSecretValue, TagResource, UntagResource
- Resource: The token secret
CloudFormation, for the CloudFormation tab only
- Actions: CreateChangeSet, DescribeChangeSet, ExecuteChangeSet, DeleteChangeSet, DescribeStacks, DescribeStackEvents, DeleteStack, GetTemplateSummary
- Resource: The stack

The setup creates two roles:

Firehose role: Trusted by Firehose in your account. It can write to the failed-data bucket, read the token secret, and write the stream’s error log.
Subscription role: Trusted by CloudWatch Logs in your account. It can put records into the stream.

The setup has no access to your existing buckets or other resources, and it doesn’t send logs written before it’s in place.

Firehose settings

Use these settings regardless of how you create the stream: Firehose bills each CloudWatch Logs record at its compressed size, rounded up to 5 KB. Log groups that send few events per batch cost more per log. Firehose also bills data transfer for every delivery attempt, including retries. While deliveries fail, data transfer grows with each retry until the retry window ends.

Source: Direct PUT.
Destination: HTTP Endpoint.
HTTP endpoint URL: https://<ORG_NAME>.chronosphere.io/api/v1/data/logs/ingest/integrations/aws.
Authentication: An AWS Secrets Manager secret containing {"api_key":"<API_TOKEN>"}. Firehose picks up a rotated token within about 10 minutes. Using the API token as the endpoint access key also works, but stores it in the stream settings.
Content encoding: GZIP for smaller requests and lower data transfer costs.
Retry duration: 7200 seconds, the longest Firehose allows. Records that still fail go to the S3 bucket.
Buffer size: 1 MiB. Keep this value at 4 MiB or less. Firehose treats the size as a hint and encodes each record in a text representation larger than the source record. Cortex XCOR rejects requests over 16 MiB with a 413, and Firehose drops a batch that gets a 413 without retrying it or backing it up.
Buffer interval: 60 seconds.
Decompression and message extraction: Off. Cortex XCOR decodes CloudWatch Logs records. Message extraction removes the log group and log stream used to identify the service.
Source record backup in Amazon S3: Failed data only.
Backup bucket lifecycle: Expire objects after 14 days. Cortex XCOR drops retried logs more than 24 hours old.
Parameters: None. For the exception for VPC Flow Logs in a custom format, see Send logs straight to Firehose.

Set up log delivery

Set up delivery in each account and region you want logs from.

Terraform
AWS Management Console
CloudFormation

Store your API token in AWS Secrets Manager in the region you’re setting up. Note the secret’s ARN. Read the token without echoing it, so it stays out of your shell history. Run the read command on its own:

printf 'API token: '; read -rs API_TOKEN; echo

Then create the secret:

if [ -n "$API_TOKEN" ]; then
  printf '{"api_key":"%s"}' "$API_TOKEN" |
    aws secretsmanager create-secret --region <REGION> \
      --name cortex-xcor/aws-logs-api-token \
      --secret-string file:///dev/stdin
else
  echo 'API_TOKEN is empty: run the read command first'
fi
unset API_TOKEN

The configuration was tested with version 6 of the aws provider. It also needs the time provider. Set the region on the aws provider:

terraform {
  required_providers {
    aws  = { source = "hashicorp/aws", version = ">= 6.0" }
    time = { source = "hashicorp/time" }
  }
}

provider "aws" {
  region = "<REGION>"
}

Set values for these variables, and apply the configuration in each account and region. It creates the Firehose stream, S3 bucket for failed data with a lifecycle rule, and IAM roles. Unless you turn it off, it also creates an account-level subscription policy that sends every log group in the region except the stream’s own error logs.

Terraform example

# Variables

variable "xcor_org_name" {
  type        = string
  description = "The name of your Cortex XCOR organization, which is the subdomain name before .chronosphere.io."
}

variable "api_token_secret_arn" {
  type        = string
  description = "The ARN of the Secrets Manager secret holding {\"api_key\":\"<API_TOKEN>\"}, in the stream's region."
}

variable "stream_name" {
  type        = string
  description = "The name of the Firehose stream."
  default     = "cortex-xcor-logs"
}

variable "create_account_policy" {
  type        = bool
  description = "Set to false if this account and region already has an account-level subscription policy."
  default     = true
}

variable "failed_data_expiration_days" {
  type        = number
  description = "Days to keep records Firehose couldn't deliver."
  default     = 14
}

locals {
  tags = {
    "cortex-xcor:template"         = "aws-logs-firehose"
    "cortex-xcor:template-version" = "1.0.0"
  }
}

data "aws_caller_identity" "current" {}

# S3 bucket and log group for failed deliveries

resource "aws_s3_bucket" "failed_data" {
  bucket_prefix = "cortex-xcor-logs-failed-"
  tags          = local.tags
}

resource "aws_s3_bucket_lifecycle_configuration" "failed_data" {
  bucket = aws_s3_bucket.failed_data.id
  rule {
    id     = "expire-failed-data"
    status = "Enabled"
    filter {}
    expiration {
      days = var.failed_data_expiration_days
    }
  }
}

resource "aws_cloudwatch_log_group" "firehose" {
  name              = "/aws/kinesisfirehose/${var.stream_name}"
  retention_in_days = 14
}

resource "aws_cloudwatch_log_stream" "firehose" {
  name           = "DestinationDelivery"
  log_group_name = aws_cloudwatch_log_group.firehose.name
}

# Firehose stream

data "aws_iam_policy_document" "firehose_trust" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["firehose.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
}

data "aws_iam_policy_document" "firehose_access" {
  statement {
    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject",
    ]
    resources = [aws_s3_bucket.failed_data.arn, "${aws_s3_bucket.failed_data.arn}/*"]
  }
  statement {
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [var.api_token_secret_arn]
  }
  statement {
    actions   = ["logs:PutLogEvents"]
    resources = ["${aws_cloudwatch_log_group.firehose.arn}:*"]
  }
}

resource "aws_iam_role" "firehose" {
  name_prefix        = "cortex-xcor-logs-firehose-"
  assume_role_policy = data.aws_iam_policy_document.firehose_trust.json
}

resource "aws_iam_role_policy" "firehose" {
  role   = aws_iam_role.firehose.id
  policy = data.aws_iam_policy_document.firehose_access.json
}

resource "aws_kinesis_firehose_delivery_stream" "logs" {
  name        = var.stream_name
  destination = "http_endpoint"
  tags        = local.tags

  http_endpoint_configuration {
    name               = "Cortex XCOR"
    url                = "https://${var.xcor_org_name}.chronosphere.io/api/v1/data/logs/ingest/integrations/aws"
    buffering_size     = 1    # MiB
    buffering_interval = 60   # seconds
    retry_duration     = 7200 # seconds
    role_arn           = aws_iam_role.firehose.arn
    s3_backup_mode     = "FailedDataOnly"

    secrets_manager_configuration {
      enabled    = true
      secret_arn = var.api_token_secret_arn
      role_arn   = aws_iam_role.firehose.arn
    }

    s3_configuration {
      role_arn           = aws_iam_role.firehose.arn
      bucket_arn         = aws_s3_bucket.failed_data.arn
      buffering_size     = 10  # MiB
      buffering_interval = 300 # seconds
      compression_format = "GZIP"
    }

    request_configuration {
      content_encoding = "GZIP"
    }

    cloudwatch_logging_options {
      enabled         = true
      log_group_name  = aws_cloudwatch_log_group.firehose.name
      log_stream_name = aws_cloudwatch_log_stream.firehose.name
    }
  }
}

# CloudWatch Logs subscription

data "aws_iam_policy_document" "subscription_trust" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["logs.amazonaws.com"]
    }
    condition {
      test     = "StringLike"
      variable = "aws:SourceArn"
      values   = ["arn:aws:logs:*:${data.aws_caller_identity.current.account_id}:*"]
    }
  }
}

data "aws_iam_policy_document" "subscription_access" {
  statement {
    actions   = ["firehose:PutRecord", "firehose:PutRecordBatch"]
    resources = [aws_kinesis_firehose_delivery_stream.logs.arn]
  }
}

resource "aws_iam_role" "subscription" {
  name_prefix        = "cortex-xcor-logs-subscription-"
  assume_role_policy = data.aws_iam_policy_document.subscription_trust.json
}

resource "aws_iam_role_policy" "subscription" {
  role   = aws_iam_role.subscription.id
  policy = data.aws_iam_policy_document.subscription_access.json
}

# CloudWatch Logs sends a test record when the policy is created, and a new
# role's permissions take a few seconds to take effect.
resource "time_sleep" "subscription_role" {
  depends_on      = [aws_iam_role_policy.subscription]
  create_duration = "30s"
}

resource "aws_cloudwatch_log_account_policy" "logs" {
  count      = var.create_account_policy ? 1 : 0
  depends_on = [time_sleep.subscription_role]

  policy_name = "cortex-xcor-logs"
  policy_type = "SUBSCRIPTION_FILTER_POLICY"
  scope       = "ALL"
  # The stream's own error logs are excluded, or they'd loop back into it.
  selection_criteria = "LogGroupName NOT IN [\"${aws_cloudwatch_log_group.firehose.name}\"]"
  policy_document = jsonencode({
    DestinationArn = aws_kinesis_firehose_delivery_stream.logs.arn
    RoleArn        = aws_iam_role.subscription.arn
    FilterPattern  = ""
  })
}

# Used to subscribe log groups individually

output "stream_arn" {
  value = aws_kinesis_firehose_delivery_stream.logs.arn
}

output "subscription_role_arn" {
  value = aws_iam_role.subscription.arn
}

If you set create_account_policy = false, follow Subscribe individual log groups with the stream_arn and subscription_role_arn outputs.

In AWS Secrets Manager, store a secret of type Other type of secret with the key api_key and your API token as the value. Name it cortex-xcor/aws-logs-api-token.
Open the Amazon Data Firehose console and select Create Firehose stream.
Set the following options:
- Source: Direct PUT
- Destination: HTTP Endpoint
- Firehose stream name: cortex-xcor-logs
- Transform records: If the console offers Decompress source records from Amazon CloudWatch Logs, leave decompression and message extraction off.
- Destination settings:
  - HTTP endpoint name: Cortex XCOR
  - HTTP endpoint URL: https://<ORG_NAME>.chronosphere.io/api/v1/data/logs/ingest/integrations/aws. Replace <ORG_NAME> with your organization name.
  - Authentication: Use AWS Secrets Manager, and select the secret you created.
  - Content encoding: GZIP
  - Retry duration: 7200 seconds
  - Expand Buffer hints.
    - Buffer size: 1 MiB
    - Buffer interval: 60 seconds
- Backup settings:
  - Source record backup in Amazon S3: Failed data only
  - S3 backup bucket: Create a bucket for this stream with default settings. In the bucket’s Management tab, add a lifecycle rule that expires objects after 14 days. Don’t add that rule to a bucket that holds other data.
  - Expand Buffer hints, compression and encryption, and set Compression for data records to GZIP.
- Advanced settings: Under Service access, let the console create an IAM role. Confirm that the role allows secretsmanager:GetSecretValue on the secret.
Select Create Firehose stream.
In the IAM console, create a role that CloudWatch Logs can assume. Set the trusted entity to logs.amazonaws.com, with the condition that aws:SourceArn is like arn:aws:logs:<REGION>:<ACCOUNT_ID>:*. Grant the role firehose:PutRecord and firehose:PutRecordBatch on the stream.

Use the AWS Command Line Interface to subscribe log groups with an account-level subscription policy. Exclude the stream’s own error log group to prevent its logs from looping back into the stream:

aws logs put-account-policy --region <REGION> \
  --policy-name cortex-xcor-logs \
  --policy-type SUBSCRIPTION_FILTER_POLICY \
  --scope ALL \
  --selection-criteria 'LogGroupName NOT IN ["/aws/kinesisfirehose/cortex-xcor-logs"]' \
  --policy-document '{"DestinationArn":"<STREAM_ARN>","RoleArn":"<ROLE_ARN>","FilterPattern":""}'

If the account already has an account-level subscription policy, subscribe log groups individually.

printf 'API token: '; read -rs API_TOKEN; echo

Then create the secret:

if [ -n "$API_TOKEN" ]; then
  printf '{"api_key":"%s"}' "$API_TOKEN" |
    aws secretsmanager create-secret --region <REGION> \
      --name cortex-xcor/aws-logs-api-token \
      --secret-string file:///dev/stdin
else
  echo 'API_TOKEN is empty: run the read command first'
fi
unset API_TOKEN

Save this template as cortex-xcor-aws-logs.yaml. It creates the Firehose stream, S3 bucket for failed data with a lifecycle rule, and IAM roles. Unless you turn it off, it also creates an account-level subscription policy that sends every log group in the region except the stream’s own error logs.

CloudFormation template

AWSTemplateFormatVersion: "2010-09-09"
Description: Sends CloudWatch Logs in this account and region to Cortex XCOR through Amazon Data Firehose. Template version 1.1.0.

Parameters:
  OrgName:
    Type: String
    Description: Your Cortex XCOR organization name, the subdomain before .chronosphere.io.
  ApiTokenSecretArn:
    Type: String
    Default: ""
    Description: ARN of the Secrets Manager secret holding {"api_key":"<API_TOKEN>"}, in this stack's region. Leave empty to set ApiToken instead.
  ApiToken:
    Type: String
    Default: ""
    NoEcho: true
    Description: The API token, for a stack that creates its own secret, such as a StackSet instance. Leave empty to set ApiTokenSecretArn instead.
  StreamName:
    Type: String
    Default: cortex-xcor-logs
  CreateAccountPolicy:
    Type: String
    Default: "true"
    AllowedValues: ["true", "false"]
    Description: Set to false if this account and region already has an account-level subscription policy.
  FailedDataExpirationDays:
    Type: Number
    Default: 14
    MinValue: 1
    Description: Days to keep records Firehose couldn't deliver.

Rules:
  OneTokenSource:
    Assertions:
      - Assert: !Or [!Not [!Equals [!Ref ApiTokenSecretArn, ""]], !Not [!Equals [!Ref ApiToken, ""]]]
        AssertDescription: Set ApiTokenSecretArn or ApiToken.
      - Assert: !Or [!Equals [!Ref ApiTokenSecretArn, ""], !Equals [!Ref ApiToken, ""]]
        AssertDescription: Set only one of ApiTokenSecretArn and ApiToken.

Conditions:
  CreatePolicy: !Equals [!Ref CreateAccountPolicy, "true"]
  CreateSecret: !Equals [!Ref ApiTokenSecretArn, ""]

Resources:
  ApiTokenSecret:
    Type: AWS::SecretsManager::Secret
    Condition: CreateSecret
    Properties:
      Description: Cortex XCOR API token for the AWS logs Firehose stream
      SecretString: !Sub '{"api_key":"${ApiToken}"}'

  FailedDataBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: RetainExceptOnCreate
    UpdateReplacePolicy: Retain
    Properties:
      LifecycleConfiguration:
        Rules:
          - Id: expire-failed-data
            Status: Enabled
            ExpirationInDays: !Ref FailedDataExpirationDays
      Tags:
        - { Key: "cortex-xcor:template", Value: aws-logs-firehose }
        - { Key: "cortex-xcor:template-version", Value: "1.1.0" }

  FirehoseLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/kinesisfirehose/${StreamName}
      RetentionInDays: 14

  FirehoseLogStream:
    Type: AWS::Logs::LogStream
    Properties:
      LogGroupName: !Ref FirehoseLogGroup
      LogStreamName: DestinationDelivery

  FirehoseRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: { Service: firehose.amazonaws.com }
            Action: sts:AssumeRole
            Condition:
              StringEquals:
                aws:SourceAccount: !Ref AWS::AccountId
      Policies:
        - PolicyName: delivery
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: [s3:AbortMultipartUpload, s3:GetBucketLocation, s3:GetObject, s3:ListBucket, s3:ListBucketMultipartUploads, s3:PutObject]
                Resource: [!GetAtt FailedDataBucket.Arn, !Sub "${FailedDataBucket.Arn}/*"]
              - Effect: Allow
                Action: secretsmanager:GetSecretValue
                Resource: !If [CreateSecret, !Ref ApiTokenSecret, !Ref ApiTokenSecretArn]
              - Effect: Allow
                Action: logs:PutLogEvents
                Resource: !GetAtt FirehoseLogGroup.Arn

  DeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: !Ref StreamName
      DeliveryStreamType: DirectPut
      Tags:
        - { Key: "cortex-xcor:template", Value: aws-logs-firehose }
        - { Key: "cortex-xcor:template-version", Value: "1.1.0" }
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          Name: Cortex XCOR
          Url: !Sub https://${OrgName}.chronosphere.io/api/v1/data/logs/ingest/integrations/aws
        SecretsManagerConfiguration:
          Enabled: true
          SecretARN: !If [CreateSecret, !Ref ApiTokenSecret, !Ref ApiTokenSecretArn]
          RoleARN: !GetAtt FirehoseRole.Arn
        RequestConfiguration:
          ContentEncoding: GZIP
        BufferingHints:
          SizeInMBs: 1
          IntervalInSeconds: 60
        RetryOptions:
          DurationInSeconds: 7200
        RoleARN: !GetAtt FirehoseRole.Arn
        S3BackupMode: FailedDataOnly
        S3Configuration:
          BucketARN: !GetAtt FailedDataBucket.Arn
          RoleARN: !GetAtt FirehoseRole.Arn
          CompressionFormat: GZIP
          BufferingHints:
            SizeInMBs: 10
            IntervalInSeconds: 300
        CloudWatchLoggingOptions:
          Enabled: true
          LogGroupName: !Ref FirehoseLogGroup
          LogStreamName: !Ref FirehoseLogStream

  SubscriptionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: { Service: logs.amazonaws.com }
            Action: sts:AssumeRole
            Condition:
              StringLike: { "aws:SourceArn": !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*" }
      Policies:
        - PolicyName: put-to-firehose
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: [firehose:PutRecord, firehose:PutRecordBatch]
                Resource: !GetAtt DeliveryStream.Arn

  AccountPolicy:
    Type: AWS::Logs::AccountPolicy
    Condition: CreatePolicy
    Properties:
      PolicyName: cortex-xcor-logs
      PolicyType: SUBSCRIPTION_FILTER_POLICY
      Scope: ALL
      # The stream's own error logs are excluded, or they'd loop back into it.
      SelectionCriteria: !Sub 'LogGroupName NOT IN ["/aws/kinesisfirehose/${StreamName}"]'
      PolicyDocument: !Sub '{"DestinationArn":"${DeliveryStream.Arn}","RoleArn":"${SubscriptionRole.Arn}","FilterPattern":""}'

Outputs:
  DeliveryStreamArn:
    Value: !GetAtt DeliveryStream.Arn
  SubscriptionRoleArn:
    Value: !GetAtt SubscriptionRole.Arn

Deploy the stack:

aws cloudformation deploy --region <REGION> \
  --template-file cortex-xcor-aws-logs.yaml \
  --stack-name cortex-xcor-aws-logs \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides OrgName=<ORG_NAME> ApiTokenSecretArn=<SECRET_ARN>

deploy keeps a parameter’s previous value when you leave it out. To switch an existing stack between ApiToken and ApiTokenSecretArn, pass the one you stop using as empty, such as ApiToken="".

If you set CreateAccountPolicy=false, follow Subscribe individual log groups with the DeliveryStreamArn and SubscriptionRoleArn stack outputs.

Set up many accounts and regions

Set up one stream in each account and region you want logs from. Each stream reports its own deliveries, so a problem appears in the account and region it affects. To deploy to many accounts in an organization in AWS Organizations, run the CloudFormation template as a StackSet with service-managed permissions. Pass the API token as ApiToken instead of ApiTokenSecretArn. Each stack instance then creates its own secret, so accounts that automatic deployment adds later need no extra setup, unless they already have an account-level subscription policy. CloudFormation shows ApiToken as ****, but keeps it as a StackSet parameter. If the token should live only in Secrets Manager, set up each account with ApiTokenSecretArn instead. A StackSet exists only in the region where you create it. Choose one region to administer it from, and pass that region as --region <ADMIN_REGION> on every StackSet command, including later rotations and removal. With a delegated administrator, pick one of the regions AWS supports for delegated administrators. Before you start, from the organization’s management account, enable all features in AWS Organizations and activate trusted access for StackSets:

aws cloudformation activate-organizations-access

If an account and region the StackSet targets already has the single-account setup from this page, remove it first, as described in Remove log delivery. The StackSet’s stacks use the same stream and log group names, so their instances fail there. The management account isn’t targeted, so keep its setup. To administer the StackSet from a delegated administrator account instead, register that account from the management account:

aws organizations register-delegated-administrator \
  --service-principal member.org.stacksets.cloudformation.amazonaws.com \
  --account-id <ADMIN_ACCOUNT_ID>

From a delegated administrator, add --call-as DELEGATED_ADMIN to every StackSet command on this page: create-stack-set, create-stack-instances, update-stack-instances, update-stack-set, describe-stack-set-operation, list-stack-instances, delete-stack-instances, and delete-stack-set. The delegated administrator also needs organizations:ListDelegatedAdministrators. The commands below read the API token with read -rs, which works in bash and zsh and doesn’t show the token as you paste it. Run the read command on its own, before the commands that use the token.

Read the API token:

printf 'API token: '; read -rs API_TOKEN; echo

Write the parameters to a file only you can read, so the token stays out of your shell history:

rm -f stackset-params.json
if [ -n "$API_TOKEN" ]; then
  (umask 077 && cat > stackset-params.json) <<EOF
[
  {"ParameterKey": "OrgName", "ParameterValue": "<ORG_NAME>"},
  {"ParameterKey": "ApiToken", "ParameterValue": "$API_TOKEN"}
]
EOF
else
  echo 'API_TOKEN is empty: run the read command first'
fi
unset API_TOKEN

Create the StackSet from the template in the CloudFormation tab, saved as cortex-xcor-aws-logs.yaml, and then delete the parameters file:

if [ -f stackset-params.json ]; then
  aws cloudformation create-stack-set --region <ADMIN_REGION> \
    --stack-set-name cortex-xcor-aws-logs \
    --template-body file://cortex-xcor-aws-logs.yaml \
    --permission-model SERVICE_MANAGED \
    --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
    --capabilities CAPABILITY_IAM \
    --parameters file://stackset-params.json
  rm -f stackset-params.json
else
  echo 'stackset-params.json is missing: run the previous step first'
fi

With automatic deployment, an account that joins a target organizational unit gets a stack, and an account that leaves has its stack deleted. The deleted stack’s failed-data bucket is kept.

Deploy the StackSet to your organizational units and regions. <OU_ID> can be the organization’s root ID (r-...) to cover every account.
OP_ID=$(aws cloudformation create-stack-instances --region <ADMIN_REGION> \ --stack-set-name cortex-xcor-aws-logs \ --deployment-targets OrganizationalUnitIds=<OU_ID> \ --regions <REGION> <REGION> \ --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \ --query OperationId --output text)
These preferences deploy to every account and region at once and keep going past accounts that fail. Without them, CloudFormation deploys to one account at a time and stops at the first failure, which also skips every later region. The preferences don’t apply to automatic deployments.

Wait for the operation to finish, and then list the stack instances that failed. With these preferences, the operation reports SUCCEEDED even when some instances failed.

while :; do
  STATUS=$(aws cloudformation describe-stack-set-operation --region <ADMIN_REGION> \
    --stack-set-name cortex-xcor-aws-logs --operation-id "$OP_ID" \
    --query StackSetOperation.Status --output text)
  case "$STATUS" in RUNNING|QUEUED|STOPPING) sleep 30 ;; *) break ;; esac
done
echo "Operation $STATUS"
aws cloudformation list-stack-instances --region <ADMIN_REGION> \
  --stack-set-name cortex-xcor-aws-logs \
  --filters Name=DETAILED_STATUS,Values=FAILED

An instance fails in an account and region that already has an account-level subscription policy, because each account and region allows only one. For each of those, override CreateAccountPolicy to false, wait for the operation with the commands in step 5, and then subscribe log groups individually in that account with the DeliveryStreamArn and SubscriptionRoleArn outputs of its stack:

OP_ID=$(aws cloudformation update-stack-instances --region <ADMIN_REGION> \
  --stack-set-name cortex-xcor-aws-logs \
  --deployment-targets Accounts=<ACCOUNT_ID> \
  --regions <REGION> \
  --parameter-overrides ParameterKey=CreateAccountPolicy,ParameterValue=false \
  --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \
  --query OperationId --output text)

Target accounts rather than an organizational unit, because an override set on an organizational unit doesn’t apply to accounts that join it later. Fix failed instances promptly. Every later update to the StackSet tries to create them again, which makes the update much slower. A StackSet with service-managed permissions doesn’t deploy to the management account. To collect that account’s logs, set it up on its own as described earlier on this page. To rotate the token, read the new token on its own:

printf 'API token: '; read -rs API_TOKEN; echo

Then update the StackSet’s ApiToken parameter, keeping the other parameters:

rm -f stackset-params.json
if [ -n "$API_TOKEN" ]; then
  (umask 077 && cat > stackset-params.json) <<EOF
[
  {"ParameterKey": "OrgName", "UsePreviousValue": true},
  {"ParameterKey": "StreamName", "UsePreviousValue": true},
  {"ParameterKey": "CreateAccountPolicy", "UsePreviousValue": true},
  {"ParameterKey": "FailedDataExpirationDays", "UsePreviousValue": true},
  {"ParameterKey": "ApiTokenSecretArn", "UsePreviousValue": true},
  {"ParameterKey": "ApiToken", "ParameterValue": "$API_TOKEN"}
]
EOF
  OP_ID=$(aws cloudformation update-stack-set --region <ADMIN_REGION> \
    --stack-set-name cortex-xcor-aws-logs \
    --use-previous-template \
    --capabilities CAPABILITY_IAM \
    --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \
    --parameters file://stackset-params.json \
    --query OperationId --output text)
  rm -f stackset-params.json
else
  echo 'API_TOKEN is empty: run the read command first'
fi
unset API_TOKEN

Wait for the operation and list failed instances with the commands in step 5. If any instance is listed, fix its cause and run the update again, which retries the instances that didn’t update. If CloudFormation reports that another operation is in progress, such as an automatic deployment, wait for it to finish and run the update again. If the management account’s own setup uses the same token, update its secret in each region where you set it up. Read the new token again:

printf 'API token: '; read -rs API_TOKEN; echo

Then update each region:

if [ -n "$API_TOKEN" ]; then
  for region in <REGION> <REGION>; do
    printf '{"api_key":"%s"}' "$API_TOKEN" |
      aws secretsmanager put-secret-value --region "$region" \
        --secret-id cortex-xcor/aws-logs-api-token \
        --secret-string file:///dev/stdin
  done
else
  echo 'API_TOKEN is empty: run the read command first'
fi
unset API_TOKEN

Each Firehose stream starts using the new token within 10 minutes of its secret changing. Don’t revoke the old token until every stack and the management account’s secret have updated, and 10 more minutes have passed. This page doesn’t cover sending logs from many accounts to one stream. To send only some log groups, or when another tool holds the account-level policy, add a subscription filter to each log group:

aws logs put-subscription-filter --region <REGION> \
  --log-group-name <LOG_GROUP> \
  --filter-name cortex-xcor-logs \
  --filter-pattern "" \
  --destination-arn <STREAM_ARN> \
  --role-arn <ROLE_ARN>

Don’t include these log groups in the account-level policy, or their logs arrive twice.

Send logs straight to Firehose

Some services deliver logs to a Firehose stream without a log group. Give each service its own stream, created in the console with the same settings under a unique name. By default, the Terraform and CloudFormation configurations also create the account subscription policy, so don’t run them again for these streams. The account-level policy sends each additional stream’s error log group, /aws/kinesisfirehose/<STREAM_NAME>, to the stream targeted by the policy unless you add the error log group to the policy exclusion. Cortex XCOR recognizes these logs on its own:

AWS WAF: By the stream name, which AWS requires to start with aws-waf-logs-, and by the log format.
Amazon API Gateway: By a stream name that starts with amazon-apigateway-.
VPC Flow Logs: By the default log format. For a custom format, add the stream parameter xcor_integration with the value aws_vpc.

For other logs sent straight to Firehose, set the xcor_integration parameter to a recognized AWS integration ID, such as aws_cloudtrail, aws_eks, aws_lambda, or aws_rds. Cortex XCOR ignores unknown values. Logs it can’t otherwise recognize are stored with the label aws_other.

Validate log ingestion

After setup, wait for logs to arrive before validating ingestion.

In the Amazon Data Firehose console, open the stream. HTTP endpoint delivery success should report 100%, and Records delivered to HTTP endpoint should be non-zero.
In Cortex XCOR, go to Explorers > Logs Explorer and run:
["xcor.integration.name"] = "aws" | summarize count() by xcor_integration
Each AWS service sending logs appears as a row.

Log labels and fields

Every log gets two labels:

xcor.integration.name: always aws, the integration’s catalog ID.
xcor_integration: the AWS service that wrote the log, such as aws_lambda, aws_rds, aws_eks, aws_vpc, or aws_cloudtrail. Cortex XCOR determines the service from the log group name, log stream name, log format, or a stream parameter. Logs it can’t attribute to an AWS service get aws_other and are stored like any other log.

Keep the default AWS log group names. The exception is Amazon ECS: Cortex XCOR recognizes ECS logs only in log groups under /aws/ecs/ or /aws/vendedlogs/ecs/, so the /ecs/<task definition family> group that the ECS console creates gets aws_other. Cortex XCOR can still recognize some services from custom log group names by using the log stream name or format. Other logs get aws_other. Every log also carries these fields, set from the delivery rather than the log message:

Field	Value
`aws.account_id`	The account that owns the log group, or the Firehose stream’s account for direct delivery
`aws.region`	The log group’s region, when available, or the Firehose stream’s region
`aws.firehose.arn`, `aws.firehose.stream`, `aws.firehose.request_id`	The stream and request
`aws.firehose.attributes.<name>`	Each parameter you set on the stream

Logs delivered through a CloudWatch Logs subscription also carry these fields:

Field	Value
`aws.log_group`, `aws.log_stream`	Where CloudWatch Logs stored the log
`aws.subscription_filters`, `aws.event_id`	From the CloudWatch Logs subscription
`aws.timestamp`	When CloudWatch Logs stored the event

For an organization trail, aws.account_id is the management account. CloudTrail’s recipientAccountId field identifies the account for each event, and its awsRegion field identifies the event’s region. Cortex XCOR parses a JSON log message into fields and stores any other message in message. Fields set by the log take precedence over the following fields that Cortex XCOR sets:

Time: The log’s timestamp field, then CloudTrail’s eventTime, then the time CloudWatch Logs stored the event. A log sent straight to Firehose without a time gets the time Cortex XCOR receives it.
Service: The service field is the xcor_integration value without aws_, such as cloudtrail, eks, or rds. Lambda logs get the function name. Logs labeled aws_other get the log group, or aws_other when there’s no log group.
Message: For CloudTrail, message is the event name, such as AssumeRole.
Severity: CloudTrail events with an errorCode get ERROR. Other CloudTrail events get INFO, and other logs without a severity get your organization’s default.

Troubleshooting

Firehose records delivery errors in the log group /aws/kinesisfirehose/<STREAM_NAME>.

401 errors, or a content-type: application/json error whose raw response starts with 401: The token isn’t an API token for a restricted service account with write-only permission in this organization. After a token change, Firehose can keep using the old token for up to 10 minutes.
Secrets Manager errors: The secret must be JSON with the key api_key, and the stream role needs secretsmanager:GetSecretValue on the secret.
404 errors, or a content-type: application/json error whose raw response starts with 404: The endpoint URL is incorrect. It must end in /api/v1/data/logs/ingest/integrations/aws.
Logs labeled aws_other: The log group has a custom name or is the ECS console’s /ecs/ group, or the logs came straight from a service that Cortex XCOR couldn’t identify from the stream name or log format. Set the stream parameter xcor_integration to a recognized AWS integration ID. For example, use aws_vpc for VPC Flow Logs in a custom format.
Delivery succeeds but some logs are missing: Cortex XCOR drops logs more than 24 hours old or more than 24 hours in the future. CloudWatch Logs can deliver older events, such as after a long outage.
Duplicate logs: After an error, Firehose sends the entire batch again, so some records can arrive twice.

Records that still fail after the retry window go to the S3 bucket. Cortex XCOR doesn’t read the bucket, so those records aren’t ingested unless they’re sent again within 24 hours of their timestamps.

Remove log delivery

Until you remove it, the account-level policy keeps sending every log group in the region to the stream, and Firehose continues billing for it.

For Terraform, run terraform destroy.
For CloudFormation, note the failed-data bucket’s name, and then delete the stack:
aws cloudformation describe-stack-resource --region <REGION> \ --stack-name cortex-xcor-aws-logs --logical-resource-id FailedDataBucket \ --query StackResourceDetail.PhysicalResourceId --output text aws cloudformation delete-stack --region <REGION> --stack-name cortex-xcor-aws-logs
From template 1.1.0, deleting the stack keeps the failed-data bucket, so leftover records can’t make the delete fail. Its records still expire, and you can delete it with aws s3 rb s3://<BUCKET> --force when you no longer need them, where <BUCKET> is the name the first command printed. If you passed ApiToken, deleting the stack also deletes its secret immediately, without a recovery window.

For a StackSet, delete the stack instances from the organizational units and regions you deployed to:

OP_ID=$(aws cloudformation delete-stack-instances --region <ADMIN_REGION> \
  --stack-set-name cortex-xcor-aws-logs \
  --deployment-targets OrganizationalUnitIds=<OU_ID> \
  --regions <REGION> <REGION> \
  --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \
  --no-retain-stacks \
  --query OperationId --output text)

Wait for the operation with the commands in step 5 of Set up many accounts and regions, and then delete the StackSet. Each account keeps its failed-data bucket.

aws cloudformation delete-stack-set --region <ADMIN_REGION> --stack-set-name cortex-xcor-aws-logs

In the console, delete the account-level policy, then the Firehose stream, subscription role, failed-data bucket, and the stream’s error log group, which the console creates:

aws logs delete-account-policy --region <REGION> \
  --policy-name cortex-xcor-logs --policy-type SUBSCRIPTION_FILTER_POLICY
aws logs delete-log-group --region <REGION> \
  --log-group-name /aws/kinesisfirehose/cortex-xcor-logs

For individual subscriptions, remove each subscription with:

aws logs delete-subscription-filter --region <REGION> \
  --log-group-name <LOG_GROUP> --filter-name cortex-xcor-logs

If you use Terraform, delete resources by hand, or delete a stack from a template before 1.1.0, empty the failed-data bucket first, or its deletion fails. If you created the token secret yourself and passed ApiTokenSecretArn, delete it too, because the setup doesn’t own it. By default, Secrets Manager schedules deletion with a 30-day recovery window. During that window, another setup attempt with the same secret name fails. The --force-delete-without-recovery option skips the recovery window. Deletion is asynchronous, so wait and retry if the name remains unavailable:

aws secretsmanager delete-secret --region <REGION> --secret-id cortex-xcor/aws-logs-api-token \
  --force-delete-without-recovery

Cortex XCOR

Palo Alto Networks Telemetry Pipeline

More information

What it collects

How it works

Before you begin

Required permissions

Firehose settings

Set up log delivery

Set up many accounts and regions

Send logs straight to Firehose

Validate log ingestion

Log labels and fields

Troubleshooting

Remove log delivery

​What it collects

​How it works

​Before you begin

​Required permissions

​Firehose settings

​Set up log delivery

​Set up many accounts and regions

​Subscribe individual log groups

​Send logs straight to Firehose

​Validate log ingestion

​Log labels and fields

​Troubleshooting

​Remove log delivery

What it collects

How it works

Before you begin

Required permissions

Firehose settings

Set up log delivery

Set up many accounts and regions

Subscribe individual log groups

Send logs straight to Firehose

Validate log ingestion

Log labels and fields

Troubleshooting

Remove log delivery