Amazon Kinesis Stream Input source plugin - Cortex XCOR Documentation

This information is for Palo Alto Networks Telemetry Pipeline, which is a standalone product separate from Palo Alto Networks Cortex XCOR.

The Amazon Kinesis Stream Input source plugin (name: aws_kinesis_stream) lets you retrieve data from Amazon Kinesis Stream and ingest it into a telemetry pipeline. This is a pull-based source plugin.

This plugin doesn’t support duplicates of itself within the same pipeline.

Supported telemetry types

The Amazon Kinesis Stream Input source plugin for Palo Alto Networks Telemetry Pipeline supports these telemetry types:

Logs	Metrics	Traces

Configuration parameters

Use the parameters in this section to configure the Amazon Kinesis Stream Input source plugin. The Palo Alto Networks Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

Required

Name	Key	Description	Default
AWS Access Key	`aws_access_key_id`	Required. Your AWS access key path.	none
AWS Secret Key	`aws_secret_access_key`	Required. Your AWS secret key path.	none
AWS Kinesis Stream Name	`stream_name`	Required. The AWS Kinesis stream name.	none
AWS Region	`aws_region`	Required. The name of the AWS region. For example, `us-east-1`.	none

Advanced

Name	Key	Description	Default
Number of fetched records	`limit`	The maximum number of records to fetch from the stream on each iteration.	`1000`
Pull Interval	`empty_interval`	How long to wait before fetching more records. If the stream returns empty, Palo Alto Networks Telemetry Pipeline will continue to pull.	`1s`
Memory Buffer Limit	`mem_buf_limit`	Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit’s rules for unit sizes. If unspecified, no limit is enforced. For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet workload type. To learn more, see the v2 backpressure guide. For v3 pipelines, this parameter is independent from the OpenTelemetry `memory_limiter` and `batch` processors.	none