Amazon Kinesis Firehose destination plugin - Cortex XCOR Documentation

This information is for Palo Alto Networks Telemetry Pipeline, which is a standalone product separate from Palo Alto Networks Cortex XCOR.

The Amazon Kinesis Firehose destination plugin (name: kinesis_firehose) lets you configure your telemetry pipeline to collect various data types such as logs, metrics, traces, and events and stream them directly to your Kinesis Firehose delivery stream.

Supported telemetry types

The Amazon Kinesis Firehose destination plugin for Palo Alto Networks Telemetry Pipeline supports these telemetry types:

Logs	Metrics	Traces

Configuration parameters

Use the parameters in this section to configure the Amazon Kinesis Firehose destination plugin. The Palo Alto Networks Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

Required

Name	Key	Description	Default
Region	`region`	Required. The AWS region to which to send your data.	`us-east-1`
Firehose delivery stream name	`delivery_stream`	Kinesis Firehose name.	none

AWS Authentication

Name	Key	Description	Default
IAM Role ARN	`role_arn`	Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access.	none
Firehose API Endpoint	`endpoint`	Custom endpoint for the Firehose API.	none
STS API Endpoint	`sts_endpoint`	Custom endpoint for the STS API.	none
External ID for STS API	`external_id`	Specifies an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID.	none

Advanced

Name	Key	Description	Default
Compression	`compression`	Compression type for Firehose records. Each log record is individually compressed and sent to Firehose. Accepted values: `gzip`, `arrow` (with `arrow` available only if Apache Arrow is enabled at compile time).	none
Time Key	`time_key`	Adds the timestamp to the record under this key. By default, the timestamp from Palo Alto Networks Telemetry Pipeline isn’t added to records sent to Kinesis.	none
Time Key Format	`time_key_format`	`strftime`-compliant format string for the timestamp. This option is used with `time_key`.	`%Y-%m-%dT%H:%M:%S`
Log Key	`log_key`	By default, the entire log record is sent to Kinesis. If you specify a key name with this option, only the value of that key is sent to Kinesis. For example, if you’re using the Fluentd Docker log driver, you can specify `log_key log`, and only the log message is sent to Kinesis.	none
Enable Auto Retry Requests	`auto_retry_requests`	Whether to immediately retry failed requests to AWS services one time. This option doesn’t affect the normal Palo Alto Networks Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: `true`, `false`.	`false`

Security and TLS

Name	Key	Description	Default
TLS	`tls`	If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`.	`false`
TLS Certificate Validation	`tls.verify`	If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`.	`on`
TLS Debug Level	`tls.debug`	Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose).	`1`
CA Certificate File Path	`tls.ca_file`	Absolute path to CA certificate file.	none
Certificate File Path	`tls.crt_file`	Absolute path to certificate file.	none
Private Key File Path	`tls.key_file`	Absolute path to private key file.	none
Private Key Path Password	`tls.key_passwd`	Password for private key file.	none
TLS SNI Hostname Extension	`tls.vhost`	Hostname to be used for TLS SNI extension.	none

Advanced Networking

Name	Key	Description	Default
DNS Mode	`net.dns.mode`	Selects the primary DNS connection type, which can be `TCP` or `UDP`.	none
DNS Resolver	`net.dns.resolver`	Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`.	none
Prefer IPv4	`net.dns.prefer_ipv4`	Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`.	`false`
Keepalive	`net.keepalive`	Enables or disables Keepalive support. Accepted values: `true`, `false`.	`true`
Keepalive Idle Timeout	`net.keepalive_idle_timeout`	Sets the maximum time allowed for an idle Keepalive connection.	`30s`
Max Connect Timeout	`net.connect_timeout`	Sets the maximum time allowed to establish a connection, which includes the TLS handshake.	`10s`
Max Connect Timeout Log Error	`net.connect_timeout_log_error`	Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`.	`true`
Source Address	`net.source_address`	Specifies the network address to bind for data traffic.	none
Max Keepalive Recycle	`net.keepalive_max_recycle`	Sets the maximum number of times a keepalive connection can be used before it’s retired.	`2000`