> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SQL DB Input source plugin

> Use the SQL DB Input source plugin with Palo Alto Networks Telemetry Pipeline.

export const entity_0 = "SQL DB Input source plugin"

export const plugin_0 = "SQL DB Input source plugin"

<Info>
  This information is for Palo Alto Networks Telemetry Pipeline, which is a standalone
  product separate from Palo Alto Networks Cortex XCOR.
</Info>

The SQL DB Input [source plugin](/ingest/pipeline/plugins/source-plugins)
(name: `sqldb`, alias: `SQL_DB_Input`) lets you ingest log data from a
relational database into a telemetry pipeline. If you write your logs to a
relational database, you can use the SQL DB Input plugin to convert that data
into a pipeline-friendly format.

This plugin supports the following SQL formats:

* Postgres
* MySQL
* Oracle
* Microsoft SQL Server
* SQLite

This is a
[pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins)
source plugin.

<Note>
  This plugin doesn't support duplicates of itself within the same pipeline.
</Note>

## Supported telemetry types

The {plugin_0} for Palo Alto Networks Telemetry Pipeline supports these telemetry types:

| Logs | Metrics | Traces |
| :-: | :-: | :-: |
| <Icon icon="circle-check" color="green" /> | <Icon icon="ban" color="red" /> | <Icon icon="ban" color="red" /> |

## Configuration parameters

Use the parameters in this section to configure the {entity_0}. The
Palo Alto Networks Telemetry Pipeline web interface uses the items in the **Name** column to
describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files)
use the items in the **Key** column as YAML keys.

### General

| Name | Key | Description | Default |
| - | - | - | - |
| **DB Type** | `driver` | Required. The SQL driver for your relational data. Accepted values: `postgres`, `mysql`, `oracle`, `sqlserver`, `sqlite`. | `postgres` |
| **Database Connection String** | `dsn` | Required. The data source name for your database, in string format. | *none* |
| **SQL Query** | `query` | Required. The SQL query to perform. This query supports named arguments in `@named` format. | *none* |

### Advanced

| Name | Key | Description | Default |
| - | - | - | - |
| **Argument Column** | `columnsForArgs` | A list of column names to turn into arguments, separated by spaces. The most recently scanned row in each specified column becomes a `@named` argument you can use to paginate data. For example, if this key has a value of `id log_time`, the plugin creates two arguments you can use in queries: `@last_id` and `@last_log_time`. | *none* |
| **Time Column** | `timeFrom` | The column in your database with the time when each log was ingested. | If unassigned, each ingested log is assigned a time value equivalent to the time the plugin runs its query. |
| **Time Format** | `timeFormat` | The time format of your `timeFrom` value. Accepted values: `unix_sec`, `unix_ms`, `unix_us`, `unix_ns`, or any RFC time format layout. | If `timeFrom` is an integer: `unix_ns`. If `timeFrom` is a string: `2006-01-02T15:04:05.999999999Z07:00`. |
| **Fetch Interval** | `fetchInterval` | How often to query your database. This value must specify one or more integer and one or more time unit, like `5m` or `1m30s`. Valid time units: `ns`, `us`, `ms`, `s`, `m`, `h`. | `1s` |
| **Storage Key** | `storageKey` | A key to store the plugin's metadata, which lets the plugin store arguments and other configuration data even if the plugin restarts. By default, the plugin writes its own metadata in `sqldb_{hash}.gob` format, where `{hash}` is a string generated from your query arguments. You can also specify your own value, which can follow any format and use any file extension. | *none* |
| **Storage Path** | `storageDir` | The path to the directory where the plugin writes its data. You can use an absolute or relative path, but Palo Alto Networks Telemetry Pipeline recommends using an absolute path. | `/data/storage` (If this directory does not exist, the plugin writes to your operating system's default directory for temporary data.) |
| **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced. <p />For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide. <p />For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* |

## Example queries

The following examples describe how to configure the SQL DB Input plugin in different
hypothetical scenarios.

### Serial ID

Given the following database structure, which includes an auto-increasing `id`:

```sql theme={null}
CREATE TABLE logs (
    id SERIAL PRIMARY KEY,
    payload JSON NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT now()
);
```

You can use the SQL DB Input plugin to paginate over your data using `id`. To include
such a query in your pipeline, configure the plugin as follows:

```yaml theme={null}
pipeline:
  inputs:
    - name: sqldb
      dsn: "postgres://user:password@localhost:5432/dbname"
      query: |-
        SELECT *
        FROM logs
        WHERE @last_id IS NULL OR id > @last_id
        ORDER BY id
        LIMIT 100
      columnsForArgs: id
```

The previous query sorts by `id`, and then uses the `id` value from the previous query to
paginate data.

### Time-based pagination

Given the following database structure, which includes a non-sortable `id`:

```sql theme={null}
CREATE TABLE logs (
    id UUID PRIMARY KEY,
    payload JSON NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT now()
);
```

You can use a combination of `id` and `created_at` to paginate data. To include
such a query in your pipeline, configure the plugin as follows:

```yaml theme={null}
pipeline:
  inputs:
    - name: sqldb
      dsn: "postgres://user:password@localhost:5432/dbname"
      query: |-
        SELECT *
        FROM logs
        WHERE @last_id IS NULL OR @last_created_at IS NULL OR (created_at >= @last_created_at OR (created_at = @last_created_at AND id > @last_id))
        ORDER BY created_at, id
        LIMIT 100
      columnsForArgs: "id created_at"
```

The previous query sorts by both `id` and `created_at`, then uses the `id` and
`created_at` values from the previous query to paginate data.


## Related topics

- [S3 Input (SQS) source plugin](/ingest/pipeline/plugins/source-plugins/s3-sqs.md)
- [S3 Input (One Time) source plugin](/ingest/pipeline/plugins/source-plugins/amazon-s3.md)
- [Amazon Kinesis Stream Input source plugin](/ingest/pipeline/plugins/source-plugins/aws-streams.md)
- [GCP PubSub Input source plugin](/ingest/pipeline/plugins/source-plugins/gcp-pubsub.md)
- [Ingest telemetry data with source plugins](/ingest/pipeline/plugins/source-plugins.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.