> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Datagen source plugin

> Use the Datagen source plugin to generate simulated log data to test your pipeline configuration.

export const entity_0 = "Datagen source plugin"

export const plugin_0 = "Datagen source plugin"

<Info>
  This information is for Palo Alto Networks Telemetry Pipeline, which is a standalone
  product separate from Palo Alto Networks Cortex XCOR.
</Info>

The Datagen [source plugin](/ingest/pipeline/plugins/source-plugins)
(name: `datagen`, alias: `apache_common`) generates simulated log data through
[Gofakeit](https://github.com/brianvoe/gofakeit). You can use this simulated
data to test your pipeline configurations.

Although this is a self-contained plugin that doesn't communicate with external
sources, it's still classified as a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins)
source plugin.

<Note>
  This plugin doesn't support the use of a
  [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the
  Pipeline Builder interface.
</Note>

<Note>
  This plugin doesn't support duplicates of itself within the same pipeline.
</Note>

## Supported telemetry types

The {plugin_0} for Palo Alto Networks Telemetry Pipeline supports these telemetry types:

| Logs | Metrics | Traces |
| :-: | :-: | :-: |
| <Icon icon="circle-check" color="green" /> | <Icon icon="ban" color="red" /> | <Icon icon="ban" color="red" /> |

## Configuration parameters

Use the parameters in this section to configure the {entity_0}. The
Palo Alto Networks Telemetry Pipeline web interface uses the items in the **Name** column to
describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files)
use the items in the **Key** column as YAML keys.

### General

| Name | Key | Description | Default |
| - | - | - | - |
| **Template** | `template` | Required. A JSON object that specifies how to generate log data. Accepted types: Apache common, Syslog rfc3164, or Syslog rfc5424. See [Templates](#templates). | *none* |
| *none* | `rate` | How often to generate logs, in seconds. | `1` |

### Advanced

| Name | Key | Description | Default |
| - | - | - | - |
| **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced. <p />For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide. <p />For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* |

## Templates

Templates define the structure and content of your simulated log data. When you
configure the Datagen source plugin in a pipeline's YAML configuration file, use
the `template` key to create a JSON object. For example:

```yaml theme={null}
pipeline:
  inputs:
    - name: datagen
      template: |-
        {
            "host":            "{{ IPv4Address }}",
            "user": {
              "user-identifier": "{{ Username }}",
              "auth-user-id":    "{{ UUID }}",
            }
            "method":          "{{ HTTPMethod }}",
            "request":         "{{ URLPath }}",
            "protocol":        "{{ HTTPVersion }}",
            "response-code":   "{{ HTTPStatusCode }}",
            "bytes":           "{{ Number 0 30_000 }}"
        }
```

When the Datagen plugin generates logs, each log matches the format of your template,
but any included [functions](#functions) are replaced by simulated data. For best
results, Palo Alto Networks Telemetry Pipeline recommends creating a JSON
object that uses custom strings
for its keys and functions for its values.

## Functions

The Datagen plugin uses functions to generate randomized values that follow a
consistent pattern. For example, `{{ HTTPMethod }}` can output a string like
`GET`, `POST`, or `PUT`, but not a string like `GREEN` or `SUNDAY`.

To invoke a function in a [template](#templates), surround the name of that
function with double curly braces.

### Suggested functions

You can use any of the available
[Gofakeit functions](https://github.com/brianvoe/gofakeit#functions)
in a template. The following functions are best suited for generating realistic
log data:

* `{{ HTTPMethod }}`: Returns an HTTP method, like `GET`.
* `{{ HTTPStatusCode }}`: Returns an HTTP status code, like `403`.
* `{{ HTTPVersion }}`: Returns an HTTP protocol version, like `HTTP/1.1`.
* `{{ IPv4Address }}`: Returns a random IPv4 address.
* `{{ IPv6Address }}`: Returns a random IPv6 address.
* `{{ Username }}`: Returns a random username, like `Bailey1270`.
* `{{ UUID }}`: Returns a string of numbers in UUID format.
* `{{ Date }}`: Returns a random time in UTC format.
* `{{ LogLevel syslog }}`: Returns a log type, like `alert`.
* `{{ Number 0 30_000 }}`: Returns a number between 0 and 30,000.
* `{{ URLPath }}`: Returns a string that resembles a URL path, like `/foo/bar`.
  This function is unique to Datagen and isn't included in Gofakeit.
* `{{ URLScheme }}`: Returns either `http` or `https`. This function is unique
  to Datagen and isn't included in Gofakeit.

## Example output

For a pipeline with the following template:

```yaml theme={null}
pipeline:
  inputs:
    - name: datagen
      template: |-
        {
            "response-code":   "{{ HTTPStatusCode }}",
            "bytes":           "{{ Number 0 30_000 }}"
            "host":            "{{ IPv4Address }}",
        }
```

The Datagen source plugin generates simulated log data in the following format:

```shell theme={null}
{"response-code"=>201.000000, "bytes"=>29600.000000, "host"=>"64.15.54.123"}

{"response-code"=>302.000000, "bytes"=>429.000000, "host"=>"214.221.240.201"}

{"response-code"=>205.000000, "bytes"=>3507.000000, "host"=>"27.79.197.250"}

{"response-code"=>504.000000, "bytes"=>758.000000, "host"=>"211.135.114.51"}

{"response-code"=>404.000000, "bytes"=>12190.000000, "host"=>"84.74.177.0"}
```


## Related topics

- [Dummy HTTP source plugin](/ingest/pipeline/plugins/source-plugins/dummy-http.md)
- [Palo Alto Networks Telemetry Pipeline release notes](/ingest/pipeline/v2/release-notes.md)
- [Ingest telemetry data with source plugins](/ingest/pipeline/plugins/source-plugins.md)
- [Mock Data source plugin](/ingest/pipeline/plugins/source-plugins/mock-data.md)
- [Absolute source plugin](/ingest/pipeline/plugins/source-plugins/absolute.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.