> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Apache Spark

> Apache Spark application metrics read from a driver's REST API, including executor memory and task throughput, garbage collection, shuffle volume, and persisted RDD storage.

The Apache Spark integration requires CXDOT Collector 1.4.0 or greater.

[Apache Spark](https://spark.apache.org/) is a unified engine for large-scale data
analytics. Use the Apache Spark CXDOT Collector integration to collect app, job,
stage, executor, storage, and streaming metrics from Spark drivers running in your
environment.

The Apache Spark CXDOT Collector integration supports Apache Spark 3.4.0 or greater.

## Supported telemetry types

The Apache Spark CXDOT Collector integration supports these telemetry types:

| Type | Supported |
| :-: | :-: |
| Logs | No |
| Metrics | Yes |
| Traces | No |
| Events | No |

## Prerequisites

The Apache Spark CXDOT Collector integration has the following prerequisites:

* Make the driver web interface for each Spark app reachable from the CXDOT
  Collector. Point targets at app driver web interfaces, not Spark standalone
  masters, Apache Hadoop Yet Another Resource Negotiator (YARN) ResourceManagers,
  or Apache Mesos masters.
* To collect Structured Streaming query metrics, set
  `spark.sql.streaming.metricsEnabled` to `true` for the app. Spark then
  [reports each query's metrics](https://spark.apache.org/docs/latest/streaming/apis-on-dataframes-and-datasets.html#reporting-metrics-using-dropwizard)
  through the metrics endpoint that the driver web interface serves at
  `/metrics/json` by default. If your apps serve that endpoint at another path,
  set `metrics_servlet_path` to match. Apps that use Spark Streaming report their
  streaming statistics without any setting. To stop collecting both kinds of
  streaming metrics, set `streaming_metrics` to `false`.
* To collect the peak memory of each driver's and executor's process tree, set
  [`spark.executor.processTreeMetrics.enabled`](https://spark.apache.org/docs/latest/monitoring.html#executor-metrics)
  to `true` for the app.

## Configure

The Apache Spark integration is enabled by default. You can configure targets either by
annotating driver pods in your Kubernetes cluster or by listing static endpoints in your
Helm values. To configure the integration, follow these steps:

1. To collect from Spark driver pods in Kubernetes, add a CXDOT integration annotation
   to the pod template. The annotation key must end with the container name. For
   example, add the following annotation to your Spark driver:

   ```yaml theme={null}
   spec:
     template:
       metadata:
         annotations:
           integrations.cxdot.paloaltonetworks.com/spark: |
             schema_version: v1.14
             spark:
               endpoints:
                 - endpoint: ${target.host}:${target.port.ui}
       spec:
         containers:
           - name: spark
             image: apache/spark:3.5.3
             ports:
               - name: ui
                 containerPort: 4040
   ```

   The Collector replaces `${target.host}` with the pod IP address and
   `${target.port.ui}` with the container port named `ui`.

   The Collector also discovers Spark driver pods labeled `spark-role=driver` that
   expose port `4040`. A CXDOT integration annotation takes precedence over the label
   for its container. To stop collecting from a container, set `spark: {enabled: false}`
   in the annotation. Metrics from each discovered driver pod include that pod's
   Kubernetes metadata. For more information, see
   [autodiscovery](https://docs-xcor.paloaltonetworks.com/ingest/xcor/collector/discover/kubernetes.md).

2. Optional: To collect from Spark apps outside your Kubernetes cluster, configure
   static driver targets. A non-empty `endpoints` list turns off discovery for that
   integration instance, and the Collector collects only the listed targets. To
   run discovery and static targets together, add a named integration instance. For
   example, add the following to the `values.yaml` for the CXDOT Collector Helm chart:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         spark: {}
         spark/external:
           endpoints:
             - endpoint: "spark-driver.example.com:4040"
   ```

   The empty `spark` entry keeps discovery active. `spark/external` is an
   independent, named instance that collects from the listed driver.

To disable the default Spark integration instance, set `enabled` to `false`. For
example, add the following to the `values.yaml` for the CXDOT Collector Helm chart:

```yaml theme={null}
collection:
  cxdot:
    integrations:
      spark:
        enabled: false
```

### Validate

To validate the Apache Spark CXDOT Collector integration, follow these steps:

1. In [Live Telemetry Analyzer](https://docs-xcor.paloaltonetworks.com/investigate/analyze/telemetry-analyzer.md),
   filter for `__name__=cxdot.integration.target.health` and
   `cxdot.integration.name=spark`. Confirm the metric reports `1` for each Spark
   target. The `cxdot.integration.check` label has the value `spark`, and
   `cxdot.integration.target` identifies the driver.

2. In [Metrics Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/explorer.md),
   run the following query:

   ```text theme={null}
   count by ("server.address", "server.port", "spark.application.name") ({"spark.executor.task.assigned"})
   ```

   Confirm the query returns a value for each active Spark app.

### Troubleshooting

* No app metrics appear for a standalone master, YARN ResourceManager, or
  Apache Mesos master: Configure each app's driver web interface as a target.
  The integration doesn't use cluster managers to discover apps.

## Configuration reference

Configure one Apache Spark integration instance with the following settings. In Helm values,
place these settings under `collection.cxdot.integrations.spark`. In a Collector configuration
file, place them under `cxdot.integrations.spark`.

### Optional settings

* **`enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to enable this Spark integration instance. If true, the Collector runs the instance.
  If false, the Collector doesn't run it.

* **`endpoints`**
  Type: `array of object`. Optional. Default: `[]`.
  Spark app driver web interface targets. When this list is empty or omitted, the integration
  uses automatic target discovery. When the list contains targets, this integration instance
  collects from exactly those targets and doesn't automatically discover targets. Configure a
  second named instance to use both methods. Specify each target as `host:port` to use HTTP, or
  as a full HTTP or HTTPS URL. The URL must identify the web interface root and must not include
  a Spark REST API path. Configure app driver web interfaces only. The integration doesn't
  discover apps through cluster managers.

* **`endpoints[].endpoint`**
  Type: `string`. Required.
  Spark app driver web interface target as `host:port` or a full HTTP or HTTPS URL. For
  `host:port`, the Collector uses HTTP.

* **`endpoints[].cluster_name`**
  Type: `string`. Optional.
  Name of the Spark cluster the driver belongs to, reported as the `spark.cluster.name`
  attribute. When omitted, the attribute is not set.

* **`application_names`**
  Type: `array of string`. Optional. Default: `[]`.
  Spark app names to collect from each target. When the list is empty, the Collector collects
  every app reported by the target.

* **`metrics_servlet_path`**
  Type: `string`. Optional. Default: `/metrics/json`.
  Path of the JSON metrics endpoint on each target's driver web interface, where Spark publishes
  Structured Streaming query metrics. Change it only for apps that move the endpoint with the
  `spark.metrics.conf.*.sink.servlet.path` Spark setting. When a target answers this path with
  HTTP 404, the Collector collects no Structured Streaming metrics from it.

* **`streaming_metrics`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to collect Spark Streaming and Structured Streaming metrics. When `false`, the
  Collector doesn't request either streaming endpoint, so a target whose JSON metrics endpoint
  returns errors no longer reports failed collections.

* **`collection_interval`**
  Type: `duration`. Optional. Default: `30s`.
  How often the Collector collects metrics from each Spark target.

* **`timeout`**
  Type: `duration`. Optional. Default: `30s`.
  Maximum time allowed to collect metrics from one Spark target. The value must not exceed
  `collection_interval`.

* **`tls`**
  Type: `object`. Optional.
  TLS certificate and client authentication settings for HTTPS connections to Spark targets.

* **`tls.ca_file`**
  Type: `string`. Optional.
  Path to the CA cert. For a client this verifies the server certificate. For a server this
  verifies client certificates. If empty uses system root CA. (optional)

* **`tls.ca_pem`**
  Type: `string`. Optional.
  In memory PEM encoded cert. (optional)

* **`tls.cert_file`**
  Type: `string`. Optional.
  Path to the TLS cert to use for TLS required connections. (optional)

* **`tls.cert_pem`**
  Type: `string`. Optional.
  In memory PEM encoded TLS cert to use for TLS required connections. (optional)

* **`tls.cipher_suites`**
  Type: `array of string`. Optional.
  CipherSuites is a list of TLS cipher suites that the TLS transport can use. If left blank, a
  safe default list is used. See
  [https://go.dev/src/crypto/tls/cipher\_suites.go](https://go.dev/src/crypto/tls/cipher_suites.go)
  for a list of supported cipher suites.

* **`tls.curve_preferences`**
  Type: `array of string`. Optional.
  contains the elliptic curves that will be used in an ECDHE handshake, in preference order
  Defaults to empty list and "crypto/tls" defaults are used, internally.

* **`tls.include_system_ca_certs_pool`**
  Type: `boolean`. Optional.
  If true, load system CA certificates pool in addition to the certificates configured in this
  struct.

* **`tls.insecure`**
  Type: `boolean`. Optional.
  In gRPC and HTTP when set to true, this is used to disable the client transport security. See
  [https://godoc.org/google.golang.org/grpc#WithInsecure](https://godoc.org/google.golang.org/grpc#WithInsecure)
  for gRPC. Please refer to
  [https://godoc.org/crypto/tls#Config](https://godoc.org/crypto/tls#Config) for more
  information. (optional, default false)

* **`tls.insecure_skip_verify`**
  Type: `boolean`. Optional.
  InsecureSkipVerify will enable TLS but not verify the certificate.

* **`tls.key_file`**
  Type: `string`. Optional.
  Path to the TLS key to use for TLS required connections. (optional)

* **`tls.key_pem`**
  Type: `string`. Optional.
  In memory PEM encoded TLS key to use for TLS required connections. (optional)

* **`tls.max_version`**
  Type: `string`. Optional.
  MaxVersion sets the maximum TLS version that is acceptable. If not set, refer to crypto/tls
  for defaults. (optional)

* **`tls.min_version`**
  Type: `string`. Optional.
  MinVersion sets the minimum TLS version that is acceptable. If not set, TLS 1.2 will be used.
  (optional)

* **`tls.reload_interval`**
  Type: `duration`. Optional.
  ReloadInterval specifies the duration after which the certificate will be reloaded If not set,
  it will never be reloaded (optional)

* **`tls.server_name_override`**
  Type: `string`. Optional.
  ServerName requested by client for virtual hosting. This sets the ServerName in the TLSConfig.
  Please refer to [https://godoc.org/crypto/tls#Config](https://godoc.org/crypto/tls#Config) for
  more information. (optional)

* **`tls.tpm`**
  Type: `object`. Optional.
  Trusted platform module configuration

* **`tls.tpm.auth`**
  Type: `string`. Optional.
  Authorization value for the trusted platform module key.

* **`tls.tpm.enabled`**
  Type: `boolean`. Optional.
  Whether to use a trusted platform module for the TLS private key. If true, the Collector loads
  the key from the configured device or socket. If false, the Collector uses the configured key
  file or in-memory key.

* **`tls.tpm.owner_auth`**
  Type: `string`. Optional.
  Owner authorization value for the trusted platform module.

* **`tls.tpm.path`**
  Type: `string`. Optional.
  The path to the TPM device or Unix domain socket. For instance /dev/tpm0 or /dev/tpmrm0.


## Related topics

- [CXDOT Collector integrations](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector.md)
- [Apache](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/apache.md)
- [Apache Mesos](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/mesos.md)
- [Apache Tomcat](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/tomcat.md)
- [Apache Cassandra](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/cassandra.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.