Skip to main content
The Apache Spark integration requires CXDOT Collector 1.4.0 or greater. Apache Spark is a unified engine for large-scale data analytics. Use the Apache Spark CXDOT Collector integration to collect app, job, stage, executor, storage, and streaming metrics from Spark drivers running in your environment. The Apache Spark CXDOT Collector integration supports Apache Spark 3.4.0 or greater.

Supported telemetry types

The Apache Spark CXDOT Collector integration supports these telemetry types:

Prerequisites

The Apache Spark CXDOT Collector integration has the following prerequisites:
  • Make the driver web interface for each Spark app reachable from the CXDOT Collector. Point targets at app driver web interfaces, not Spark standalone masters, Apache Hadoop Yet Another Resource Negotiator (YARN) ResourceManagers, or Apache Mesos masters.
  • To collect Structured Streaming query metrics, set spark.sql.streaming.metricsEnabled to true for the app. Spark then reports each query’s metrics through the metrics endpoint that the driver web interface serves at /metrics/json by default. If your apps serve that endpoint at another path, set metrics_servlet_path to match. Apps that use Spark Streaming report their streaming statistics without any setting. To stop collecting both kinds of streaming metrics, set streaming_metrics to false.
  • To collect the peak memory of each driver’s and executor’s process tree, set spark.executor.processTreeMetrics.enabled to true for the app.

Configure

The Apache Spark integration is enabled by default. You can configure targets either by annotating driver pods in your Kubernetes cluster or by listing static endpoints in your Helm values. To configure the integration, follow these steps:
  1. To collect from Spark driver pods in Kubernetes, add a CXDOT integration annotation to the pod template. The annotation key must end with the container name. For example, add the following annotation to your Spark driver:
    The Collector replaces ${target.host} with the pod IP address and ${target.port.ui} with the container port named ui. The Collector also discovers Spark driver pods labeled spark-role=driver that expose port 4040. A CXDOT integration annotation takes precedence over the label for its container. To stop collecting from a container, set spark: {enabled: false} in the annotation. Metrics from each discovered driver pod include that pod’s Kubernetes metadata. For more information, see autodiscovery.
  2. Optional: To collect from Spark apps outside your Kubernetes cluster, configure static driver targets. A non-empty endpoints list turns off discovery for that integration instance, and the Collector collects only the listed targets. To run discovery and static targets together, add a named integration instance. For example, add the following to the values.yaml for the CXDOT Collector Helm chart:
    The empty spark entry keeps discovery active. spark/external is an independent, named instance that collects from the listed driver.
To disable the default Spark integration instance, set enabled to false. For example, add the following to the values.yaml for the CXDOT Collector Helm chart:

Validate

To validate the Apache Spark CXDOT Collector integration, follow these steps:
  1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health and cxdot.integration.name=spark. Confirm the metric reports 1 for each Spark target. The cxdot.integration.check label has the value spark, and cxdot.integration.target identifies the driver.
  2. In Metrics Explorer, run the following query:
    Confirm the query returns a value for each active Spark app.

Troubleshooting

  • No app metrics appear for a standalone master, YARN ResourceManager, or Apache Mesos master: Configure each app’s driver web interface as a target. The integration doesn’t use cluster managers to discover apps.

Configuration reference

Configure one Apache Spark integration instance with the following settings. In Helm values, place these settings under collection.cxdot.integrations.spark. In a Collector configuration file, place them under cxdot.integrations.spark.

Optional settings

  • enabled Type: boolean. Optional. Default: true. Whether to enable this Spark integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it.
  • endpoints Type: array of object. Optional. Default: []. Spark app driver web interface targets. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains targets, this integration instance collects from exactly those targets and doesn’t automatically discover targets. Configure a second named instance to use both methods. Specify each target as host:port to use HTTP, or as a full HTTP or HTTPS URL. The URL must identify the web interface root and must not include a Spark REST API path. Configure app driver web interfaces only. The integration doesn’t discover apps through cluster managers.
  • endpoints[].endpoint Type: string. Required. Spark app driver web interface target as host:port or a full HTTP or HTTPS URL. For host:port, the Collector uses HTTP.
  • endpoints[].cluster_name Type: string. Optional. Name of the Spark cluster the driver belongs to, reported as the spark.cluster.name attribute. When omitted, the attribute is not set.
  • application_names Type: array of string. Optional. Default: []. Spark app names to collect from each target. When the list is empty, the Collector collects every app reported by the target.
  • metrics_servlet_path Type: string. Optional. Default: /metrics/json. Path of the JSON metrics endpoint on each target’s driver web interface, where Spark publishes Structured Streaming query metrics. Change it only for apps that move the endpoint with the spark.metrics.conf.*.sink.servlet.path Spark setting. When a target answers this path with HTTP 404, the Collector collects no Structured Streaming metrics from it.
  • streaming_metrics Type: boolean. Optional. Default: true. Whether to collect Spark Streaming and Structured Streaming metrics. When false, the Collector doesn’t request either streaming endpoint, so a target whose JSON metrics endpoint returns errors no longer reports failed collections.
  • collection_interval Type: duration. Optional. Default: 30s. How often the Collector collects metrics from each Spark target.
  • timeout Type: duration. Optional. Default: 30s. Maximum time allowed to collect metrics from one Spark target. The value must not exceed collection_interval.
  • tls Type: object. Optional. TLS certificate and client authentication settings for HTTPS connections to Spark targets.
  • tls.ca_file Type: string. Optional. Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty uses system root CA. (optional)
  • tls.ca_pem Type: string. Optional. In memory PEM encoded cert. (optional)
  • tls.cert_file Type: string. Optional. Path to the TLS cert to use for TLS required connections. (optional)
  • tls.cert_pem Type: string. Optional. In memory PEM encoded TLS cert to use for TLS required connections. (optional)
  • tls.cipher_suites Type: array of string. Optional. CipherSuites is a list of TLS cipher suites that the TLS transport can use. If left blank, a safe default list is used. See https://go.dev/src/crypto/tls/cipher_suites.go for a list of supported cipher suites.
  • tls.curve_preferences Type: array of string. Optional. contains the elliptic curves that will be used in an ECDHE handshake, in preference order Defaults to empty list and “crypto/tls” defaults are used, internally.
  • tls.include_system_ca_certs_pool Type: boolean. Optional. If true, load system CA certificates pool in addition to the certificates configured in this struct.
  • tls.insecure Type: boolean. Optional. In gRPC and HTTP when set to true, this is used to disable the client transport security. See https://godoc.org/google.golang.org/grpc#WithInsecure for gRPC. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional, default false)
  • tls.insecure_skip_verify Type: boolean. Optional. InsecureSkipVerify will enable TLS but not verify the certificate.
  • tls.key_file Type: string. Optional. Path to the TLS key to use for TLS required connections. (optional)
  • tls.key_pem Type: string. Optional. In memory PEM encoded TLS key to use for TLS required connections. (optional)
  • tls.max_version Type: string. Optional. MaxVersion sets the maximum TLS version that is acceptable. If not set, refer to crypto/tls for defaults. (optional)
  • tls.min_version Type: string. Optional. MinVersion sets the minimum TLS version that is acceptable. If not set, TLS 1.2 will be used. (optional)
  • tls.reload_interval Type: duration. Optional. ReloadInterval specifies the duration after which the certificate will be reloaded If not set, it will never be reloaded (optional)
  • tls.server_name_override Type: string. Optional. ServerName requested by client for virtual hosting. This sets the ServerName in the TLSConfig. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional)
  • tls.tpm Type: object. Optional. Trusted platform module configuration
  • tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
  • tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
  • tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
  • tls.tpm.path Type: string. Optional. The path to the TPM device or Unix domain socket. For instance /dev/tpm0 or /dev/tpmrm0.