Supported telemetry types
The Apache Spark CXDOT Collector integration supports these telemetry types:Prerequisites
The Apache Spark CXDOT Collector integration has the following prerequisites:- Make the driver web interface for each Spark app reachable from the CXDOT Collector. Point targets at app driver web interfaces, not Spark standalone masters, Apache Hadoop Yet Another Resource Negotiator (YARN) ResourceManagers, or Apache Mesos masters.
- To collect Structured Streaming query metrics, set
spark.sql.streaming.metricsEnabledtotruefor the app. Spark then reports each query’s metrics through the metrics endpoint that the driver web interface serves at/metrics/jsonby default. If your apps serve that endpoint at another path, setmetrics_servlet_pathto match. Apps that use Spark Streaming report their streaming statistics without any setting. To stop collecting both kinds of streaming metrics, setstreaming_metricstofalse. - To collect the peak memory of each driver’s and executor’s process tree, set
spark.executor.processTreeMetrics.enabledtotruefor the app.
Configure
The Apache Spark integration is enabled by default. You can configure targets either by annotating driver pods in your Kubernetes cluster or by listing static endpoints in your Helm values. To configure the integration, follow these steps:-
To collect from Spark driver pods in Kubernetes, add a CXDOT integration annotation
to the pod template. The annotation key must end with the container name. For
example, add the following annotation to your Spark driver:
The Collector replaces
${target.host}with the pod IP address and${target.port.ui}with the container port namedui. The Collector also discovers Spark driver pods labeledspark-role=driverthat expose port4040. A CXDOT integration annotation takes precedence over the label for its container. To stop collecting from a container, setspark: {enabled: false}in the annotation. Metrics from each discovered driver pod include that pod’s Kubernetes metadata. For more information, see autodiscovery. -
Optional: To collect from Spark apps outside your Kubernetes cluster, configure
static driver targets. A non-empty
endpointslist turns off discovery for that integration instance, and the Collector collects only the listed targets. To run discovery and static targets together, add a named integration instance. For example, add the following to thevalues.yamlfor the CXDOT Collector Helm chart:The emptysparkentry keeps discovery active.spark/externalis an independent, named instance that collects from the listed driver.
enabled to false. For
example, add the following to the values.yaml for the CXDOT Collector Helm chart:
Validate
To validate the Apache Spark CXDOT Collector integration, follow these steps:-
In Live Telemetry Analyzer,
filter for
__name__=cxdot.integration.target.healthandcxdot.integration.name=spark. Confirm the metric reports1for each Spark target. Thecxdot.integration.checklabel has the valuespark, andcxdot.integration.targetidentifies the driver. -
In Metrics Explorer,
run the following query:
Confirm the query returns a value for each active Spark app.
Troubleshooting
- No app metrics appear for a standalone master, YARN ResourceManager, or Apache Mesos master: Configure each app’s driver web interface as a target. The integration doesn’t use cluster managers to discover apps.
Configuration reference
Configure one Apache Spark integration instance with the following settings. In Helm values, place these settings undercollection.cxdot.integrations.spark. In a Collector configuration
file, place them under cxdot.integrations.spark.
Optional settings
-
enabledType:boolean. Optional. Default:true. Whether to enable this Spark integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it. -
endpointsType:array of object. Optional. Default:[]. Spark app driver web interface targets. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains targets, this integration instance collects from exactly those targets and doesn’t automatically discover targets. Configure a second named instance to use both methods. Specify each target ashost:portto use HTTP, or as a full HTTP or HTTPS URL. The URL must identify the web interface root and must not include a Spark REST API path. Configure app driver web interfaces only. The integration doesn’t discover apps through cluster managers. -
endpoints[].endpointType:string. Required. Spark app driver web interface target ashost:portor a full HTTP or HTTPS URL. Forhost:port, the Collector uses HTTP. -
endpoints[].cluster_nameType:string. Optional. Name of the Spark cluster the driver belongs to, reported as thespark.cluster.nameattribute. When omitted, the attribute is not set. -
application_namesType:array of string. Optional. Default:[]. Spark app names to collect from each target. When the list is empty, the Collector collects every app reported by the target. -
metrics_servlet_pathType:string. Optional. Default:/metrics/json. Path of the JSON metrics endpoint on each target’s driver web interface, where Spark publishes Structured Streaming query metrics. Change it only for apps that move the endpoint with thespark.metrics.conf.*.sink.servlet.pathSpark setting. When a target answers this path with HTTP 404, the Collector collects no Structured Streaming metrics from it. -
streaming_metricsType:boolean. Optional. Default:true. Whether to collect Spark Streaming and Structured Streaming metrics. Whenfalse, the Collector doesn’t request either streaming endpoint, so a target whose JSON metrics endpoint returns errors no longer reports failed collections. -
collection_intervalType:duration. Optional. Default:30s. How often the Collector collects metrics from each Spark target. -
timeoutType:duration. Optional. Default:30s. Maximum time allowed to collect metrics from one Spark target. The value must not exceedcollection_interval. -
tlsType:object. Optional. TLS certificate and client authentication settings for HTTPS connections to Spark targets. -
tls.ca_fileType:string. Optional. Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty uses system root CA. (optional) -
tls.ca_pemType:string. Optional. In memory PEM encoded cert. (optional) -
tls.cert_fileType:string. Optional. Path to the TLS cert to use for TLS required connections. (optional) -
tls.cert_pemType:string. Optional. In memory PEM encoded TLS cert to use for TLS required connections. (optional) -
tls.cipher_suitesType:array of string. Optional. CipherSuites is a list of TLS cipher suites that the TLS transport can use. If left blank, a safe default list is used. See https://go.dev/src/crypto/tls/cipher_suites.go for a list of supported cipher suites. -
tls.curve_preferencesType:array of string. Optional. contains the elliptic curves that will be used in an ECDHE handshake, in preference order Defaults to empty list and “crypto/tls” defaults are used, internally. -
tls.include_system_ca_certs_poolType:boolean. Optional. If true, load system CA certificates pool in addition to the certificates configured in this struct. -
tls.insecureType:boolean. Optional. In gRPC and HTTP when set to true, this is used to disable the client transport security. See https://godoc.org/google.golang.org/grpc#WithInsecure for gRPC. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional, default false) -
tls.insecure_skip_verifyType:boolean. Optional. InsecureSkipVerify will enable TLS but not verify the certificate. -
tls.key_fileType:string. Optional. Path to the TLS key to use for TLS required connections. (optional) -
tls.key_pemType:string. Optional. In memory PEM encoded TLS key to use for TLS required connections. (optional) -
tls.max_versionType:string. Optional. MaxVersion sets the maximum TLS version that is acceptable. If not set, refer to crypto/tls for defaults. (optional) -
tls.min_versionType:string. Optional. MinVersion sets the minimum TLS version that is acceptable. If not set, TLS 1.2 will be used. (optional) -
tls.reload_intervalType:duration. Optional. ReloadInterval specifies the duration after which the certificate will be reloaded If not set, it will never be reloaded (optional) -
tls.server_name_overrideType:string. Optional. ServerName requested by client for virtual hosting. This sets the ServerName in the TLSConfig. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional) -
tls.tpmType:object. Optional. Trusted platform module configuration -
tls.tpm.authType:string. Optional. Authorization value for the trusted platform module key. -
tls.tpm.enabledType:boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key. -
tls.tpm.owner_authType:string. Optional. Owner authorization value for the trusted platform module. -
tls.tpm.pathType:string. Optional. The path to the TPM device or Unix domain socket. For instance /dev/tpm0 or /dev/tpmrm0.