Supported telemetry types
The Hadoop YARN integration supports these telemetry types:Prerequisites
The Hadoop YARN integration has the following prerequisites:-
Make the ResourceManager’s web interface, which listens on port
8088by default, reachable from the CXDOT Collector. The integration reads the ResourceManager’s REST API, and the web interface can serve it over HTTP or HTTPS. To confirm that the CXDOT Collector can read it, run the following command from a pod or host on the same network as the CXDOT Collector, replacingRESOURCEMANAGER_ADDRESSwith the ResourceManager’s address. The command returns the HTTP status code200. On a cluster with ResourceManager high availability, a standby ResourceManager returns307instead, which is expected: - Check how the ResourceManager’s web interface authenticates requests. The integration supports unauthenticated access, HTTP basic authentication, custom request headers such as a bearer token for an authenticating proxy, and Transport Layer Security (TLS). The integration doesn’t support Kerberos (SPNEGO) authentication, so a web interface that requires Kerberos returns an error for every request.
-
On a cluster with ResourceManager high availability, choose how to target the
ResourceManagers. A standby ResourceManager redirects every request to the active one
on another host. Use one of the following:
- Annotate the ResourceManager pods, or list each ResourceManager as a static target. The integration treats a target that redirects to another host as a standby. The active ResourceManager reports the cluster’s metrics, and each standby reports as healthy and reports no other metrics, so the cluster’s metrics aren’t duplicated.
- Annotate a Kubernetes Service that selects the ResourceManager pods. The integration follows the standby’s redirect for a target it discovers from a Service, so the target reports the cluster’s metrics whichever ResourceManager answers.
- List one address that reaches both ResourceManagers, such as a virtual IP address or
a load balancer, as a static target, and set
follow_standby_redirectstotrue. Without that setting, the target reports as healthy and reports no metrics whenever one of its requests reaches the standby. You don’t need the setting if the address routes requests only to the active ResourceManager. For example, a load balancer can run a health check on the ResourceManager’s/isActivepath, which returns the HTTP status code200only on the active ResourceManager.
headersvalues to the active ResourceManager but not theusernameandpasswordcredentials.
Configure
The Hadoop YARN integration is enabled by default, but it collects only from the ResourceManagers you give it. You can configure targets either by annotating ResourceManager pods in your Kubernetes cluster or by listing static endpoints in your Helm values. To configure the integration, follow these steps:-
To collect from a ResourceManager running in Kubernetes, add a CXDOT integration
annotation to the pod template. The annotation key must end with the name of the
ResourceManager’s container, which is
resourcemanagerin the following example. Theyarn:block in the annotation’s value chooses the Hadoop YARN integration. For example, add the following annotation to your ResourceManager workload:The Collector replaces${target.host}with the pod IP address and${target.port.ui}with the container port namedui, and it reads the ResourceManager over HTTP. Thecluster_namevalue becomes theyarn.cluster.nameattribute on the metrics, which tells clusters apart because a ResourceManager doesn’t report its cluster’s name. On a cluster with ResourceManager high availability, an annotation on the pod template reaches every ResourceManager. The active ResourceManager reports the cluster’s metrics. Each standby ResourceManager reportscxdot.integration.target.healthas1and no other metrics. To collect through a Service that selects the ResourceManager pods instead, add the same annotation value to the Service under theintegrations.cxdot.paloaltonetworks.com/servicekey. The Collector then replaces${target.host}with the Service’s cluster IP address and${target.port.ui}with the Service port namedui, and it follows a standby ResourceManager’s redirect to the active one. -
Optional: To collect from ResourceManagers outside your Kubernetes cluster, configure
static targets. For example, add the following to the Helm values file for your CXDOT
Collector:
Write a target as
host:portto read it over HTTP, or as a full HTTP or HTTPS URL. When you configure static targets, this integration instance collects from exactly those targets instead of discovered targets. In cluster mode, the cluster Collector that wins leader election scrapes static targets. -
Optional: Configure a second integration instance to collect from both annotated
pods and static targets. For example, add the following to the Helm values file for
your CXDOT Collector:
The
yarn/externalkey defines a named instance for the static targets. The bareyarnkey continues to collect from annotated pods. -
Optional: If the ResourceManager’s web interface requires HTTP basic authentication
or HTTPS, configure the matching settings for static targets, and provide the password
through your deployment’s secret management. For example, the following Helm values
reference the
YARN_PASSWORDenvironment variable and a mounted certificate authority file:Theusername,password, andtlssettings in your Helm values apply to the targets you list there. Setusernameandpasswordtogether. You can combine them withheaders. Ifheadersalso sets anAuthorizationheader, the integration sends the basic credentials in its place. The integration follows an HTTP redirect only when the redirect stays on the target’s host, for example from a proxy to another path on the proxy. To make such a request fail instead, setfollow_redirectstofalse. The integration follows a redirect to another host only for a target it discovers from a Service, or whenfollow_standby_redirectsistrue, as described in Prerequisites. Otherwise, it sends the credentials and theheadersvalues only to the host you configured. Set the password variable on both the node and cluster Collectors with thenodeCollector.extraEnvandclusterCollector.extraEnvHelm values. Mount the certificate authority file into the cluster Collector, which collects from static targets, with theclusterCollector.extraVolumesandclusterCollector.extraVolumeMountsHelm values. For annotated pods, set these settings in the CXDOT integration annotation instead. Reference the password withpassword: ${k8s_secret:yarn-credentials/password}and supply the certificate authority from a Kubernetes Secret withtls.ca_pem: ${k8s_secret:yarn-credentials/ca.crt}. To allow the Collector to read the Secret, add the pod’s namespace toapiServer.metadataServer.secretAllowlistin the Helm values for your CXDOT Collector. Use a URL that starts withhttps://in the annotation’sendpointvalue. Because${target.host}is the pod IP address, settls.server_name_overrideto a name that the ResourceManager’s certificate carries, unless the certificate carries the pod IP address. -
Optional: To limit the series the application, node, and queue metrics create, adjust
which applications, nodes, and queues report. For example, add the following to the Helm
values file for your CXDOT Collector:
These settings in your Helm values apply to the static targets of the same instance. For annotated pods, set
application_states,application_id,max_applications,node_metrics,max_nodes,inactive_node_retention,queue_metrics,exclude_queues, andmax_queuesin the CXDOT integration annotation instead. The integration reportsyarn.application.statusfor every application that hasn’t ended, and for every failed or killed application that the ResourceManager still lists. The status is2, which means critical, when the application’s state or its final status isFAILEDorKILLED. An application whose application master reports a failure and then exits cleanly, such as a Spark application whose driver fails, ends in theFINISHEDstate with the final statusFAILED. The integration also reports a critical status for those applications, so an alert on a critical status covers failed and killed applications without changing a setting. Theapplication_statessetting chooses which applications also report resource and progress metrics, and defaults toRUNNING. Other finished applications report a status only if you addFINISHEDtoapplication_states. That includes a finished application whose final status isKILLED, which then reports a critical status. Theyarn.cluster.application.terminationsmetric counts ended applications by final state without this setting. By default, everyyarn.application.*series carries the application’s ID in theyarn.application.idattribute, so each run of an application creates a new set of series. On a cluster that runs many short-lived applications, setapplication_idtofalseto leave the ID off. Applications that share a name, queue, state, and user then report as one, with their memory, virtual cores, and containers summed. The integration also stops reportingyarn.application.memory.consumedandyarn.application.vcore.consumed. Themax_applicationslimit, which defaults to2000, caps how many applications report, or how many groups of applications whenapplication_idisfalse. Applications that haven’t ended report first, starting with the one that has run longest, so an application that reports keeps reporting while newer ones arrive. Ended applications follow, starting with the one that ended most recently. If the applications that haven’t ended fill the limit, failed and killed applications report noyarn.application.status. A ResourceManager keeps listing a node that has left the cluster until the ResourceManager restarts. A NodeManager that listens on an ephemeral port, which is the default, registers as a new node each time it starts, so the listed nodes grow with every NodeManager restart. The integration reportsyarn.node.*metrics for a node whose state isLOST,SHUTDOWN,DECOMMISSIONED, orREBOOTEDonly while the node’s last health report is newer thaninactive_node_retention, which defaults to4h. Themax_nodeslimit, which defaults to1000, caps how many nodes report, and nodes that are part of the cluster report first. Theyarn.cluster.node.countmetric counts every node by state regardless of both settings. The integration fills themax_queueslimit, which defaults to100, from the queue nearest the root outward, one level at a time, and each level follows the order that the scheduler lists its queues in. On a scheduler with more queues, the deepest queues stop reporting first. Useexclude_queuesto choose which queues report, by name or by full path. A queue you exclude also excludes its child queues. On a cluster that uses the Fair scheduler or the first in, first out (FIFO) scheduler, which report no queue metrics, setqueue_metricstofalseso the integration doesn’t request scheduler information. -
Optional: Disable the integration. For example, add the following to the Helm values
file for your CXDOT Collector:
Validate
To validate the Hadoop YARN integration, follow these steps:-
In Live Telemetry Analyzer,
filter for
__name__=cxdot.integration.target.healthandcxdot.integration.name=yarn. Confirm thatcxdot.integration.target.healthreports1for each ResourceManager, including each standby ResourceManager. A value of0means that at least one request from the CXDOT Collector to that ResourceManager failed. Thecxdot.integration.targetattribute identifies the target. -
In Metrics Explorer,
run the following query:
Confirm that the query returns a time series for each node state of each active ResourceManager, and that the
runningseries reports the number of NodeManagers that are running. The integration reports memory in MiB and times in milliseconds, as the ResourceManager reports them.
Troubleshooting
cxdot.integration.target.healthreads0, but some metrics still report: The CXDOT Collector makes up to five requests to the ResourceManager, for cluster, application, node, and queue data. A request that fails or times out costs its own metrics, and the others still report. Check which metric prefix is missing:yarn.cluster.*,yarn.application.*,yarn.node.*, oryarn.queue.*.- Every request fails for a secured cluster: The integration doesn’t support Kerberos
(SPNEGO) authentication. Reach a ResourceManager that requires it through an address
that accepts basic authentication or a header-based proxy, and set
usernameandpasswordorheadersto match. - No
yarn.queue.*metrics appear: The integration reads queues only from the CapacityScheduler. With the Fair scheduler or the first in, first out scheduler, the integration still reports theyarn.cluster.*,yarn.application.*, andyarn.node.*metrics. Thequeue_metricssetting also turns these metrics off. - Some queues report no metrics: The scheduler has more queues than
max_queues, and the Collector logs a warning that says so. Raisemax_queuesor useexclude_queues. - Some applications report no metrics, or failed and killed applications report no
yarn.application.status: The ResourceManager lists more applications thanmax_applications, and the Collector logs a warning that says so. Applications that haven’t ended fill the limit first. Raisemax_applications, or setapplication_idtofalseso that applications that share a name, queue, state, and user count once. cxdot.integration.target.healthreads1, but noyarn.*metrics report: The target redirects to another host, so the integration treats it as a standby ResourceManager, and the Collector logs a message that names the host. For a standby ResourceManager, the active ResourceManager’s target reports the metrics. For a target that’s one address for both ResourceManagers, such as a virtual IP address or a load balancer, a collection reports no metrics whenever one of its requests reaches the standby. Setfollow_standby_redirectstotruefor that target, or route its requests only to the active ResourceManager, as described in Prerequisites. For any other target, such as a proxy that redirects to another host name, target the address it redirects to.- Every request fails with
without a usable Location: A standby ResourceManager answers that way while no ResourceManager is active. Check the state of the ResourceManagers. - Some nodes report no
yarn.node.*metrics: The ResourceManager lists more nodes thanmax_nodes, and the Collector logs a warning that says so. Nodes that are part of the cluster fill the limit first. Raisemax_nodes. To stop collecting theyarn.node.*metrics, setnode_metricstofalse. Theyarn.cluster.*metrics still report the cluster’s totals and its node counts by state. - A lost, shut down, decommissioned, or rebooted node stops reporting
yarn.node.*metrics: The node’s last health report is older thaninactive_node_retention. A NodeManager reports its health when its health checks run. If the NodeManager runs neither the disk health check nor a health script, its last health report is the time it started, so a node that ran longer thaninactive_node_retentionstops reporting as soon as it leaves the cluster. Raiseinactive_node_retentionto keep such nodes reporting longer. Theyarn.cluster.node.countmetric still counts the node. - A redirected request fails with
redirect not followed: The target redirects to another address on the same host, andfollow_redirectsisfalseor your deployment denies redirects for annotated pods. Target the address the ResourceManager redirects to. yarn.application.memory.consumedandyarn.application.vcore.consumedare missing: The integration doesn’t report them whenapplication_idisfalse.- Metrics for a Spark application’s own executors and jobs don’t appear: The integration reads only the ResourceManager. The Spark integration collects those metrics from the Spark driver.
Configuration reference
Configure one Hadoop YARN integration instance with the following settings. In Helm values, place these settings undercollection.cxdot.integrations.yarn. In a Collector configuration
file, place them under cxdot.integrations.yarn.
Optional settings
-
enabledType:boolean. Optional. Default:true. Whether to enable this Hadoop YARN integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it. -
endpointsType:array of object. Optional. Default:[]. ResourceManager targets. Specify a target ashost:portto use HTTP, or as a full HTTP or HTTPS URL of the ResourceManager’s web interface, without a REST API path. On a cluster with ResourceManager high availability, list each ResourceManager. A standby ResourceManager redirects to the active one, and the Collector treats a target that redirects to another host as a standby: the target reports as healthy and reports no metrics. If you list one address that reaches both ResourceManagers instead, such as a virtual IP address or a load balancer, setfollow_standby_redirectstotrue, unless the address routes requests only to the active ResourceManager. -
endpoints[].endpointType:string. Required.host:portof the ResourceManager’s web interface, or its HTTP or HTTPS URL. -
endpoints[].cluster_nameType:string. Optional. Name of the YARN cluster the ResourceManager belongs to, reported as theyarn.cluster.nameattribute. When omitted, the attribute isn’t set. -
application_metricsType:boolean. Optional. Default:true. Whether to collect theyarn.application.*metrics. Whenfalse, the Collector doesn’t request the ResourceManager’s application list. -
application_idType:boolean. Optional. Default:true. Whether theyarn.application.*metrics carry the application’s ID in theyarn.application.idattribute. Whentrue, each run of an application is its own set of series. Whenfalse, the Collector leaves the ID off, and applications that share a name, queue, state, and user report as one. Their memory, virtual cores, and containers are summed. Progress is the lowest, start time the earliest, and finish time and elapsed time the latest and longest. The Collector then doesn’t reportyarn.application.memory.consumedandyarn.application.vcore.consumed, because a total summed over applications that come and go would fall whenever one ends. Set this tofalseon a cluster that runs many short-lived applications. -
max_applicationsType:integer. Optional. Default:2000. Minimum:1. Cap on how many applications reportyarn.application.*metrics, or, whenapplication_idisfalse, on how many groups of applications that report as one. Applications that haven’t ended report first, starting with the one that has run longest, so an application that reports keeps reporting while newer ones arrive. Ended applications follow, starting with the one that ended most recently. On a ResourceManager with more applications than the cap, the rest report no metrics and the Collector logs a warning. If the applications that haven’t ended fill the cap, failed and killed applications report noyarn.application.status. -
application_statesType:array of string. Optional. Default:["RUNNING"]. Application states to collect resource and progress metrics for. This setting doesn’t limityarn.application.status, which the Collector reports for every application that hasn’t ended (NEW,NEW_SAVING,SUBMITTED,ACCEPTED, andRUNNING) and for every failed or killed application the ResourceManager still lists. The exception is finished applications, which make up most of the ended applications a ResourceManager retains. Unless you listFINISHEDhere, the Collector asks the ResourceManager only for the finished applications whose final status isFAILED, such as a Spark application whose driver reported a failure, and reports their status as critical. When you listFINISHED, every finished application reports a status, which is critical if its final status isFAILEDorKILLED. The Collector reports a finished application whose final status isKILLEDonly when you listFINISHED. Theyarn.cluster.application.terminationsmetric counts ended applications by final state. -
node_metricsType:boolean. Optional. Default:true. Whether to collect theyarn.node.*metrics. Whenfalse, the Collector doesn’t request the ResourceManager’s node list. -
max_nodesType:integer. Optional. Default:1000. Minimum:1. Cap on how many nodes reportyarn.node.*metrics. Nodes that are part of the cluster report first, in order of node ID. Nodes that have left the cluster and are still withininactive_node_retentionfollow, starting with the one that reported its health most recently. On a ResourceManager with more nodes than the cap, the rest report noyarn.node.*metrics and the Collector logs a warning. Theyarn.cluster.node.countmetric counts every node regardless of this setting. -
inactive_node_retentionType:duration. Optional. Default:4h. How long a node that has left the cluster keeps reportingyarn.node.*metrics, measured from its last health report. A node has left the cluster when its state isLOST,SHUTDOWN,DECOMMISSIONED, orREBOOTED. A ResourceManager keeps listing such a node until the ResourceManager restarts. A NodeManager that listens on an ephemeral port, which is the default, registers as a new node each time it starts, so without this limit the number of nodes that report grows with every NodeManager restart. Set this to0sto report no nodes that have left the cluster. Theyarn.cluster.node.countmetric counts every node regardless of this setting. -
queue_metricsType:boolean. Optional. Default:true. Whether to collect theyarn.queue.*metrics. Whenfalse, the Collector doesn’t request the ResourceManager’s scheduler information, and theyarn.queue.pathattribute of theyarn.application.*metrics is the queue as the ResourceManager reports it for the application. Set this tofalseon a cluster that uses the Fair scheduler or the first in, first out scheduler, which report no queue metrics. -
exclude_queuesType:array of string. Optional. Default:[]. CapacityScheduler queues to leave out, by name (adhoc) or by full path (root.analytics.adhoc). Leaving a queue out also leaves out its child queues. -
max_queuesType:integer. Optional. Default:100. Minimum:1. Cap on how many queues, not counting the root, report metrics, filled from the queue nearest the root outward, one level at a time, with each level in the order the scheduler lists its queues. On a scheduler with more queues, useexclude_queuesto choose which ones report. -
usernameType:string. Optional. Username for HTTP basic authentication to the targets inendpoints. Setusernameandpasswordtogether. The Collector sends the credentials in anAuthorizationheader on every request, in place of anAuthorizationheader set inheaders. Kerberos (SPNEGO) authentication isn’t supported. -
passwordType:string. Optional. Password for HTTP basic authentication to the targets inendpoints. Setusernameandpasswordtogether. The Collector masks this value in diagnostic output, logs, and errors. Kerberos (SPNEGO) authentication isn’t supported. -
headersType:object. Optional. Additional request headers as name-value pairs, sent to every target inendpoints, for example a bearer token inAuthorizationfor a ResourceManager behind an authenticating proxy. You can combineheaderswithusernameandpassword. If you do, the Collector doesn’t send anAuthorizationheader set here, and sends the basic credentials instead. The Collector sends these headers only to the target’s own host, unlessfollow_standby_redirectsistrue. In that case, the Collector also sends them to the active ResourceManager that a standby redirects to. The Collector masks the values in diagnostic output, logs, and errors. This setting doesn’t enable Kerberos (SPNEGO) authentication, which isn’t supported. -
follow_redirectsType:boolean. Optional. Default:true. Whether to follow HTTP redirects that stay on the target’s host, for example from a proxy to another path on the proxy. If false, such a redirect causes the metric collection to fail. Redirects to another host are controlled byfollow_standby_redirects, and the Collector follows them only if both settings are true. -
follow_standby_redirectsType:boolean. Optional. Default:false. Whether to follow a standby ResourceManager’s redirect to the active ResourceManager. A standby ResourceManager answers every request with a redirect to the active one on another host. If false, the Collector doesn’t follow that redirect and treats the target as a standby: the target reports as healthy and reports no metrics. Use false when each target is a single ResourceManager, so that only the active one reports the cluster’s metrics. If true, the Collector follows the redirect, and the target reports the cluster’s metrics whichever ResourceManager answers. Use true when a target is one address that reaches both ResourceManagers, such as a virtual IP address or a load balancer that doesn’t route only to the active ResourceManager. With false, such a target reports no metrics whenever one of its requests reaches the standby. The Collector always follows the redirect for a target discovered from a Kubernetes Service, which is one address for every ResourceManager behind it. A redirect to another host is followed only iffollow_redirectsis also true. The Collector doesn’t send theusernameandpasswordcredentials to the active ResourceManager that a standby redirects to, so if the active ResourceManager requires authentication, authenticate withheaders. -
collection_intervalType:duration. Optional. Default:15s. How often the Collector collects metrics from each ResourceManager. -
timeoutType:duration. Optional. Default:15s. Maximum time allowed for one collection from a ResourceManager. The limit applies to each request the Collector makes and to the collection as a whole. The value must not exceedcollection_interval. -
tlsType:object. Optional. Transport layer security (TLS) settings for the HTTPS targets inendpoints. Certificate verification is enabled unlessinsecure_skip_verifyis true. These settings don’t apply to HTTP targets. -
tls.ca_fileType:string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the ResourceManager’s certificate. If omitted, the Collector uses the system certificate authority pool. -
tls.ca_pemType:string. Optional. PEM-encoded certificate authority certificate content used to verify the ResourceManager’s certificate. -
tls.cert_fileType:string. Optional. Path to the PEM-encoded client certificate file to present when the ResourceManager requires mutual TLS authentication. -
tls.cert_pemType:string. Optional. PEM-encoded client certificate content to present when the ResourceManager requires mutual TLS authentication. -
tls.cipher_suitesType:array of string. Optional. TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector uses secure default cipher suites. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go. -
tls.curve_preferencesType:array of string. Optional. Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses its default curves. -
tls.include_system_ca_certs_poolType:boolean. Optional. Whether to add certificates from the system certificate authority pool to the configured certificate authorities. If true, the Collector trusts both sources. If false, the Collector trusts only the configured certificate authorities. -
tls.insecureType:boolean. Optional. Whether to ignore the other TLS settings. Leave this setting false. If true and neitherca_filenorca_pemis set, the Collector ignores every othertlssetting, includinginsecure_skip_verify, and verifies HTTPS targets against the system certificate authority pool. To skip certificate verification, setinsecure_skip_verifyinstead. -
tls.insecure_skip_verifyType:boolean. Optional. Whether to skip verification of the ResourceManager’s certificate. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the certificate. -
tls.key_fileType:string. Optional. Path to the PEM-encoded client private key file used withcert_filefor mutual TLS authentication. -
tls.key_pemType:string. Optional. PEM-encoded client private key content used withcert_pemfor mutual TLS authentication. -
tls.max_versionType:string. Optional. Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the highest TLS version it supports. -
tls.min_versionType:string. Optional. Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS 1.2. -
tls.reload_intervalType:duration. Optional. How often the Collector reloads certificate and key files. If omitted, the Collector doesn’t reload the files. -
tls.server_name_overrideType:string. Optional. Server name to verify in the ResourceManager’s certificate instead of the endpoint hostname. Use this setting when the certificate name differs from the endpoint hostname. -
tls.tpmType:object. Optional. Trusted platform module settings for the TLS private key. -
tls.tpm.authType:string. Optional. Authorization value for the trusted platform module key. -
tls.tpm.enabledType:boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key. -
tls.tpm.owner_authType:string. Optional. Owner authorization value for the trusted platform module. -
tls.tpm.pathType:string. Optional. Path to the trusted platform module device or Unix domain socket, such as/dev/tpm0or/dev/tpmrm0.