Skip to main content
The Hadoop YARN integration requires CXDOT Collector 1.18.0 or greater. Apache Hadoop is an open source framework for distributed storage and processing, and its Yet Another Resource Negotiator (YARN) component schedules the applications that run on a cluster. Use the Hadoop YARN integration with the CXDOT Collector to collect cluster, application, node, and queue metrics from the ResourceManagers of Hadoop clusters running in your environment. The Hadoop YARN integration supports Apache Hadoop 3.4 or greater.

Supported telemetry types

The Hadoop YARN integration supports these telemetry types:

Prerequisites

The Hadoop YARN integration has the following prerequisites:
  • Make the ResourceManager’s web interface, which listens on port 8088 by default, reachable from the CXDOT Collector. The integration reads the ResourceManager’s REST API, and the web interface can serve it over HTTP or HTTPS. To confirm that the CXDOT Collector can read it, run the following command from a pod or host on the same network as the CXDOT Collector, replacing RESOURCEMANAGER_ADDRESS with the ResourceManager’s address. The command returns the HTTP status code 200. On a cluster with ResourceManager high availability, a standby ResourceManager returns 307 instead, which is expected:
  • Check how the ResourceManager’s web interface authenticates requests. The integration supports unauthenticated access, HTTP basic authentication, custom request headers such as a bearer token for an authenticating proxy, and Transport Layer Security (TLS). The integration doesn’t support Kerberos (SPNEGO) authentication, so a web interface that requires Kerberos returns an error for every request.
  • On a cluster with ResourceManager high availability, choose how to target the ResourceManagers. A standby ResourceManager redirects every request to the active one on another host. Use one of the following:
    • Annotate the ResourceManager pods, or list each ResourceManager as a static target. The integration treats a target that redirects to another host as a standby. The active ResourceManager reports the cluster’s metrics, and each standby reports as healthy and reports no other metrics, so the cluster’s metrics aren’t duplicated.
    • Annotate a Kubernetes Service that selects the ResourceManager pods. The integration follows the standby’s redirect for a target it discovers from a Service, so the target reports the cluster’s metrics whichever ResourceManager answers.
    • List one address that reaches both ResourceManagers, such as a virtual IP address or a load balancer, as a static target, and set follow_standby_redirects to true. Without that setting, the target reports as healthy and reports no metrics whenever one of its requests reaches the standby. You don’t need the setting if the address routes requests only to the active ResourceManager. For example, a load balancer can run a health check on the ResourceManager’s /isActive path, which returns the HTTP status code 200 only on the active ResourceManager.
    When the integration follows a standby’s redirect, it sends the headers values to the active ResourceManager but not the username and password credentials.

Configure

The Hadoop YARN integration is enabled by default, but it collects only from the ResourceManagers you give it. You can configure targets either by annotating ResourceManager pods in your Kubernetes cluster or by listing static endpoints in your Helm values. To configure the integration, follow these steps:
  1. To collect from a ResourceManager running in Kubernetes, add a CXDOT integration annotation to the pod template. The annotation key must end with the name of the ResourceManager’s container, which is resourcemanager in the following example. The yarn: block in the annotation’s value chooses the Hadoop YARN integration. For example, add the following annotation to your ResourceManager workload:
    The Collector replaces ${target.host} with the pod IP address and ${target.port.ui} with the container port named ui, and it reads the ResourceManager over HTTP. The cluster_name value becomes the yarn.cluster.name attribute on the metrics, which tells clusters apart because a ResourceManager doesn’t report its cluster’s name. On a cluster with ResourceManager high availability, an annotation on the pod template reaches every ResourceManager. The active ResourceManager reports the cluster’s metrics. Each standby ResourceManager reports cxdot.integration.target.health as 1 and no other metrics. To collect through a Service that selects the ResourceManager pods instead, add the same annotation value to the Service under the integrations.cxdot.paloaltonetworks.com/service key. The Collector then replaces ${target.host} with the Service’s cluster IP address and ${target.port.ui} with the Service port named ui, and it follows a standby ResourceManager’s redirect to the active one.
  2. Optional: To collect from ResourceManagers outside your Kubernetes cluster, configure static targets. For example, add the following to the Helm values file for your CXDOT Collector:
    Write a target as host:port to read it over HTTP, or as a full HTTP or HTTPS URL. When you configure static targets, this integration instance collects from exactly those targets instead of discovered targets. In cluster mode, the cluster Collector that wins leader election scrapes static targets.
  3. Optional: Configure a second integration instance to collect from both annotated pods and static targets. For example, add the following to the Helm values file for your CXDOT Collector:
    The yarn/external key defines a named instance for the static targets. The bare yarn key continues to collect from annotated pods.
  4. Optional: If the ResourceManager’s web interface requires HTTP basic authentication or HTTPS, configure the matching settings for static targets, and provide the password through your deployment’s secret management. For example, the following Helm values reference the YARN_PASSWORD environment variable and a mounted certificate authority file:
    The username, password, and tls settings in your Helm values apply to the targets you list there. Set username and password together. You can combine them with headers. If headers also sets an Authorization header, the integration sends the basic credentials in its place. The integration follows an HTTP redirect only when the redirect stays on the target’s host, for example from a proxy to another path on the proxy. To make such a request fail instead, set follow_redirects to false. The integration follows a redirect to another host only for a target it discovers from a Service, or when follow_standby_redirects is true, as described in Prerequisites. Otherwise, it sends the credentials and the headers values only to the host you configured. Set the password variable on both the node and cluster Collectors with the nodeCollector.extraEnv and clusterCollector.extraEnv Helm values. Mount the certificate authority file into the cluster Collector, which collects from static targets, with the clusterCollector.extraVolumes and clusterCollector.extraVolumeMounts Helm values. For annotated pods, set these settings in the CXDOT integration annotation instead. Reference the password with password: ${k8s_secret:yarn-credentials/password} and supply the certificate authority from a Kubernetes Secret with tls.ca_pem: ${k8s_secret:yarn-credentials/ca.crt}. To allow the Collector to read the Secret, add the pod’s namespace to apiServer.metadataServer.secretAllowlist in the Helm values for your CXDOT Collector. Use a URL that starts with https:// in the annotation’s endpoint value. Because ${target.host} is the pod IP address, set tls.server_name_override to a name that the ResourceManager’s certificate carries, unless the certificate carries the pod IP address.
  5. Optional: To limit the series the application, node, and queue metrics create, adjust which applications, nodes, and queues report. For example, add the following to the Helm values file for your CXDOT Collector:
    These settings in your Helm values apply to the static targets of the same instance. For annotated pods, set application_states, application_id, max_applications, node_metrics, max_nodes, inactive_node_retention, queue_metrics, exclude_queues, and max_queues in the CXDOT integration annotation instead. The integration reports yarn.application.status for every application that hasn’t ended, and for every failed or killed application that the ResourceManager still lists. The status is 2, which means critical, when the application’s state or its final status is FAILED or KILLED. An application whose application master reports a failure and then exits cleanly, such as a Spark application whose driver fails, ends in the FINISHED state with the final status FAILED. The integration also reports a critical status for those applications, so an alert on a critical status covers failed and killed applications without changing a setting. The application_states setting chooses which applications also report resource and progress metrics, and defaults to RUNNING. Other finished applications report a status only if you add FINISHED to application_states. That includes a finished application whose final status is KILLED, which then reports a critical status. The yarn.cluster.application.terminations metric counts ended applications by final state without this setting. By default, every yarn.application.* series carries the application’s ID in the yarn.application.id attribute, so each run of an application creates a new set of series. On a cluster that runs many short-lived applications, set application_id to false to leave the ID off. Applications that share a name, queue, state, and user then report as one, with their memory, virtual cores, and containers summed. The integration also stops reporting yarn.application.memory.consumed and yarn.application.vcore.consumed. The max_applications limit, which defaults to 2000, caps how many applications report, or how many groups of applications when application_id is false. Applications that haven’t ended report first, starting with the one that has run longest, so an application that reports keeps reporting while newer ones arrive. Ended applications follow, starting with the one that ended most recently. If the applications that haven’t ended fill the limit, failed and killed applications report no yarn.application.status. A ResourceManager keeps listing a node that has left the cluster until the ResourceManager restarts. A NodeManager that listens on an ephemeral port, which is the default, registers as a new node each time it starts, so the listed nodes grow with every NodeManager restart. The integration reports yarn.node.* metrics for a node whose state is LOST, SHUTDOWN, DECOMMISSIONED, or REBOOTED only while the node’s last health report is newer than inactive_node_retention, which defaults to 4h. The max_nodes limit, which defaults to 1000, caps how many nodes report, and nodes that are part of the cluster report first. The yarn.cluster.node.count metric counts every node by state regardless of both settings. The integration fills the max_queues limit, which defaults to 100, from the queue nearest the root outward, one level at a time, and each level follows the order that the scheduler lists its queues in. On a scheduler with more queues, the deepest queues stop reporting first. Use exclude_queues to choose which queues report, by name or by full path. A queue you exclude also excludes its child queues. On a cluster that uses the Fair scheduler or the first in, first out (FIFO) scheduler, which report no queue metrics, set queue_metrics to false so the integration doesn’t request scheduler information.
  6. Optional: Disable the integration. For example, add the following to the Helm values file for your CXDOT Collector:
Metrics from each discovered pod include that pod’s Kubernetes metadata. For more information, see autodiscovery.

Validate

To validate the Hadoop YARN integration, follow these steps:
  1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health and cxdot.integration.name=yarn. Confirm that cxdot.integration.target.health reports 1 for each ResourceManager, including each standby ResourceManager. A value of 0 means that at least one request from the CXDOT Collector to that ResourceManager failed. The cxdot.integration.target attribute identifies the target.
  2. In Metrics Explorer, run the following query:
    Confirm that the query returns a time series for each node state of each active ResourceManager, and that the running series reports the number of NodeManagers that are running. The integration reports memory in MiB and times in milliseconds, as the ResourceManager reports them.

Troubleshooting

  • cxdot.integration.target.health reads 0, but some metrics still report: The CXDOT Collector makes up to five requests to the ResourceManager, for cluster, application, node, and queue data. A request that fails or times out costs its own metrics, and the others still report. Check which metric prefix is missing: yarn.cluster.*, yarn.application.*, yarn.node.*, or yarn.queue.*.
  • Every request fails for a secured cluster: The integration doesn’t support Kerberos (SPNEGO) authentication. Reach a ResourceManager that requires it through an address that accepts basic authentication or a header-based proxy, and set username and password or headers to match.
  • No yarn.queue.* metrics appear: The integration reads queues only from the CapacityScheduler. With the Fair scheduler or the first in, first out scheduler, the integration still reports the yarn.cluster.*, yarn.application.*, and yarn.node.* metrics. The queue_metrics setting also turns these metrics off.
  • Some queues report no metrics: The scheduler has more queues than max_queues, and the Collector logs a warning that says so. Raise max_queues or use exclude_queues.
  • Some applications report no metrics, or failed and killed applications report no yarn.application.status: The ResourceManager lists more applications than max_applications, and the Collector logs a warning that says so. Applications that haven’t ended fill the limit first. Raise max_applications, or set application_id to false so that applications that share a name, queue, state, and user count once.
  • cxdot.integration.target.health reads 1, but no yarn.* metrics report: The target redirects to another host, so the integration treats it as a standby ResourceManager, and the Collector logs a message that names the host. For a standby ResourceManager, the active ResourceManager’s target reports the metrics. For a target that’s one address for both ResourceManagers, such as a virtual IP address or a load balancer, a collection reports no metrics whenever one of its requests reaches the standby. Set follow_standby_redirects to true for that target, or route its requests only to the active ResourceManager, as described in Prerequisites. For any other target, such as a proxy that redirects to another host name, target the address it redirects to.
  • Every request fails with without a usable Location: A standby ResourceManager answers that way while no ResourceManager is active. Check the state of the ResourceManagers.
  • Some nodes report no yarn.node.* metrics: The ResourceManager lists more nodes than max_nodes, and the Collector logs a warning that says so. Nodes that are part of the cluster fill the limit first. Raise max_nodes. To stop collecting the yarn.node.* metrics, set node_metrics to false. The yarn.cluster.* metrics still report the cluster’s totals and its node counts by state.
  • A lost, shut down, decommissioned, or rebooted node stops reporting yarn.node.* metrics: The node’s last health report is older than inactive_node_retention. A NodeManager reports its health when its health checks run. If the NodeManager runs neither the disk health check nor a health script, its last health report is the time it started, so a node that ran longer than inactive_node_retention stops reporting as soon as it leaves the cluster. Raise inactive_node_retention to keep such nodes reporting longer. The yarn.cluster.node.count metric still counts the node.
  • A redirected request fails with redirect not followed: The target redirects to another address on the same host, and follow_redirects is false or your deployment denies redirects for annotated pods. Target the address the ResourceManager redirects to.
  • yarn.application.memory.consumed and yarn.application.vcore.consumed are missing: The integration doesn’t report them when application_id is false.
  • Metrics for a Spark application’s own executors and jobs don’t appear: The integration reads only the ResourceManager. The Spark integration collects those metrics from the Spark driver.

Configuration reference

Configure one Hadoop YARN integration instance with the following settings. In Helm values, place these settings under collection.cxdot.integrations.yarn. In a Collector configuration file, place them under cxdot.integrations.yarn.

Optional settings

  • enabled Type: boolean. Optional. Default: true. Whether to enable this Hadoop YARN integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it.
  • endpoints Type: array of object. Optional. Default: []. ResourceManager targets. Specify a target as host:port to use HTTP, or as a full HTTP or HTTPS URL of the ResourceManager’s web interface, without a REST API path. On a cluster with ResourceManager high availability, list each ResourceManager. A standby ResourceManager redirects to the active one, and the Collector treats a target that redirects to another host as a standby: the target reports as healthy and reports no metrics. If you list one address that reaches both ResourceManagers instead, such as a virtual IP address or a load balancer, set follow_standby_redirects to true, unless the address routes requests only to the active ResourceManager.
  • endpoints[].endpoint Type: string. Required. host:port of the ResourceManager’s web interface, or its HTTP or HTTPS URL.
  • endpoints[].cluster_name Type: string. Optional. Name of the YARN cluster the ResourceManager belongs to, reported as the yarn.cluster.name attribute. When omitted, the attribute isn’t set.
  • application_metrics Type: boolean. Optional. Default: true. Whether to collect the yarn.application.* metrics. When false, the Collector doesn’t request the ResourceManager’s application list.
  • application_id Type: boolean. Optional. Default: true. Whether the yarn.application.* metrics carry the application’s ID in the yarn.application.id attribute. When true, each run of an application is its own set of series. When false, the Collector leaves the ID off, and applications that share a name, queue, state, and user report as one. Their memory, virtual cores, and containers are summed. Progress is the lowest, start time the earliest, and finish time and elapsed time the latest and longest. The Collector then doesn’t report yarn.application.memory.consumed and yarn.application.vcore.consumed, because a total summed over applications that come and go would fall whenever one ends. Set this to false on a cluster that runs many short-lived applications.
  • max_applications Type: integer. Optional. Default: 2000. Minimum: 1. Cap on how many applications report yarn.application.* metrics, or, when application_id is false, on how many groups of applications that report as one. Applications that haven’t ended report first, starting with the one that has run longest, so an application that reports keeps reporting while newer ones arrive. Ended applications follow, starting with the one that ended most recently. On a ResourceManager with more applications than the cap, the rest report no metrics and the Collector logs a warning. If the applications that haven’t ended fill the cap, failed and killed applications report no yarn.application.status.
  • application_states Type: array of string. Optional. Default: ["RUNNING"]. Application states to collect resource and progress metrics for. This setting doesn’t limit yarn.application.status, which the Collector reports for every application that hasn’t ended (NEW, NEW_SAVING, SUBMITTED, ACCEPTED, and RUNNING) and for every failed or killed application the ResourceManager still lists. The exception is finished applications, which make up most of the ended applications a ResourceManager retains. Unless you list FINISHED here, the Collector asks the ResourceManager only for the finished applications whose final status is FAILED, such as a Spark application whose driver reported a failure, and reports their status as critical. When you list FINISHED, every finished application reports a status, which is critical if its final status is FAILED or KILLED. The Collector reports a finished application whose final status is KILLED only when you list FINISHED. The yarn.cluster.application.terminations metric counts ended applications by final state.
  • node_metrics Type: boolean. Optional. Default: true. Whether to collect the yarn.node.* metrics. When false, the Collector doesn’t request the ResourceManager’s node list.
  • max_nodes Type: integer. Optional. Default: 1000. Minimum: 1. Cap on how many nodes report yarn.node.* metrics. Nodes that are part of the cluster report first, in order of node ID. Nodes that have left the cluster and are still within inactive_node_retention follow, starting with the one that reported its health most recently. On a ResourceManager with more nodes than the cap, the rest report no yarn.node.* metrics and the Collector logs a warning. The yarn.cluster.node.count metric counts every node regardless of this setting.
  • inactive_node_retention Type: duration. Optional. Default: 4h. How long a node that has left the cluster keeps reporting yarn.node.* metrics, measured from its last health report. A node has left the cluster when its state is LOST, SHUTDOWN, DECOMMISSIONED, or REBOOTED. A ResourceManager keeps listing such a node until the ResourceManager restarts. A NodeManager that listens on an ephemeral port, which is the default, registers as a new node each time it starts, so without this limit the number of nodes that report grows with every NodeManager restart. Set this to 0s to report no nodes that have left the cluster. The yarn.cluster.node.count metric counts every node regardless of this setting.
  • queue_metrics Type: boolean. Optional. Default: true. Whether to collect the yarn.queue.* metrics. When false, the Collector doesn’t request the ResourceManager’s scheduler information, and the yarn.queue.path attribute of the yarn.application.* metrics is the queue as the ResourceManager reports it for the application. Set this to false on a cluster that uses the Fair scheduler or the first in, first out scheduler, which report no queue metrics.
  • exclude_queues Type: array of string. Optional. Default: []. CapacityScheduler queues to leave out, by name (adhoc) or by full path (root.analytics.adhoc). Leaving a queue out also leaves out its child queues.
  • max_queues Type: integer. Optional. Default: 100. Minimum: 1. Cap on how many queues, not counting the root, report metrics, filled from the queue nearest the root outward, one level at a time, with each level in the order the scheduler lists its queues. On a scheduler with more queues, use exclude_queues to choose which ones report.
  • username Type: string. Optional. Username for HTTP basic authentication to the targets in endpoints. Set username and password together. The Collector sends the credentials in an Authorization header on every request, in place of an Authorization header set in headers. Kerberos (SPNEGO) authentication isn’t supported.
  • password Type: string. Optional. Password for HTTP basic authentication to the targets in endpoints. Set username and password together. The Collector masks this value in diagnostic output, logs, and errors. Kerberos (SPNEGO) authentication isn’t supported.
  • headers Type: object. Optional. Additional request headers as name-value pairs, sent to every target in endpoints, for example a bearer token in Authorization for a ResourceManager behind an authenticating proxy. You can combine headers with username and password. If you do, the Collector doesn’t send an Authorization header set here, and sends the basic credentials instead. The Collector sends these headers only to the target’s own host, unless follow_standby_redirects is true. In that case, the Collector also sends them to the active ResourceManager that a standby redirects to. The Collector masks the values in diagnostic output, logs, and errors. This setting doesn’t enable Kerberos (SPNEGO) authentication, which isn’t supported.
  • follow_redirects Type: boolean. Optional. Default: true. Whether to follow HTTP redirects that stay on the target’s host, for example from a proxy to another path on the proxy. If false, such a redirect causes the metric collection to fail. Redirects to another host are controlled by follow_standby_redirects, and the Collector follows them only if both settings are true.
  • follow_standby_redirects Type: boolean. Optional. Default: false. Whether to follow a standby ResourceManager’s redirect to the active ResourceManager. A standby ResourceManager answers every request with a redirect to the active one on another host. If false, the Collector doesn’t follow that redirect and treats the target as a standby: the target reports as healthy and reports no metrics. Use false when each target is a single ResourceManager, so that only the active one reports the cluster’s metrics. If true, the Collector follows the redirect, and the target reports the cluster’s metrics whichever ResourceManager answers. Use true when a target is one address that reaches both ResourceManagers, such as a virtual IP address or a load balancer that doesn’t route only to the active ResourceManager. With false, such a target reports no metrics whenever one of its requests reaches the standby. The Collector always follows the redirect for a target discovered from a Kubernetes Service, which is one address for every ResourceManager behind it. A redirect to another host is followed only if follow_redirects is also true. The Collector doesn’t send the username and password credentials to the active ResourceManager that a standby redirects to, so if the active ResourceManager requires authentication, authenticate with headers.
  • collection_interval Type: duration. Optional. Default: 15s. How often the Collector collects metrics from each ResourceManager.
  • timeout Type: duration. Optional. Default: 15s. Maximum time allowed for one collection from a ResourceManager. The limit applies to each request the Collector makes and to the collection as a whole. The value must not exceed collection_interval.
  • tls Type: object. Optional. Transport layer security (TLS) settings for the HTTPS targets in endpoints. Certificate verification is enabled unless insecure_skip_verify is true. These settings don’t apply to HTTP targets.
  • tls.ca_file Type: string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the ResourceManager’s certificate. If omitted, the Collector uses the system certificate authority pool.
  • tls.ca_pem Type: string. Optional. PEM-encoded certificate authority certificate content used to verify the ResourceManager’s certificate.
  • tls.cert_file Type: string. Optional. Path to the PEM-encoded client certificate file to present when the ResourceManager requires mutual TLS authentication.
  • tls.cert_pem Type: string. Optional. PEM-encoded client certificate content to present when the ResourceManager requires mutual TLS authentication.
  • tls.cipher_suites Type: array of string. Optional. TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector uses secure default cipher suites. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go.
  • tls.curve_preferences Type: array of string. Optional. Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses its default curves.
  • tls.include_system_ca_certs_pool Type: boolean. Optional. Whether to add certificates from the system certificate authority pool to the configured certificate authorities. If true, the Collector trusts both sources. If false, the Collector trusts only the configured certificate authorities.
  • tls.insecure Type: boolean. Optional. Whether to ignore the other TLS settings. Leave this setting false. If true and neither ca_file nor ca_pem is set, the Collector ignores every other tls setting, including insecure_skip_verify, and verifies HTTPS targets against the system certificate authority pool. To skip certificate verification, set insecure_skip_verify instead.
  • tls.insecure_skip_verify Type: boolean. Optional. Whether to skip verification of the ResourceManager’s certificate. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the certificate.
  • tls.key_file Type: string. Optional. Path to the PEM-encoded client private key file used with cert_file for mutual TLS authentication.
  • tls.key_pem Type: string. Optional. PEM-encoded client private key content used with cert_pem for mutual TLS authentication.
  • tls.max_version Type: string. Optional. Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the highest TLS version it supports.
  • tls.min_version Type: string. Optional. Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS 1.2.
  • tls.reload_interval Type: duration. Optional. How often the Collector reloads certificate and key files. If omitted, the Collector doesn’t reload the files.
  • tls.server_name_override Type: string. Optional. Server name to verify in the ResourceManager’s certificate instead of the endpoint hostname. Use this setting when the certificate name differs from the endpoint hostname.
  • tls.tpm Type: object. Optional. Trusted platform module settings for the TLS private key.
  • tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
  • tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
  • tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
  • tls.tpm.path Type: string. Optional. Path to the trusted platform module device or Unix domain socket, such as /dev/tpm0 or /dev/tpmrm0.