Apache Mesos

The Apache Mesos integration requires CXDOT Collector 1.8.0 or greater. Apache Mesos is a cluster manager that shares the resources of a cluster among distributed applications. Use the Apache Mesos integration to collect cluster, framework, task, resource, and process metrics from the Mesos masters and agents running in your environment. The integration also collects scheduler metrics from the Apache Aurora schedulers that run on them. The Apache Mesos integration supports Apache Mesos 1.4 or greater and Apache Aurora 0.17 or greater.

​

Supported telemetry types

The Apache Mesos integration supports these telemetry types:

​

Prerequisites

The Apache Mesos integration has the following prerequisites:

• Make the HTTP API of each Mesos master and agent, and the HTTP server of each Aurora scheduler, reachable from the CXDOT Collector. By default, masters listen on port 5050 and agents listen on port 5051. An Aurora scheduler listens on the port that its -http_port flag sets.
• Allow unauthenticated read access to each master and agent. The CXDOT Collector doesn’t send credentials, so a master or agent that runs with --authenticate_http_readonly=true rejects its requests. Mesos allows unauthenticated read access by default.

​

Configure

The Apache Mesos integration is enabled by default and collects from the masters, agents, and schedulers that you list. In a typical deployment, the CXDOT Collector runs on each Mesos node and collects from the Mesos processes on that node. To configure the Apache Mesos integration, follow these steps:

1. On each agent node, list the local agent and set cluster_name. For example, add the following to the Collector configuration file on an agent node:
   cxdot:
     mode: host
     integrations:
       mesos:
         agent:
           cluster_name: production
           endpoints:
             - endpoint: localhost:5051
   
   The CXDOT Collector reports cluster_name as the mesos.cluster attribute on every metric. The Collector doesn’t read the cluster name from Mesos, so set the same value on every node in one cluster. The Collector adds the node’s host.name to every metric, which identifies the node when every endpoint is localhost.
2. On each master node, list the local master and set cluster_name. For example, add the following to the Collector configuration file on a master node:
   cxdot:
     mode: host
     integrations:
       mesos:
         master:
           cluster_name: production
           endpoints:
             - endpoint: localhost:5050
   
   Configure every master node, not only the current leader. Every master reports its own process and host metrics, but only the elected leading master reports cluster-wide metrics, such as task, agent, framework, and resource counts. Configuring every master keeps cluster-wide metrics arriving after Mesos elects a new leader, and still reports one copy of them.
3. Optional: Collect from Aurora schedulers. On each scheduler node, list the local scheduler at the port that its -http_port flag sets, and set cluster_name to the value you set for the Mesos cluster that the schedulers run on. For example:
   cxdot:
     mode: host
     integrations:
       mesos:
         aurora:
           cluster_name: production
           endpoints:
             - endpoint: localhost:8081
   
   Configure every scheduler node. Every scheduler reports its own process metrics, but only the leading scheduler reports cluster metrics, such as task counts, resource offers, preempted tasks, and cron activity.
4. Optional: Report per-framework, per-role, and per-task metrics. Set frameworks and roles on master nodes and tasks on agent nodes. For example, the following values on a master node report per-framework metrics for the marathon framework and per-role metrics for the production role:
   cxdot:
     mode: host
     integrations:
       mesos:
         master:
           cluster_name: production
           endpoints:
             - endpoint: localhost:5050
           frameworks:
             - marathon
           roles:
             - production
   
   On an agent node, tasks: [checkout] under agent reports the allocated resources of each task whose name contains checkout. These metrics are off by default because each framework name, role name, and matching task adds time series. Framework and role names must match exactly. Task name fragments match any part of a task name, ignoring case. To report every framework or every role instead, set master.all_frameworks or master.all_roles to true.

On a node that runs more than one of these processes, such as a master and an Aurora scheduler, list them under one mesos key, with a master, agent, or aurora block for each process. To collect from a node that doesn’t run the CXDOT Collector, list that node’s address instead of localhost in another node’s configuration. List each master, agent, and scheduler for only one Collector, because a target that two Collectors list reports its metrics twice. If you run the Collector in Kubernetes instead, list each node’s address under config.integrations.mesos in the Helm values file for your CXDOT Collector.

​

Validate

To validate the Apache Mesos integration, follow these steps:

1. In Metrics Explorer, run the following query:
   {"cxdot.integration.target.health", "cxdot.integration.name"="mesos"}
   
   Confirm that the query returns one time series for each master, agent, and scheduler that you configured, identified by host.name and the URL in the cxdot.integration.target attribute, and that each one reports 1.
2. In Metrics Explorer, run the following query:
   max by ("mesos.cluster") ({"mesos.cluster.agent.count", "mesos.agent.state"="active"})
   
   Confirm that the result matches the number of active agents in each cluster.

​

Troubleshooting

• A target reports 0 for cxdot.integration.target.health: Confirm that the endpoint is reachable from the CXDOT Collector, allows unauthenticated read access, and is listed under the role its process runs. A master listed under agent.endpoints, or an agent listed under master.endpoints, reports 0.
• An Aurora scheduler reports 0 for cxdot.integration.target.health: Confirm that the port in its endpoint matches the -http_port flag that the scheduler runs with. A scheduler started without -http_port listens on a random port that can change each time it restarts. To test the endpoint, request /vars.json from it, such as curl http://localhost:8081/vars.json, and confirm that it returns a successful status code.
• Per-task metrics don’t appear: Confirm that a fragment in agent.tasks matches part of the task name and that the task is active. When the fragments match more tasks on one agent than agent.max_tasks allows, the CXDOT Collector reports no per-task metrics for that agent and reports cxdot.integration.target.health as 0 for it.
• Per-framework or per-role metrics don’t appear with all_frameworks or all_roles set: When the elected master reports more framework names than master.max_frameworks, or more roles than master.max_roles, the CXDOT Collector reports none of those metrics and reports cxdot.integration.target.health as 0 for the master.

​

Configuration reference

Configure one Apache Mesos integration instance with the following settings. In Helm values, place these settings under config.integrations.mesos. In a Collector configuration file, place them under cxdot.integrations.mesos.

​

Optional settings

• enabled Type: boolean. Optional. Default: true. Whether to enable this Apache Mesos integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it.
• collection_interval Type: duration. Optional. Default: 2m. How often the Collector collects metrics from each master, agent, and Aurora scheduler.
• timeout Type: duration. Optional. Default: 1m. Maximum time for one collection from a master, agent, or Aurora scheduler. The Collector caps the timeout at collection_interval.
• tls Type: object. Optional. TLS settings for masters, agents, and Aurora schedulers whose endpoints are https URLs. The Collector doesn’t use these settings for an endpoint in host:port form, which it reaches over HTTP. The Collector verifies server certificates by default.
• tls.ca_file Type: string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the server certificates of masters, agents, and Aurora schedulers. If omitted, the Collector uses the system certificate authority pool.
• tls.ca_pem Type: string. Optional. PEM-encoded certificate authority certificate content used to verify the server certificates of masters, agents, and Aurora schedulers.
• tls.cert_file Type: string. Optional. Path to the PEM-encoded client certificate file to present when masters, agents, and Aurora schedulers require mutual TLS authentication.
• tls.cert_pem Type: string. Optional. PEM-encoded client certificate content to present when masters, agents, and Aurora schedulers require mutual TLS authentication.
• tls.cipher_suites Type: array of string. Optional. TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector uses the secure defaults from the Go TLS library. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go.
• tls.curve_preferences Type: array of string. Optional. Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses the Go TLS library defaults.
• tls.include_system_ca_certs_pool Type: boolean. Optional. Whether to add certificates from the system certificate authority pool to the configured certificate authorities. If true, the Collector trusts both sources. If false, the Collector trusts only the configured certificate authorities.
• tls.insecure Type: boolean. Optional. Whether to ignore the other TLS settings when no certificate authority is configured. If true and neither ca_file nor ca_pem is set, the Collector connects to HTTPS endpoints with default TLS settings and verifies their certificates against the system certificate authority pool. If false, the Collector applies the configured TLS settings. The scheme of each endpoint, not this setting, decides whether the Collector uses TLS.
• tls.insecure_skip_verify Type: boolean. Optional. Whether to skip verification of the server certificates of masters, agents, and Aurora schedulers. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the certificates.
• tls.key_file Type: string. Optional. Path to the PEM-encoded client private key file used with cert_file for mutual TLS authentication.
• tls.key_pem Type: string. Optional. PEM-encoded client private key content used with cert_pem for mutual TLS authentication.
• tls.max_version Type: string. Optional. Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the maximum version supported by the Go TLS library.
• tls.min_version Type: string. Optional. Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS 1.2.
• tls.reload_interval Type: duration. Optional. How often the Collector reloads certificate and key files. If omitted, the Collector doesn’t reload the files.
• tls.server_name_override Type: string. Optional. Server name to verify in the certificates of masters, agents, and Aurora schedulers instead of the endpoint hostname. Use this setting when the certificate name differs from the endpoint hostname.
• tls.tpm Type: object. Optional. Trusted platform module settings for the TLS private key.
• tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
• tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
• tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
• tls.tpm.path Type: string. Optional. Path to the trusted platform module device or Unix domain socket, such as /dev/tpm0 or /dev/tpmrm0.
• master Type: object. Optional. Settings for collecting from Mesos masters. Every master reports its own process and host metrics. Only the elected leading master reports cluster-wide metrics, because a standby master’s copies of them are stale or zero. Listing every master therefore reports one copy of the cluster-wide metrics.
• master.enabled Type: boolean. Optional. Default: true. Whether to collect from Mesos masters. If true, the Collector collects from the masters in master.endpoints. If false, the Collector doesn’t collect from any master.
• master.endpoints Type: array of object. Optional. Default: []. Masters to collect from. Each entry is the IP address or hostname of one master, as the Collector reaches it, and the port of the master’s HTTP API, which is 5050 by default. Don’t include a path. The Collector reaches an entry in host:port form over HTTP. To reach a master over HTTPS, give the full URL, such as https://mesos-master-1.example.com:5050. List every master so that cluster-wide metrics continue after a new leader is elected.
• master.endpoints[].endpoint Type: string. Required. Master endpoint as either host:port or a full HTTP or HTTPS URL without a path. For host:port, the Collector uses HTTP.
• master.cluster_name Type: string. Optional. Minimum length: 1. Value of the mesos.cluster attribute on every metric from a master. Required when master.endpoints lists a master. The Collector doesn’t read the cluster name from Mesos, so set the same value for every master and agent in one cluster.
• master.frameworks Type: array of string. Optional. Default: []. Exact names of the frameworks for which the elected master reports per-framework task counts and allocated resources. Frameworks that share a name are reported together. When empty, the Collector reports no per-framework metrics unless all_frameworks is true. Each name adds time series.
• master.all_frameworks Type: boolean. Optional. Default: false. Whether to report per-framework metrics for every framework registered with the elected master. If true, frameworks must be empty, and each distinct framework name adds four time series. Frameworks that share a name are reported together, but a cluster that registers a framework under a new name for every job adds time series with each job. If false, the Collector reports per-framework metrics only for the names in frameworks.
• master.max_frameworks Type: integer. Optional. Default: 1000. Minimum: 1. Maximum number of distinct framework names that one collection reports when all_frameworks is true. When the elected master reports more, the Collector reports no per-framework metrics from that collection and reports cxdot.integration.target.health as 0 for the master. The collection’s other metrics are still reported.
• master.roles Type: array of string. Optional. Default: []. Exact names of the roles for which the elected master reports per-role framework counts, weights, and allocated resources. When empty, the Collector reports no per-role metrics and doesn’t request role information unless all_roles is true. Each name adds time series.
• master.all_roles Type: boolean. Optional. Default: false. Whether to report per-role metrics for every role that the elected master lists, including the default role *. If true, roles must be empty, and each role adds up to five time series. If false, the Collector reports per-role metrics only for the names in roles.
• master.max_roles Type: integer. Optional. Default: 1000. Minimum: 1. Maximum number of roles that one collection reports when all_roles is true. When the elected master lists more, the Collector reports no per-role metrics from that collection and reports cxdot.integration.target.health as 0 for the master. The collection’s other metrics are still reported.
• agent Type: object. Optional. Settings for collecting from Mesos agents.
• agent.enabled Type: boolean. Optional. Default: true. Whether to collect from Mesos agents. If true, the Collector collects from the agents in agent.endpoints. If false, the Collector doesn’t collect from any agent.
• agent.endpoints Type: array of object. Optional. Default: []. Agents to collect from. Each entry is the IP address or hostname of one agent, as the Collector reaches it, and the port of the agent’s HTTP API, which is 5051 by default. Don’t include a path. Entries take the same host:port and URL forms as master.endpoints.
• agent.endpoints[].endpoint Type: string. Required. Agent endpoint as either host:port or a full HTTP or HTTPS URL without a path. For host:port, the Collector uses HTTP.
• agent.cluster_name Type: string. Optional. Minimum length: 1. Value of the mesos.cluster attribute on every metric from an agent. Required when agent.endpoints lists an agent. The Collector doesn’t read the cluster name from Mesos, so set the same value for every master and agent in one cluster.
• agent.tasks Type: array of string. Optional. Default: []. Task name fragments that select tasks for per-task metrics. For each active task on an agent whose name contains a fragment, ignoring case, the Collector reports the task’s allocated CPUs, memory, and disk, labeled with the task name, task ID, and framework ID. A nonempty list adds a request for the agent’s full state to every collection. When empty, the Collector reports no per-task metrics. A blank fragment is rejected because it matches every task.
• agent.max_tasks Type: integer. Optional. Default: 1000. Minimum: 1. Maximum number of active tasks that agent.tasks can select on one agent. When the fragments match more tasks, the Collector reports no per-task metrics for that agent and reports cxdot.integration.target.health as 0 for it.
• aurora Type: object. Optional. Settings for collecting from Apache Aurora schedulers. Every scheduler reports its own process metrics. Only the leading scheduler reports cluster metrics, such as task counts, resource offers, preempted tasks, and cron activity, so listing every scheduler reports one copy of them.
• aurora.enabled Type: boolean. Optional. Default: true. Whether to collect from Apache Aurora schedulers. If true, the Collector collects from the schedulers in aurora.endpoints. If false, the Collector doesn’t collect from any scheduler.
• aurora.endpoints Type: array of object. Optional. Default: []. Aurora schedulers to collect from. Each entry is the IP address or hostname of one scheduler, as the Collector reaches it, and the port of its HTTP server, which its -http_port flag sets. Don’t include a path. Entries take the same host:port and URL forms as master.endpoints. List every scheduler so that cluster metrics continue after a new leader is elected.
• aurora.endpoints[].endpoint Type: string. Required. Aurora scheduler endpoint as either host:port or a full HTTP or HTTPS URL without a path. For host:port, the Collector uses HTTP.
• aurora.cluster_name Type: string. Optional. Minimum length: 1. Value of the mesos.cluster attribute on every metric from an Aurora scheduler. Required when aurora.endpoints lists a scheduler. Aurora doesn’t report the Mesos cluster that it runs on, so set the master.cluster_name of that cluster to select scheduler and cluster metrics together.