> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Apache Mesos

> Apache Mesos cluster metrics read from the masters' and agents' HTTP APIs, including cluster-wide task, agent, and resource counts, per-framework and per-role allocations, and each agent's tasks, executors, and resources, plus Apache Aurora scheduler metrics, including task counts by state, resource offers, preemptions, and cron activity.

The Apache Mesos integration requires CXDOT Collector 1.8.0 or greater.

[Apache Mesos](https://mesos.apache.org/) is a cluster manager that shares the
resources of a cluster among distributed applications. Use the Apache Mesos
integration to collect cluster, framework, task, resource, and process metrics from
the Mesos masters and agents running in your environment. The integration also
collects scheduler metrics from the [Apache Aurora](https://aurora.apache.org/)
schedulers that run on them.

The Apache Mesos integration supports Apache Mesos 1.4 or greater and Apache Aurora
0.17 or greater.

## Supported telemetry types

The Apache Mesos integration supports these telemetry types:

| Type | Supported |
| :-: | :-: |
| Logs | No |
| Metrics | Yes |
| Traces | No |
| Events | No |

## Prerequisites

The Apache Mesos integration has the following prerequisites:

* Make the HTTP API of each Mesos master and agent, and the HTTP server of each
  Aurora scheduler, reachable from the CXDOT Collector. By default, masters listen on
  port `5050` and agents listen on port `5051`. An Aurora scheduler listens on the
  port that its `-http_port` flag sets.
* Allow unauthenticated read access to each master and agent. The CXDOT Collector
  doesn't send credentials, so a master or agent that runs with
  [`--authenticate_http_readonly=true`](https://mesos.apache.org/documentation/latest/authentication/)
  rejects its requests. Mesos allows unauthenticated read access by default.

## Configure

The Apache Mesos integration is enabled by default and collects from the masters,
agents, and schedulers that you list. In a typical deployment, the CXDOT Collector
runs on each Mesos node and collects from the Mesos processes on that node. To
configure the Apache Mesos integration, follow these steps:

1. On each agent node, list the local agent and set `cluster_name`. For example, add
   the following to the Collector configuration file on an agent node:

   ```yaml theme={null}
   cxdot:
     mode: host
     integrations:
       mesos:
         agent:
           cluster_name: production
           endpoints:
             - endpoint: localhost:5051
   ```

   The CXDOT Collector reports `cluster_name` as the `mesos.cluster` attribute on every
   metric. The Collector doesn't read the cluster name from Mesos, so set the same
   value on every node in one cluster. The Collector adds the node's `host.name` to
   every metric, which identifies the node when every endpoint is `localhost`.

2. On each master node, list the local master and set `cluster_name`. For example,
   add the following to the Collector configuration file on a master node:

   ```yaml theme={null}
   cxdot:
     mode: host
     integrations:
       mesos:
         master:
           cluster_name: production
           endpoints:
             - endpoint: localhost:5050
   ```

   Configure every master node, not only the current leader. Every master reports
   its own process and host metrics, but only the elected leading master reports
   cluster-wide metrics, such as task, agent, framework, and resource counts.
   Configuring every master keeps cluster-wide metrics arriving after Mesos elects a
   new leader, and still reports one copy of them.

3. Optional: Collect from Aurora schedulers. On each scheduler node, list the local
   scheduler at the port that its `-http_port` flag sets, and set `cluster_name` to
   the value you set for the Mesos cluster that the schedulers run on. For example:

   ```yaml theme={null}
   cxdot:
     mode: host
     integrations:
       mesos:
         aurora:
           cluster_name: production
           endpoints:
             - endpoint: localhost:8081
   ```

   Configure every scheduler node. Every scheduler reports its own process metrics,
   but only the leading scheduler reports cluster metrics, such as task counts,
   resource offers, preempted tasks, and cron activity.

4. Optional: Report per-framework, per-role, and per-task metrics. Set `frameworks`
   and `roles` on master nodes and `tasks` on agent nodes. For example, the following
   values on a master node report per-framework metrics for the `marathon` framework
   and per-role metrics for the `production` role:

   ```yaml theme={null}
   cxdot:
     mode: host
     integrations:
       mesos:
         master:
           cluster_name: production
           endpoints:
             - endpoint: localhost:5050
           frameworks:
             - marathon
           roles:
             - production
   ```

   On an agent node, `tasks: [checkout]` under `agent` reports the allocated
   resources of each task whose name contains `checkout`. These metrics are off by
   default because each framework name, role name, and matching task adds time
   series. Framework and role names must match exactly. Task name fragments match any
   part of a task name, ignoring case. To report every framework or every role
   instead, set `master.all_frameworks` or `master.all_roles` to `true`.

On a node that runs more than one of these processes, such as a master and an Aurora
scheduler, list them under one `mesos` key, with a `master`, `agent`, or `aurora`
block for each process.

To collect from a node that doesn't run the CXDOT Collector, list that node's address
instead of `localhost` in another node's configuration. List each master, agent, and
scheduler for only one Collector, because a target that two Collectors list reports
its metrics twice. If you run the Collector in Kubernetes instead, list each node's
address under `config.integrations.mesos` in the Helm values file for your CXDOT
Collector.

### Validate

To validate the Apache Mesos integration, follow these steps:

1. In Metrics Explorer, run the following query:

   ```text theme={null}
   {"cxdot.integration.target.health", "cxdot.integration.name"="mesos"}
   ```

   Confirm that the query returns one time series for each master, agent, and
   scheduler that you configured, identified by `host.name` and the URL in the
   `cxdot.integration.target` attribute, and that each one reports `1`.

2. In Metrics Explorer, run the following query:

   ```text theme={null}
   max by ("mesos.cluster") ({"mesos.cluster.agent.count", "mesos.agent.state"="active"})
   ```

   Confirm that the result matches the number of active agents in each cluster.

### Troubleshooting

* A target reports `0` for `cxdot.integration.target.health`: Confirm that the
  endpoint is reachable from the CXDOT Collector, allows unauthenticated read access,
  and is listed under the role its process runs. A master listed under
  `agent.endpoints`, or an agent listed under `master.endpoints`, reports `0`.
* An Aurora scheduler reports `0` for `cxdot.integration.target.health`: Confirm that
  the port in its endpoint matches the `-http_port` flag that the scheduler runs with.
  A scheduler started without `-http_port` listens on a random port that can change
  each time it restarts. To test the endpoint, request `/vars.json` from it, such as
  `curl http://localhost:8081/vars.json`, and confirm that it returns a successful
  status code.
* Per-task metrics don't appear: Confirm that a fragment in `agent.tasks` matches part
  of the task name and that the task is active. When the fragments match more tasks on
  one agent than `agent.max_tasks` allows, the CXDOT Collector reports no per-task
  metrics for that agent and reports `cxdot.integration.target.health` as `0` for it.
* Per-framework or per-role metrics don't appear with `all_frameworks` or `all_roles`
  set: When the elected master reports more framework names than
  `master.max_frameworks`, or more roles than `master.max_roles`, the CXDOT Collector
  reports none of those metrics and reports `cxdot.integration.target.health` as `0`
  for the master.

## Configuration reference

Configure one Apache Mesos integration instance with the following settings. In Helm values,
place these settings under `config.integrations.mesos`. In a Collector configuration file, place
them under `cxdot.integrations.mesos`.

### Optional settings

* **`enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to enable this Apache Mesos integration instance. If true, the Collector runs the
  instance. If false, the Collector doesn't run it.

* **`collection_interval`**
  Type: `duration`. Optional. Default: `2m`.
  How often the Collector collects metrics from each master, agent, and Aurora scheduler.

* **`timeout`**
  Type: `duration`. Optional. Default: `1m`.
  Maximum time for one collection from a master, agent, or Aurora scheduler. The Collector caps
  the timeout at `collection_interval`.

* **`tls`**
  Type: `object`. Optional.
  TLS settings for masters, agents, and Aurora schedulers whose endpoints are `https` URLs. The
  Collector doesn't use these settings for an endpoint in `host:port` form, which it reaches
  over HTTP. The Collector verifies server certificates by default.

* **`tls.ca_file`**
  Type: `string`. Optional.
  Path to a PEM-encoded certificate authority (CA) certificate file used to verify the server
  certificates of masters, agents, and Aurora schedulers. If omitted, the Collector uses the
  system certificate authority pool.

* **`tls.ca_pem`**
  Type: `string`. Optional.
  PEM-encoded certificate authority certificate content used to verify the server certificates
  of masters, agents, and Aurora schedulers.

* **`tls.cert_file`**
  Type: `string`. Optional.
  Path to the PEM-encoded client certificate file to present when masters, agents, and Aurora
  schedulers require mutual TLS authentication.

* **`tls.cert_pem`**
  Type: `string`. Optional.
  PEM-encoded client certificate content to present when masters, agents, and Aurora schedulers
  require mutual TLS authentication.

* **`tls.cipher_suites`**
  Type: `array of string`. Optional.
  TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector
  uses the secure defaults from the Go TLS library. For supported names, see
  [https://go.dev/src/crypto/tls/cipher\_suites.go](https://go.dev/src/crypto/tls/cipher_suites.go).

* **`tls.curve_preferences`**
  Type: `array of string`. Optional.
  Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral
  handshake, in preference order. If omitted, the Collector uses the Go TLS library defaults.

* **`tls.include_system_ca_certs_pool`**
  Type: `boolean`. Optional.
  Whether to add certificates from the system certificate authority pool to the configured
  certificate authorities. If true, the Collector trusts both sources. If false, the Collector
  trusts only the configured certificate authorities.

* **`tls.insecure`**
  Type: `boolean`. Optional.
  Whether to ignore the other TLS settings when no certificate authority is configured. If true
  and neither `ca_file` nor `ca_pem` is set, the Collector connects to HTTPS endpoints with
  default TLS settings and verifies their certificates against the system certificate authority
  pool. If false, the Collector applies the configured TLS settings. The scheme of each
  endpoint, not this setting, decides whether the Collector uses TLS.

* **`tls.insecure_skip_verify`**
  Type: `boolean`. Optional.
  Whether to skip verification of the server certificates of masters, agents, and Aurora
  schedulers. If true, the connection is encrypted but isn't authenticated and is vulnerable to
  man-in-the-middle attacks. If false, the Collector verifies the certificates.

* **`tls.key_file`**
  Type: `string`. Optional.
  Path to the PEM-encoded client private key file used with `cert_file` for mutual TLS
  authentication.

* **`tls.key_pem`**
  Type: `string`. Optional.
  PEM-encoded client private key content used with `cert_pem` for mutual TLS authentication.

* **`tls.max_version`**
  Type: `string`. Optional.
  Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the maximum
  version supported by the Go TLS library.

* **`tls.min_version`**
  Type: `string`. Optional.
  Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS
  1.2.

* **`tls.reload_interval`**
  Type: `duration`. Optional.
  How often the Collector reloads certificate and key files. If omitted, the Collector doesn't
  reload the files.

* **`tls.server_name_override`**
  Type: `string`. Optional.
  Server name to verify in the certificates of masters, agents, and Aurora schedulers instead of
  the endpoint hostname. Use this setting when the certificate name differs from the endpoint
  hostname.

* **`tls.tpm`**
  Type: `object`. Optional.
  Trusted platform module settings for the TLS private key.

* **`tls.tpm.auth`**
  Type: `string`. Optional.
  Authorization value for the trusted platform module key.

* **`tls.tpm.enabled`**
  Type: `boolean`. Optional.
  Whether to use a trusted platform module for the TLS private key. If true, the Collector loads
  the key from the configured device or socket. If false, the Collector uses the configured key
  file or in-memory key.

* **`tls.tpm.owner_auth`**
  Type: `string`. Optional.
  Owner authorization value for the trusted platform module.

* **`tls.tpm.path`**
  Type: `string`. Optional.
  Path to the trusted platform module device or Unix domain socket, such as `/dev/tpm0` or
  `/dev/tpmrm0`.

* **`master`**
  Type: `object`. Optional.
  Settings for collecting from Mesos masters. Every master reports its own process and host
  metrics. Only the elected leading master reports cluster-wide metrics, because a standby
  master's copies of them are stale or zero. Listing every master therefore reports one copy of
  the cluster-wide metrics.

* **`master.enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to collect from Mesos masters. If true, the Collector collects from the masters in
  `master.endpoints`. If false, the Collector doesn't collect from any master.

* **`master.endpoints`**
  Type: `array of object`. Optional. Default: `[]`.
  Masters to collect from. Each entry is the IP address or hostname of one master, as the
  Collector reaches it, and the port of the master's HTTP API, which is `5050` by default. Don't
  include a path. The Collector reaches an entry in `host:port` form over HTTP. To reach a
  master over HTTPS, give the full URL, such as `https://mesos-master-1.example.com:5050`. List
  every master so that cluster-wide metrics continue after a new leader is elected.

* **`master.endpoints[].endpoint`**
  Type: `string`. Required.
  Master endpoint as either `host:port` or a full HTTP or HTTPS URL without a path. For
  `host:port`, the Collector uses HTTP.

* **`master.cluster_name`**
  Type: `string`. Optional. Minimum length: `1`.
  Value of the `mesos.cluster` attribute on every metric from a master. Required when
  `master.endpoints` lists a master. The Collector doesn't read the cluster name from Mesos, so
  set the same value for every master and agent in one cluster.

* **`master.frameworks`**
  Type: `array of string`. Optional. Default: `[]`.
  Exact names of the frameworks for which the elected master reports per-framework task counts
  and allocated resources. Frameworks that share a name are reported together. When empty, the
  Collector reports no per-framework metrics unless `all_frameworks` is `true`. Each name adds
  time series.

* **`master.all_frameworks`**
  Type: `boolean`. Optional. Default: `false`.
  Whether to report per-framework metrics for every framework registered with the elected
  master. If true, `frameworks` must be empty, and each distinct framework name adds four time
  series. Frameworks that share a name are reported together, but a cluster that registers a
  framework under a new name for every job adds time series with each job. If false, the
  Collector reports per-framework metrics only for the names in `frameworks`.

* **`master.max_frameworks`**
  Type: `integer`. Optional. Default: `1000`. Minimum: `1`.
  Maximum number of distinct framework names that one collection reports when `all_frameworks`
  is `true`. When the elected master reports more, the Collector reports no per-framework
  metrics from that collection and reports `cxdot.integration.target.health` as `0` for the
  master. The collection's other metrics are still reported.

* **`master.roles`**
  Type: `array of string`. Optional. Default: `[]`.
  Exact names of the roles for which the elected master reports per-role framework counts,
  weights, and allocated resources. When empty, the Collector reports no per-role metrics and
  doesn't request role information unless `all_roles` is `true`. Each name adds time series.

* **`master.all_roles`**
  Type: `boolean`. Optional. Default: `false`.
  Whether to report per-role metrics for every role that the elected master lists, including the
  default role `*`. If true, `roles` must be empty, and each role adds up to five time series.
  If false, the Collector reports per-role metrics only for the names in `roles`.

* **`master.max_roles`**
  Type: `integer`. Optional. Default: `1000`. Minimum: `1`.
  Maximum number of roles that one collection reports when `all_roles` is `true`. When the
  elected master lists more, the Collector reports no per-role metrics from that collection and
  reports `cxdot.integration.target.health` as `0` for the master. The collection's other
  metrics are still reported.

* **`agent`**
  Type: `object`. Optional.
  Settings for collecting from Mesos agents.

* **`agent.enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to collect from Mesos agents. If true, the Collector collects from the agents in
  `agent.endpoints`. If false, the Collector doesn't collect from any agent.

* **`agent.endpoints`**
  Type: `array of object`. Optional. Default: `[]`.
  Agents to collect from. Each entry is the IP address or hostname of one agent, as the
  Collector reaches it, and the port of the agent's HTTP API, which is `5051` by default. Don't
  include a path. Entries take the same `host:port` and URL forms as `master.endpoints`.

* **`agent.endpoints[].endpoint`**
  Type: `string`. Required.
  Agent endpoint as either `host:port` or a full HTTP or HTTPS URL without a path. For
  `host:port`, the Collector uses HTTP.

* **`agent.cluster_name`**
  Type: `string`. Optional. Minimum length: `1`.
  Value of the `mesos.cluster` attribute on every metric from an agent. Required when
  `agent.endpoints` lists an agent. The Collector doesn't read the cluster name from Mesos, so
  set the same value for every master and agent in one cluster.

* **`agent.tasks`**
  Type: `array of string`. Optional. Default: `[]`.
  Task name fragments that select tasks for per-task metrics. For each active task on an agent
  whose name contains a fragment, ignoring case, the Collector reports the task's allocated
  CPUs, memory, and disk, labeled with the task name, task ID, and framework ID. A nonempty list
  adds a request for the agent's full state to every collection. When empty, the Collector
  reports no per-task metrics. A blank fragment is rejected because it matches every task.

* **`agent.max_tasks`**
  Type: `integer`. Optional. Default: `1000`. Minimum: `1`.
  Maximum number of active tasks that `agent.tasks` can select on one agent. When the fragments
  match more tasks, the Collector reports no per-task metrics for that agent and reports
  `cxdot.integration.target.health` as `0` for it.

* **`aurora`**
  Type: `object`. Optional.
  Settings for collecting from Apache Aurora schedulers. Every scheduler reports its own process
  metrics. Only the leading scheduler reports cluster metrics, such as task counts, resource
  offers, preempted tasks, and cron activity, so listing every scheduler reports one copy of
  them.

* **`aurora.enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to collect from Apache Aurora schedulers. If true, the Collector collects from the
  schedulers in `aurora.endpoints`. If false, the Collector doesn't collect from any scheduler.

* **`aurora.endpoints`**
  Type: `array of object`. Optional. Default: `[]`.
  Aurora schedulers to collect from. Each entry is the IP address or hostname of one scheduler,
  as the Collector reaches it, and the port of its HTTP server, which its `-http_port` flag
  sets. Don't include a path. Entries take the same `host:port` and URL forms as
  `master.endpoints`. List every scheduler so that cluster metrics continue after a new leader
  is elected.

* **`aurora.endpoints[].endpoint`**
  Type: `string`. Required.
  Aurora scheduler endpoint as either `host:port` or a full HTTP or HTTPS URL without a path.
  For `host:port`, the Collector uses HTTP.

* **`aurora.cluster_name`**
  Type: `string`. Optional. Minimum length: `1`.
  Value of the `mesos.cluster` attribute on every metric from an Aurora scheduler. Required when
  `aurora.endpoints` lists a scheduler. Aurora doesn't report the Mesos cluster that it runs on,
  so set the `master.cluster_name` of that cluster to select scheduler and cluster metrics
  together.


## Related topics

- [Apache Spark](/ingest/xcor/integrations/collector/spark.md)
- [CXDOT Collector integrations](/ingest/xcor/integrations/collector.md)
- [Apache](/ingest/xcor/integrations/collector/apache.md)
- [Apache Tomcat](/ingest/xcor/integrations/collector/tomcat.md)
- [Kafka](/ingest/xcor/integrations/collector/kafka.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.