Apache Cassandra

The Apache Cassandra integration requires CXDOT Collector 1.13.0 or greater. Apache Cassandra is an open source distributed NoSQL database. Use the Apache Cassandra integration with the CXDOT Collector to collect node, table, and Java Virtual Machine (JVM) runtime metrics from Cassandra nodes running in your environment. The Apache Cassandra integration supports Apache Cassandra 3.0.29 or greater.

​

Supported telemetry types

The Apache Cassandra integration supports these telemetry types:

​

Prerequisites

The Apache Cassandra integration has the following prerequisites:

• Enable remote Java Management Extensions (JMX) on each Cassandra node. By default, Cassandra accepts JMX connections only from the machine it runs on. Set the LOCAL_JMX environment variable to no before Cassandra starts, or set LOCAL_JMX=no in cassandra-env.sh. Cassandra then accepts remote JMX connections on port 7199 and requires authentication for them. For more information, see JMX access in the Cassandra documentation.
• Create a JMX user for the CXDOT Collector. Cassandra reads JMX users from /etc/cassandra/jmxremote.password, one user name and password per line. Make the file readable only by the user that runs Cassandra.
• Give the JMX user for the CXDOT Collector read-only access, since the integration only reads metrics. Name an access file with the com.sun.management.jmxremote.access.file system property, and list the user in it with readonly access.
• In production, encrypt JMX connections with Transport Layer Security (TLS). Set the com.sun.management.jmxremote.ssl and com.sun.management.jmxremote.registry.ssl system properties to true, and point the javax.net.ssl.keyStore system property at the file that holds the node’s certificate and private key. For more information, see JMX with SSL in the Cassandra documentation. For example, the following container environment variables enable remote JMX, an access file, and TLS for the official Cassandra image. The variables expect the access file and the certificate file at the paths they name, and the certificate file’s password in the cassandra-jmx-tls Secret:
  env:
    - name: LOCAL_JMX
      value: "no"
    - name: KEYSTORE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: cassandra-jmx-tls
          key: keystore-password
    - name: JVM_EXTRA_OPTS
      value: >-
        -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access
        -Dcom.sun.management.jmxremote.ssl=true
        -Dcom.sun.management.jmxremote.registry.ssl=true
        -Djavax.net.ssl.keyStore=/etc/cassandra/tls/keystore.p12
        -Djavax.net.ssl.keyStorePassword=$(KEYSTORE_PASSWORD)
  
• Allow connections to port 7199 on each Cassandra node from the CXDOT Collector. With remote JMX turned on, Cassandra serves the JMX registry and the JMX server it returns on this one port.

​

Configure

The Apache Cassandra integration is enabled by default. To configure the Apache Cassandra integration, follow these steps:

1. Choose how the CXDOT Collector finds your Cassandra nodes:
   • On Cassandra pods, add autodiscovery annotations for Cassandra. Supply the JMX address and the JMX credentials in each annotation.
   • On Services, add the same autodiscovery annotations. The cluster Collector that wins leader election collects from each annotated Service once. Each Cassandra node reports its own metrics, so annotate a Service only when it routes to a single Cassandra node.
   For more information, see autodiscovery. Metrics from discovered pods arrive with that pod’s Kubernetes metadata attached.
2. Optional: Configure static targets instead of discovered targets, such as Cassandra nodes outside your Kubernetes cluster. For example, add the following to the Helm values file for your CXDOT Collector to read the JMX password from the cassandra-jmx Secret:
   nodeCollector:
     extraEnv:
       - name: CASSANDRA_JMX_PASSWORD
         valueFrom:
           secretKeyRef:
             name: cassandra-jmx
             key: password
   clusterCollector:
     extraEnv:
       - name: CASSANDRA_JMX_PASSWORD
         valueFrom:
           secretKeyRef:
             name: cassandra-jmx
             key: password
   config:
     integrations:
       cassandra:
         endpoints:
           - endpoint: cassandra-0.example.com:7199
           - endpoint: cassandra-1.example.com:7199
         username: cxdot
         password: "${env:CASSANDRA_JMX_PASSWORD}"
   
   When you configure static targets, this integration instance collects from exactly those targets instead of discovered targets. In Kubernetes, the cluster Collector that wins leader election collects from static targets. Every Collector loads the same integration settings and rejects a username without a password, so set the password variable on both the node and cluster Collectors. The username and password settings apply to static targets only.
3. Optional: Configure a second integration instance to collect from both discovered pods and static targets. For example, add the following to the Helm values file for your CXDOT Collector:
   config:
     integrations:
       cassandra: {}
       cassandra/external:
         endpoints:
           - endpoint: cassandra-0.example.com:7199
         username: cxdot
         password: "${env:CASSANDRA_JMX_PASSWORD}"
   
   The cassandra/external key defines a named instance for the static targets. The bare cassandra key continues to use autodiscovery. Set the CASSANDRA_JMX_PASSWORD environment variable on the node and cluster Collectors, as the previous step shows.
4. Optional: If your Cassandra nodes use TLS for JMX, configure the CXDOT Collector to trust their certificates. For example, add the following to the Helm values file for your CXDOT Collector:
   config:
     integrations:
       cassandra:
         registry_tls: true
         tls:
           ca_file: /etc/cxdot/cassandra-tls/ca.crt
           server_name_override: cassandra.example.com
   
   The tls settings apply to every target. The registry_tls setting applies to static targets. For targets that autodiscovery annotations define, the annotation turns on registry TLS instead. The CXDOT Collector verifies each node’s certificate against the host in that target’s endpoint, which for a discovered pod is the pod’s IP address. If your certificates name a different host, set server_name_override to a name that every node’s certificate carries. Mount the certificate authority file into each Collector that collects from Cassandra: the node Collectors for annotated pods, and the cluster Collector for static targets and annotated Services. To mount the file from a Secret, use the extraVolumes and extraVolumeMounts Helm values of nodeCollector and clusterCollector.
5. Optional: Disable the integration. For example, add the following to the Helm values file for your CXDOT Collector:
   config:
     integrations:
       cassandra:
         enabled: false
   

​

Validate

To validate the Apache Cassandra integration, follow these steps:

1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health and cxdot.integration.name=cassandra. Confirm that cxdot.integration.target.health reports 1 for each reachable Cassandra node. A value of 0 means the CXDOT Collector hit an error connecting to or collecting from that node. The cxdot.integration.target, server.address, and server.port attributes identify the node.
2. In Metrics Explorer, run the following query:
   sum by ("server.address", "server.port") ({"cassandra.endpoint.count", "cassandra.endpoint.state"="up", "cxdot.integration.name"="cassandra"})
   
   Confirm that the query returns one time series per Cassandra node, each reporting the number of nodes that node sees as up. While every node is up, the value is the number of nodes in your cluster.

​

Troubleshooting

• Metrics for a new table, or for a thread pool the node hasn’t used yet, are missing: Cassandra registers a table’s metrics when you create the table, and Cassandra 4.0 and later create some thread pools when they’re first used. The integration refreshes its list of JMX beans at each bean_detection_interval, which defaults to 10 minutes, so the missing time series appear after the next refresh.
• cassandra.table.memtable.flushed and cassandra.table.compaction.written are missing: Cassandra reports these table metrics starting with Cassandra 3.11. A Cassandra 3.0 node doesn’t report them.
• cassandra.table.memtable.space_wait.duration.p75 and cassandra.table.memtable.space_wait.duration.p95 always report 0, or are missing: Cassandra releases before 5.0 register these percentiles but never record a wait into them, so they report 0 whatever the write load. Cassandra 5.0 doesn’t report them.
• A query on one cassandra.message.type value finds no dropped messages on some Cassandra nodes: The message types depend on the Cassandra release. On Cassandra 4.0 and later, the integration reports nine message types under the names that earlier releases use, such as MUTATION and READ, and the rest under the request and response verb names that Cassandra 4.0 introduced, such as MUTATION_RSP. A sum over every message type counts all of a node’s dropped messages on every release.

​

Configuration reference

Configure one Apache Cassandra integration instance with the following settings. In Helm values, place these settings under config.integrations.cassandra. In a Collector configuration file, place them under cxdot.integrations.cassandra.

​

Optional settings

• enabled Type: boolean. Optional. Default: true. Whether to enable this Cassandra integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it.
• endpoints Type: array of object. Optional. Default: []. Static Cassandra node Java Management Extensions (JMX) targets. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains targets, this integration instance collects from exactly those targets and doesn’t automatically discover targets. Configure a second named instance to use both methods. Each endpoint can be a host:port Remote Method Invocation (RMI) registry address or a service:jmx:rmi URL.
• endpoints[].endpoint Type: string. Required. Cassandra node JMX endpoint as either a host:port Remote Method Invocation (RMI) registry address or a service:jmx:rmi URL.
• registry_tls Type: boolean. Optional. Default: false. Whether to use Transport Layer Security (TLS) for the RMI registry connection. If true, the Collector uses TLS, which Cassandra requires when it sets com.sun.management.jmxremote.registry.ssl to true. If false, the Collector uses an unencrypted registry connection. The connection to the JMX server that the registry returns uses TLS whenever Cassandra requires it, whatever this setting. For targets that autodiscovery annotations define, the annotation sets this value instead.
• advertised_host_override Type: string. Optional. Default: empty string. Host the Collector connects to for the JMX server that the RMI registry returns, instead of the host that Cassandra advertises. When empty, the Collector connects to the host in the target’s endpoint, so the java.rmi.server.hostname setting on the Cassandra node doesn’t affect the Collector.
• advertised_port_override Type: integer. Optional. Default: 0. Minimum: 0. Maximum: 65535. Port the Collector connects to for the JMX server that the RMI registry returns, instead of the port that Cassandra advertises. When set to 0, the Collector uses the advertised port.
• username Type: string. Optional. Username for JMX authentication. Configure username and password together. For targets that autodiscovery annotations define, the annotation supplies the credentials instead.
• password Type: string. Optional. Password for JMX authentication. Configure username and password together. For targets that autodiscovery annotations define, the annotation supplies the credentials instead.
• collection_interval Type: duration. Optional. Default: 15s. How often the Collector collects metrics from each Cassandra JMX endpoint. For targets that autodiscovery annotations define, the annotation sets the interval instead.
• bean_detection_interval Type: duration. Optional. Default: 10m. How often the Collector refreshes the set of available JMX beans. Cassandra registers beans for tables when they’re created and for some thread pools when they’re first used, so their metrics appear after the next refresh.
• bean_limit Type: integer. Optional. Default: 35000. Minimum: 1. Maximum number of JMX beans the Collector reads from each Cassandra node in one collection. Cassandra registers a set of beans for each table, so the number of beans grows with the number of tables. When a node has more matching beans than this limit, the Collector skips per-table beans first, so node and Java Virtual Machine (JVM) metrics keep reporting.
• target_system Type: string. Optional. Default: cassandra. Comma-delimited attribute mapping names to apply. The Cassandra mapping is enabled by default, and the JVM mapping is included automatically. Add -jvm to disable JVM metrics when at least one other mapping is configured. For targets that autodiscovery annotations define, the annotation decides whether to collect JVM metrics, and this setting doesn’t apply.
• attribute_mapping_paths Type: array of string. Optional. Default: []. Directories and files containing custom JMX attribute mappings. The Collector searches these paths before the embedded Cassandra and JVM mappings.
• tls Type: object. Optional. Default: {}. TLS certificate and client authentication settings for RMI registry and server connections.
• tls.ca_file Type: string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the JMX server certificate. If omitted, the Collector uses the system certificate authority pool.
• tls.ca_pem Type: string. Optional. PEM-encoded certificate authority certificate content used to verify the JMX server certificate.
• tls.cert_file Type: string. Optional. Path to the PEM-encoded client certificate file to present when the JMX server requires mutual TLS authentication.
• tls.cert_pem Type: string. Optional. PEM-encoded client certificate content to present when the JMX server requires mutual TLS authentication.
• tls.cipher_suites Type: array of string. Optional. TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector uses secure default cipher suites. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go.
• tls.curve_preferences Type: array of string. Optional. Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses its default curves.
• tls.include_system_ca_certs_pool Type: boolean. Optional. Whether to add certificates from the system certificate authority pool to the configured certificate authorities. If true, the Collector trusts both sources. If false, the Collector trusts only the configured certificate authorities.
• tls.insecure Type: boolean. Optional. Whether to disable transport security. This setting must be false. Use registry_tls to control transport security for the RMI registry.
• tls.insecure_skip_verify Type: boolean. Optional. Whether to skip verification of the JMX server certificate. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the certificate.
• tls.key_file Type: string. Optional. Path to the PEM-encoded client private key file used with cert_file for mutual TLS authentication.
• tls.key_pem Type: string. Optional. PEM-encoded client private key content used with cert_pem for mutual TLS authentication.
• tls.max_version Type: string. Optional. Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the highest TLS version it supports.
• tls.min_version Type: string. Optional. Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS 1.2.
• tls.reload_interval Type: duration. Optional. How often the Collector reloads certificate and key files. If omitted, the Collector doesn’t reload the files.
• tls.server_name_override Type: string. Optional. Server name to verify in the JMX server certificate instead of the host in the target’s endpoint. Use this setting when the certificate names a different host, such as for a discovered pod, whose endpoint host is the pod’s IP address.
• tls.tpm Type: object. Optional. Trusted platform module settings for the TLS private key.
• tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
• tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
• tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
• tls.tpm.path Type: string. Optional. Path to the trusted platform module device or Unix domain socket, such as /dev/tpm0 or /dev/tpmrm0.