Apache HBase

The Apache HBase integration requires CXDOT Collector 1.17.0 or greater. Apache HBase is an open source distributed database that stores large tables on Apache Hadoop. Use the Apache HBase integration with the CXDOT Collector to collect master, RegionServer, and Java Virtual Machine (JVM) runtime metrics from HBase clusters running in your environment. The Apache HBase integration supports Apache HBase 2.5 or greater.

​

Supported telemetry types

The Apache HBase integration supports these telemetry types:

​

Prerequisites

The Apache HBase integration has the following prerequisites:

• Enable remote Java Management Extensions (JMX) monitoring on the master and on each RegionServer. HBase disables remote JMX by default. Each role runs its own JVM, so set the com.sun.management.jmxremote system properties in HBASE_MASTER_OPTS and in HBASE_REGIONSERVER_OPTS in hbase-env.sh. The commented example in hbase-env.sh uses port 10101 for the master and port 10102 for the RegionServer. For example, the following lines enable remote JMX on those ports without authentication or Transport Layer Security (TLS):
  export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Djava.rmi.server.hostname=SERVER_ADDRESS"
  export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE \
    -Dcom.sun.management.jmxremote.port=10101 \
    -Dcom.sun.management.jmxremote.rmi.port=10101"
  export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE \
    -Dcom.sun.management.jmxremote.port=10102 \
    -Dcom.sun.management.jmxremote.rmi.port=10102"
  
  Replace SERVER_ADDRESS with the server’s address as the CXDOT Collector reaches it. Setting com.sun.management.jmxremote.rmi.port to the same value as com.sun.management.jmxremote.port serves the JMX registry and the connection it returns on one port, so each process needs one open port for JMX. For more information, see JMX in the HBase documentation.
• Make the JMX port and the address each process advertises for Remote Method Invocation (RMI) connections reachable from the CXDOT Collector. Set java.rmi.server.hostname to an address the CXDOT Collector can resolve, or configure the integration’s advertised host and port overrides when that address isn’t reachable.
• Restrict JMX access to trusted clients. In production, enable JMX password authentication and TLS on the master and RegionServers, and configure the integration’s username, password, registry_tls, and tls settings to match. The JMX section of the HBase documentation shows the HBASE_JMX_BASE settings for both.

​

Configure

The Apache HBase integration is enabled by default. You can configure targets either by annotating master and RegionServer pods in your Kubernetes cluster or by listing static endpoints in your Helm values. The integration collects from masters and RegionServers separately: a CXDOT integration annotation or a static endpoint list names the role it configures, and the metrics from each target carry hbase.role set to master or regionserver. To configure the integration, follow these steps:

1. To collect from HBase pods in Kubernetes, add a CXDOT integration annotation to the pod template of the master workload and to the pod template of the RegionServer workload. The annotation key must end with the container name, its value names the role under hbase, and it can reference JMX credentials stored in a Kubernetes Secret in the same namespace. For example, add the following annotation to your HBase master workload:
   spec:
     template:
       metadata:
         annotations:
           integrations.cxdot.paloaltonetworks.com/master: |
             schema_version: v1.17
             hbase:
               master:
                 endpoints:
                   - endpoint: ${target.host}:${target.port.jmx}
               username: cxdot
               password: ${k8s_secret:hbase-jmx/password}
       spec:
         containers:
           - name: master
             image: example.com/hbase:2.6.7
             ports:
               - name: jmx
                 containerPort: 10101
   
   Add the following annotation to your RegionServer workload:
   spec:
     template:
       metadata:
         annotations:
           integrations.cxdot.paloaltonetworks.com/regionserver: |
             schema_version: v1.17
             hbase:
               regionserver:
                 endpoints:
                   - endpoint: ${target.host}:${target.port.jmx}
               username: cxdot
               password: ${k8s_secret:hbase-jmx/password}
       spec:
         containers:
           - name: regionserver
             image: example.com/hbase:2.6.7
             ports:
               - name: jmx
                 containerPort: 10102
   
   The Collector replaces ${target.host} with the pod IP address, ${target.port.jmx} with the container port named jmx, and ${k8s_secret:hbase-jmx/password} with the password key of the hbase-jmx Secret. Each master and RegionServer pod becomes its own target. To allow the Collector to read the Secret, add the pod’s namespace to apiServer.metadataServer.secretAllowlist in the Helm values for your CXDOT Collector.
2. Optional: To collect from HBase servers outside your Kubernetes cluster, configure static targets for each role. For example, add the following to the Helm values file for your CXDOT Collector:
   collection:
     cxdot:
       integrations:
         hbase:
           master:
             endpoints:
               - endpoint: hbase-master.example.com:10101
           regionserver:
             endpoints:
               - endpoint: hbase-rs-0.example.com:10102
               - endpoint: hbase-rs-1.example.com:10102
   
   When you configure static targets for a role, this integration instance collects from exactly those targets for that role instead of discovered targets. Each role’s list stands on its own: listing masters under master.endpoints leaves RegionServer discovery on. In cluster mode, the cluster Collector that wins leader election scrapes static targets.
3. Optional: Configure a second integration instance to collect from both annotated pods and static targets. For example, add the following to the Helm values file for your CXDOT Collector:
   collection:
     cxdot:
       integrations:
         hbase: {}
         hbase/external:
           master:
             endpoints:
               - endpoint: hbase-master.example.com:10101
           regionserver:
             endpoints:
               - endpoint: hbase-rs-0.example.com:10102
   
   The hbase/external key defines a named instance for the static targets. The bare hbase key continues to discover pods.
4. Optional: If the master and RegionServers require JMX authentication or TLS, configure the matching settings for static targets, and provide the password through your deployment’s secret management. For example, the following Helm values reference the HBASE_JMX_PASSWORD environment variable and a mounted certificate authority file:
   collection:
     cxdot:
       integrations:
         hbase:
           master:
             endpoints:
               - endpoint: hbase-master.example.com:10101
           regionserver:
             endpoints:
               - endpoint: hbase-rs-0.example.com:10102
           username: cxdot
           password: "${env:HBASE_JMX_PASSWORD}"
           registry_tls: true
           tls:
             ca_file: /etc/cxdot/hbase-tls/ca.crt
   
   The username, password, registry_tls, and tls settings apply to every master and RegionServer that the instance collects from. Every Collector loads the same integration settings and rejects a username without a password, so set the password variable on both the node and cluster Collectors with the nodeCollector.extraEnv and clusterCollector.extraEnv Helm values. Mount the certificate authority file into the cluster Collector, which collects from static targets, with the clusterCollector.extraVolumes and clusterCollector.extraVolumeMounts Helm values. For annotated pods, set these settings in the CXDOT integration annotation instead, and supply the certificate authority from a Kubernetes Secret with tls.ca_pem: ${k8s_secret:hbase-tls/ca.crt}. The Collector rejects an annotation that sets tls.ca_file, or any other setting that names a file, unless the file is under collection.cxdot.integrations.discovery.kubernetes.annotation_restrictions.allowed_collector_file_paths in your Helm values, which allows no files by default.
5. Optional: Disable the integration, or one of its roles. For example, add the following to the Helm values file for your CXDOT Collector to stop collecting from masters and RegionServers alike:
   collection:
     cxdot:
       integrations:
         hbase:
           enabled: false
   
   To keep collecting from RegionServers and stop collecting from masters, set master.enabled to false instead.

Metrics from each discovered pod include that pod’s Kubernetes metadata. For more information, see autodiscovery.

​

Validate

To validate the Apache HBase integration, follow these steps:

1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health and cxdot.integration.name=hbase. Confirm that cxdot.integration.target.health reports 1 for each master and each RegionServer. A value of 0 means the CXDOT Collector hit an error connecting to or collecting from that target. The hbase.role, cxdot.integration.target, server.address, and server.port attributes identify the target.
2. In Metrics Explorer, run the following query:
   sum by ("server.address", "server.port") ({"hbase.master.region_server.count", "hbase.region_server.state"="live"})
   
   Confirm that the query returns one time series per master, each reporting the number of RegionServers that master sees as live.
3. In Metrics Explorer, run the following query while the cluster serves reads and writes:
   sum by ("server.address", "server.port", "hbase.request.type") (rate({"hbase.region_server.request.count"}[$__rate_interval]))
   
   Confirm that the query returns a read and a write time series for each RegionServer.

​

Troubleshooting

• hbase.wal.* time series appear for a master: The master keeps a write-ahead log for its procedure store and reports it through the same metrics as a RegionServer’s log. Filter on hbase.role="regionserver" to see the logs that record table writes.
• A duration statistic such as hbase.region_server.operation.duration.p99 reads 0 while the cluster serves requests: HBase computes its minimum, maximum, mean, median, and p99 statistics over the operations since its last metrics snapshot, which it takes every 10 seconds by default, as *.period in hadoop-metrics2-hbase.properties sets. An interval without an operation of that kind reads 0, and an operation faster than a millisecond rounds down to 0. The matching .count metrics are cumulative, so query a rate of them for throughput.

For more information about CXDOT Collector integrations, see the integrations catalog.

​

Configuration reference

Configure one Apache HBase integration instance with the following settings. In Helm values, place these settings under collection.cxdot.integrations.hbase. In a Collector configuration file, place them under cxdot.integrations.hbase.

​

Optional settings

• enabled Type: boolean. Optional. Default: true. Whether to enable this Apache HBase integration instance. If true, the Collector collects from the masters and RegionServers the instance configures or discovers. If false, the Collector doesn’t run the instance.
• master Type: object. Optional. Settings for collecting from HBase masters. Masters are discovered from workload annotations, or listed here as endpoints.
• master.enabled Type: boolean. Optional. Default: true. Whether this instance collects from HBase masters. If true, the Collector collects from the masters it discovers or the targets in master.endpoints. If false, the Collector doesn’t collect from masters, and RegionServer collection isn’t affected.
• master.endpoints Type: array of object. Optional. Default: []. HBase master Java Management Extensions (JMX) targets. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains targets, this integration instance collects from exactly those targets and doesn’t automatically discover masters. Configure a second named instance to use both methods. Each endpoint can be a host:port Remote Method Invocation (RMI) registry address or a service:jmx:rmi URL.
• master.endpoints[].endpoint Type: string. Required. HBase master JMX endpoint as either a host:port RMI registry address or a service:jmx:rmi URL.
• regionserver Type: object. Optional. Settings for collecting from HBase RegionServers. RegionServers are discovered from workload annotations, or listed here as endpoints.
• regionserver.enabled Type: boolean. Optional. Default: true. Whether this instance collects from HBase RegionServers. If true, the Collector collects from the RegionServers it discovers or the targets in regionserver.endpoints. If false, the Collector doesn’t collect from RegionServers, and master collection isn’t affected.
• regionserver.endpoints Type: array of object. Optional. Default: []. HBase RegionServer Java Management Extensions (JMX) targets. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains targets, this integration instance collects from exactly those targets and doesn’t automatically discover RegionServers. Configure a second named instance to use both methods. Each endpoint can be a host:port Remote Method Invocation (RMI) registry address or a service:jmx:rmi URL.
• regionserver.endpoints[].endpoint Type: string. Required. HBase RegionServer JMX endpoint as either a host:port RMI registry address or a service:jmx:rmi URL.
• registry_tls Type: boolean. Optional. Default: false. Whether to use Transport Layer Security (TLS) for the RMI registry connection. If true, the Collector uses TLS. If false, the Collector uses an unencrypted registry connection.
• advertised_host_override Type: string. Optional. Default: empty string. Host the Collector uses for RMI server connections instead of the host advertised by returned RMI stubs. When empty, the Collector uses the registry endpoint host.
• advertised_port_override Type: integer. Optional. Default: 0. Minimum: 0. Maximum: 65535. Port the Collector uses for RMI server connections instead of the port advertised by returned RMI stubs. When set to 0, the Collector uses the advertised port.
• username Type: string. Optional. Username for JMX authentication, used for every master and RegionServer. Configure username and password together.
• password Type: string. Optional. Password for JMX authentication, used for every master and RegionServer. Configure username and password together.
• collection_interval Type: duration. Optional. Default: 15s. How often the Collector collects metrics from each HBase JMX endpoint.
• bean_detection_interval Type: duration. Optional. Default: 10m. How often the Collector refreshes the set of available JMX beans.
• bean_limit Type: integer. Optional. Default: 35000. Minimum: 1. Maximum number of JMX bean matches collected from each endpoint. When the number of matches exceeds this value, the integration drops beans according to the attribute mapping priority.
• target_system Type: string. Optional. Default: hbase. Comma-delimited attribute mapping names to apply. The HBase mapping, which covers both masters and RegionServers, is enabled by default, and the Java Virtual Machine (JVM) mapping is included automatically. Add -jvm to disable JVM metrics when at least one other mapping is configured.
• attribute_mapping_paths Type: array of string. Optional. Default: []. Directories and files containing custom JMX attribute mappings. The Collector searches these paths before the embedded HBase and JVM mappings.
• tls Type: object. Optional. Default: {}. TLS certificate and client authentication settings for RMI registry and server connections.
• tls.ca_file Type: string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the JMX server certificate. If omitted, the Collector uses the system certificate authority pool.
• tls.ca_pem Type: string. Optional. PEM-encoded certificate authority certificate content used to verify the JMX server certificate.
• tls.cert_file Type: string. Optional. Path to the PEM-encoded client certificate file to present when the JMX server requires mutual TLS authentication.
• tls.cert_pem Type: string. Optional. PEM-encoded client certificate content to present when the JMX server requires mutual TLS authentication.
• tls.cipher_suites Type: array of string. Optional. TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector uses secure default cipher suites. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go.
• tls.curve_preferences Type: array of string. Optional. Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses its default curves.
• tls.include_system_ca_certs_pool Type: boolean. Optional. Whether to add certificates from the system certificate authority pool to the configured certificate authorities. If true, the Collector trusts both sources. If false, the Collector trusts only the configured certificate authorities.
• tls.insecure Type: boolean. Optional. Whether to disable transport security. This setting must be false. Use registry_tls to control transport security for the RMI registry.
• tls.insecure_skip_verify Type: boolean. Optional. Whether to skip verification of the JMX server certificate. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the certificate.
• tls.key_file Type: string. Optional. Path to the PEM-encoded client private key file used with cert_file for mutual TLS authentication.
• tls.key_pem Type: string. Optional. PEM-encoded client private key content used with cert_pem for mutual TLS authentication.
• tls.max_version Type: string. Optional. Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the highest TLS version it supports.
• tls.min_version Type: string. Optional. Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS 1.2.
• tls.reload_interval Type: duration. Optional. How often the Collector reloads certificate and key files. If omitted, the Collector doesn’t reload the files.
• tls.server_name_override Type: string. Optional. Server name to verify in the JMX certificate instead of the endpoint hostname. Use this setting when the certificate name differs from the endpoint hostname.
• tls.tpm Type: object. Optional. Trusted platform module settings for the TLS private key.
• tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
• tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
• tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
• tls.tpm.path Type: string. Optional. Path to the trusted platform module device or Unix domain socket, such as /dev/tpm0 or /dev/tpmrm0.