Linux

The Linux integration requires CXDOT Collector 1.7.0 or greater. Linux is an open source operating system kernel used by server distributions such as Debian, Ubuntu, and Red Hat Enterprise Linux. Use the Linux integration to collect systemd journal records, systemd unit states, and forwarded syslog records from Linux hosts. The journald and systemd capabilities support Linux distributions running systemd 239 or greater. The syslog capability doesn’t require systemd.

​

Supported telemetry types

The Linux integration supports these telemetry types:

​

Prerequisites

The Linux integration has the following prerequisites:

• Run the Collector directly on each Linux host with mode: host.
• The journald capability is enabled by default. When enabled, the host must have journalctl, and the account that runs the Collector must be able to read the system journal. The root account and members of the systemd-journal group normally have access. Some distributions also grant access through groups such as adm or wheel. Run id -nG ACCOUNT to list an account’s groups, and use getfacl to inspect journal access control lists. To run the Linux integration without journal access, set cxdot.integrations.linux.journald.enabled to false. The systemd and syslog capabilities don’t require journal access.
• The systemd capability, enabled by default, requires the host to run systemd as its service manager and make the D-Bus system bus available. Run systemctl --version to confirm the systemd version.
• Disable a capability if the host doesn’t meet its requirements. A host that enables only the syslog capability doesn’t require systemd or journalctl.

​

Configure

The Linux integration runs by default on Linux hosts. It collects journal records of priority info and more severe from the whole system journal, and the active state of each matching, currently loaded systemd service unit. Journal collection starts at the end of the journal when the Collector starts, so records already in the journal aren’t collected. Forwarded syslog collection is off by default. To configure the Linux integration, follow these steps:

1. Run the Collector in host mode and configure a logs exporter. The default configuration requires no linux block. If you use tenant egress, enable logs as shown in the following Collector configuration:
   cxdot:
     mode: host
     egress:
       tenant:
         org_name: MY_ORG
         api_token: ${env:CHRONOSPHERE_API_TOKEN}
         signals:
           logs:
             enabled: true
   
2. Optional: Keep the journal read position across Collector restarts. By default the Collector resumes at the end of the journal after a restart, so records written while it was down aren’t collected. Enable checkpoint storage and name a directory that already exists and that the account running the Collector can write to. For example, add the following to the Collector configuration file:
   cxdot:
     mode: host
     storage:
       checkpoints:
         enabled: true
         directory: /var/lib/cxdot-collector
   
3. Optional: Accept syslog forwarded by the host’s syslog daemon or by network devices. By default, the listener accepts UDP on 127.0.0.1:5514 and parses RFC 5424 messages. For example, enable the listener in the Collector configuration:
   cxdot:
     mode: host
     integrations:
       linux:
         syslog:
           enabled: true
   
   Then configure the sender to forward to the listener. For example, for rsyslog, create /etc/rsyslog.d/50-cxdot.conf with the following content and restart rsyslog:
   if $programname != 'cxdot-collector' then {
     action(type="omfwd"
            target="127.0.0.1"
            port="5514"
            protocol="udp"
            template="RSYSLOG_SyslogProtocol23Format")
   }
   
   The protocol-23 template is broadly available across versions and is accepted by the listener’s RFC 5424 parser. The condition on the program name excludes log records that the Collector writes. Without it, the error the Collector writes when an export fails is forwarded to the listener and exported again, which fails again. The journald capability drops records that the Collector writes; the syslog capability doesn’t, so the sender must exclude them. If the host’s syslog daemon forwards other records that the journald capability also collects, the two capabilities produce duplicate records. Narrow the sender’s selection, or disable journald only when you intend syslog to replace journal collection.
4. Optional: Change the listener’s address, transport, or message format. Set udp or tcp, not both, because the listener uses one transport. Set protocol to the format the sender emits. For example, add the following to the Collector configuration file to accept RFC 3164 messages from other hosts over TCP:
   cxdot:
     mode: host
     integrations:
       linux:
         syslog:
           enabled: true
           tcp:
             listen_address: "0.0.0.0:6514"
           protocol: rfc3164
   
   With the default Linux privileged-port threshold, a listen address on a port lower than 1024 requires the Collector to run as root or with the CAP_NET_BIND_SERVICE Linux capability. RFC 3164 timestamps carry no time zone. If the senders’ clocks show a zone other than UTC, set location to that zone so the Collector reads their timestamps correctly.
5. Optional: Secure the TCP listener, or match the sender’s framing. A listener that accepts connections from other hosts carries log records across the network, so configure TLS for it with a certificate and key the Collector can read, and set client_ca_file to accept only senders that present a certificate signed by that authority. If the sender frames each message with its length in octets, set enable_octet_counting: true. For example, add the following to the Collector configuration file:
   cxdot:
     mode: host
     integrations:
       linux:
         syslog:
           enabled: true
           tcp:
             listen_address: "0.0.0.0:6514"
             enable_octet_counting: true
             tls:
               cert_file: /etc/cxdot-collector/syslog.crt
               key_file: /etc/cxdot-collector/syslog.key
               client_ca_file: /etc/cxdot-collector/senders-ca.crt
           location: America/New_York
   
   Then configure the senders to connect with TLS and to use the same framing. The udp listener has no TLS or framing settings.
6. Optional: Narrow journal collection. Set priority to a more severe level to collect fewer records, or set units to collect from specific systemd units only. For example, add the following to the Collector configuration file:
   cxdot:
     mode: host
     integrations:
       linux:
         journald:
           priority: warning
           units:
             - sshd.service
             - nginx.service
   
7. Optional: Select the currently loaded systemd units to report state for. The units list replaces the default *.service pattern, so include that pattern to continue reporting services. Each unit produces one time series per possible state, so a wide pattern on a host with many units produces many time series. For example, add the following to the Collector configuration file to report timers and services:
   cxdot:
     mode: host
     integrations:
       linux:
         systemd:
           units:
             - "*.service"
             - "*.timer"
   
   The Collector also reports systemd.service.restarts, the number of times systemd has automatically restarted each service. Set service_restarts: false to stop collecting it. A configuration with a linux block that declares a schema_version before 1.8 keeps this metric off until you raise the version. Set service_cpu_time: true to also collect CPU time per service. This setting requires cgroup v2 on the host.
8. Optional: Disable the integration, or one of its capabilities, to stop collecting from a host. For example, add the following to the Collector configuration file to stop collecting unit states while continuing to collect journal records:
   cxdot:
     mode: host
     integrations:
       linux:
         systemd:
           enabled: false
   
   Set enabled: false directly under linux to disable the whole integration.

​

Validate

To validate the Linux integration, follow these steps:

1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health cxdot.integration.name=linux. Confirm that each enabled capability reports 1. Use the cxdot.integration.check and cxdot.integration.target attributes to identify the capability and the host for each result.
2. Optional: If the systemd capability is enabled, in Metrics Explorer, run the following query:
   {"systemd.unit.state", "systemd.unit.name"="systemd-journald.service",
    "systemd.unit.active_state"="active"}
   
   Confirm that the query returns a time series with the value 1 for each Linux host. If you narrowed units, query a unit that the list includes.
3. Optional: If the journald capability is enabled, on a Linux host, write a test record to the journal:
   logger -t cxdot-test "Linux integration test"
   
   In Logs Explorer, filter for ["cxdot.integration.name"] = "linux" AND "Linux integration test". Confirm that the record appears with its SYSLOG_IDENTIFIER field set to cxdot-test.
4. Optional: If you enabled the syslog listener, send a test message to it from the host. Adjust the address, port, and transport if you changed the listener:
   logger -n 127.0.0.1 -P 5514 -d -t cxdot-test "Linux syslog test"
   
   In Logs Explorer, filter for ["cxdot.integration.name"] = "linux" AND "Linux syslog test". Confirm that the record appears.

​

Troubleshooting

• The journald capability reports health 0: The account that runs the Collector can’t read the system journal, journalctl isn’t on its PATH, or the host has no system journal. Run journalctl --system --lines=1 as that account. A message that mentions insufficient permissions means the account lacks access to one or more system journal files. Check its groups and the journal access control lists.
• The journald capability reports health 1, but no records appear: The Collector collects only records written after it started, and only records of priority info and more severe unless priority says otherwise. Also confirm that a logs exporter is configured. If you use tenant egress, enable cxdot.egress.tenant.signals.logs.
• No records from the service that runs the Collector appear through journald: The journald capability drops records written by the cxdot-collector.service unit and the cxdot-collector process. A syslog forwarder can reintroduce those records unless its selector excludes them.
• The systemd capability reports health 0: The Collector can’t reach the system bus. The host doesn’t run systemd, or the Collector runs in a container without the system bus socket. Disable the systemd capability on such hosts.
• The syslog capability reports health 0: Another process holds the listen address, usually the host’s own syslog daemon or a second Collector, or the port is lower than 1024 and the Collector lacks the privilege to bind it. Change listen_address or grant the privilege, then restart the Collector.
• Forwarded syslog records arrive as a single opaque line: The sender’s format doesn’t match protocol. The default rsyslog forwarding template and syslog-ng’s network() destination emit RFC 3164, so set protocol: rfc3164 for those senders.
• Records forwarded over TCP run together or arrive cut mid-message: The sender’s framing doesn’t match the listener’s. A sender that prefixes each message with its length needs enable_octet_counting: true.
• RFC 3164 records carry a timestamp that’s off by a whole number of hours: The senders’ clocks show a time zone other than UTC. Set location to that zone.
• The syslog capability reports health 0 and the Collector logs a failed-to-load-location error at startup: The location value isn’t a valid time zone name. Use an Internet Assigned Numbers Authority (IANA) name such as America/New_York. The Collector reads zone names from the host’s time zone database, typically the tzdata package under /usr/share/zoneinfo, so a valid name also fails on a host that lacks this data. Install the package or use UTC.
• systemd.service.restarts is missing and the Collector logs a collection error: The host runs a systemd release older than 235, which the integration doesn’t support. Set service_restarts: false on such hosts to stop the error.
• service_cpu_time is enabled, but systemd.service.cpu.time is missing and the Collector logs a collection error: The host uses cgroup v1 or a hybrid hierarchy, and per-service CPU time requires cgroup v2. Run stat -fc %T /sys/fs/cgroup, which prints cgroup2fs on a host that supports it.

​

Configuration reference

Configure one Linux integration instance with the following settings. In a Collector configuration file, place them under cxdot.integrations.linux.

​

Optional settings

• enabled Type: boolean. Optional. Default: true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it.
• collection_interval Type: duration. Optional. Default: 60s. How often the Collector collects systemd unit state and checks whether each enabled capability can reach what it reads. Journal and syslog records arrive as they’re written, so this setting doesn’t pace log collection.
• timeout Type: duration. Optional. Default: 60s. Time limit for each unit-state collection and each health check. The journal health check runs journalctl, and this is the deadline on that command.
• journald Type: object. Optional. Settings for collecting records from the systemd journal on each host, and for reporting whether the Collector can read that journal.
• journald.enabled Type: boolean. Optional. Default: true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it.
• journald.priority Type: string. Optional. Default: info. Lowest journal priority to collect. The value follows the journalctl convention of this level and more severe. This setting controls how many journal records the Collector sends, so watch the volume after you configure a logs exporter. info is the level at which sshd, sudo, cron, and most daemons write records such as logins. Set a more severe level, such as warning, when your hosts produce more volume than you want to collect. Allowed values: emerg, alert, crit, err, warning, notice, info, debug.
• journald.units Type: array of string. Optional. Minimum items: 1. Names of systemd units to collect journal records from, for example sshd.service. When unset, the Collector collects the whole journal. Use priority to control volume, and this setting to limit collection to the units you need. Whatever this setting contains, the Collector drops its own records, matched by the cxdot-collector.service unit or the cxdot-collector process name. When systemd runs the Collector, its log output lands in the journal, so without this exclusion an export failure would be read back as a record and sent through the pipeline that’s failing.
• journald.drop_fields Type: regex. Optional. Default: ^(__|_AUDIT_|_(BOOT|MACHINE)_ID|_CAP_EFFECTIVE|_CMDLINE|_EXE|_SELINUX_CONTEXT|_STREAM_ID|_RUNTIME_SCOPE|_SOURCE_.*TIMESTAMP|_SYSTEMD_(CGROUP|SLICE|USER_SLICE|OWNER_UID|SESSION|INVOCATION_ID)|_HOSTNAME|SYSLOG_(PID|TIMESTAMP|RAW)). Minimum length: 1. Regular expression matched against each field name in a journal record’s body. Every field that matches is deleted before export. A journal record’s body contains nearly the entire journalctl --output=json entry, typically 20 to 30 fields and roughly ten times the size of the message itself. The Collector uses __REALTIME_TIMESTAMP as the record timestamp and removes that field before this expression runs. The default removes identifier and bookkeeping fields: __CURSOR, _BOOT_ID, _MACHINE_ID, _CMDLINE, _EXE, the _SYSTEMD_ cgroup and session fields, _HOSTNAME, and the SYSLOG_ transport fields. _HOSTNAME is removed because every record already carries the host as a resource attribute. SYSLOG_PID, SYSLOG_TIMESTAMP, and SYSLOG_RAW are sender-provided values that can differ from the journal’s trusted process ID, timestamp, and normalized message. The fields that remain are the ones an operator filters on: MESSAGE, PRIORITY, which the Collector also maps to the record’s severity, _SYSTEMD_UNIT, SYSLOG_IDENTIFIER, SYSLOG_FACILITY, _PID, _COMM, _TRANSPORT, _UID, _GID, and any field a program wrote into its own journal entries, such as CODE_*, UNIT, and USER_UNIT. SYSLOG_FACILITY remains because nothing else on the record says whether a syslog-transported message came from auth, authpriv, cron, or daemon. To keep a field the default removes, set a pattern that doesn’t match it. To keep every field, set a pattern that matches no field name, such as ^$. The pattern can’t contain backslashes or double quotes, so write [0-9] rather than \d. Journal field names contain only uppercase letters, digits, and underscores, so a pattern over them needs no escapes. Matching is unanchored unless the pattern anchors it, which is why the default begins with ^. The pattern must be a valid RE2 regular expression. A pattern that doesn’t compile, such as one with an unbalanced parenthesis, is rejected when the Collector loads its configuration, with an error that names this setting.
• systemd Type: object. Optional. Settings for collecting the state of systemd units on each host, and for reporting whether the Collector can reach the system bus it reads them from.
• systemd.enabled Type: boolean. Optional. Default: true. Whether to enable this configuration block. If true, the Collector runs the integration or capability. If false, the Collector doesn’t run it.
• systemd.units Type: array of string. Optional. Default: ["*.service"]. Minimum items: 1. Patterns for the systemd units to report state for, in the systemd sense of unit: .service, .timer, .socket, .mount, and so on. The list replaces the default, so include *.service to keep reporting services. The Collector reports matching, currently loaded units; it doesn’t enumerate every unit file installed on disk. This is the setting to narrow when the number of time series matters. systemd.unit.state is emitted once per unit per possible active_state, and there are eight states, so a host with 120 loaded matching service units produces around 960 time series from it. Each unit is also its own resource, carrying systemd.unit.name. An explicit list of unit names, or a subset of services such as ["sshd.service", "nginx.service"], brings the count down on a host with an unusual number of units. Widen the list the same way, for example ["*.service", "*.timer"].
• systemd.service_cpu_time Type: boolean. Optional. Default: false. Whether to collect systemd.service.cpu.time, the total CPU time per service. If true, the Collector reads each service’s CPU time from its cgroup on every collection. If false, the metric isn’t collected. Requires cgroup v2, the unified hierarchy. On a cgroup v1 or hybrid host, each service contributes a partial collection error, and the Collector logs the combined error once per collection interval. RHEL and CentOS 7 and 8, Ubuntu before 21.10, Amazon Linux 2, and Debian 10 all default to a non-unified hierarchy. To confirm the hierarchy, run stat -fc %T /sys/fs/cgroup, which prints cgroup2fs on a unified host.
• systemd.service_restarts Type: boolean. Optional. Default: true. Whether to collect systemd.service.restarts, the number of times systemd has automatically restarted each service since the service was last started by hand. The value is the unit’s NRestarts property, which counts restarts that a Restart= setting triggered after the process exited. A manual systemctl restart isn’t counted and resets the count to zero. The Collector reads the property over the system bus on every collection, one time series per matching .service unit, with no cgroup requirement. A rising value identifies a service that keeps exiting while systemd.unit.state reports it active between restarts. The property exists on systemd 235 and later. If false, the metric isn’t collected.
• syslog Type: object. Optional. Settings for the syslog listener each host opens, and for reporting whether that listener is up.
• syslog.enabled Type: boolean. Optional. Default: false. Whether to accept syslog forwarded by rsyslog, syslog-ng, or a network device. If true, the Collector opens a listener and collects the messages sent to it as log records. If false, the Collector opens no listener. The other capabilities read state the Collector can already see, while this one collects nothing until a sender is configured to forward to it. On a host that runs rsyslog or syslog-ng, the journal generally carries the same records already. enabled: true on its own listens on UDP 127.0.0.1:5514 and parses RFC 5424. Set udp or tcp to move or change the listener, and protocol to match what the sender emits. Journal and syslog records require a logs exporter. If you use tenant egress, enable cxdot.egress.tenant.signals.logs.
• syslog.udp Type: object. Optional. Default: {"listen_address":"127.0.0.1:5514"}. UDP listener. The default is the loopback address on a high port, which suits a forwarder on the same host and is the only address that’s safe to choose by default: 0.0.0.0 would open the host to the network, and the conventional port 514 needs privilege to bind and is usually already held by the host’s syslog daemon. Widen the address to accept from other hosts, for example 0.0.0.0:5514. Set tcp instead for a TCP listener. The two blocks are exclusive.
• syslog.udp.listen_address Type: string. Required. Minimum length: 1. <host>:<port> to receive syslog messages over UDP on. Required when syslog.udp is configured.
• syslog.tcp Type: object. Optional. TCP listener, for forwarders configured to use TCP. Unset by default. Set it in place of udp, not beside it. The listener splits the stream on newlines and accepts plaintext connections unless enable_octet_counting or tls say otherwise.
• syslog.tcp.listen_address Type: string. Required. Minimum length: 1. <host>:<port> to accept syslog connections on. Required when syslog.tcp is configured.
• syslog.tcp.enable_octet_counting Type: boolean. Optional. Default: false. Whether each message on the TCP stream starts with its length in octets, the octet-counting framing of RFC 6587. If false, the Collector splits the stream on newlines, which is the framing rsyslog and syslog-ng emit over TCP unless configured otherwise. Set true for a sender configured for octet counting, such as rsyslog with TCP_Framing="octet-counted". A stream framed one way and read the other produces records that run together or are split mid-message.
• syslog.tcp.tls Type: object. Optional. Transport layer security (TLS) settings for the TCP listener. When set, the listener accepts only TLS connections. Requires a certificate and key: either cert_file and key_file, or cert_pem and key_pem. A file and its PEM alternative can’t both be set. The Collector masks key_pem and the trusted platform module authorization values in diagnostic output. Set client_ca_file to require and verify a client certificate from each sender. When unset, the listener accepts plaintext connections, which suits a forwarder on the same host. A listener that accepts connections from other hosts carries log records across the network, so configure TLS for it and configure the senders to use TLS. Only TCP supports TLS; udp has no equivalent setting.
• syslog.tcp.tls.cert_file Type: string. Optional. Path to the PEM-encoded certificate file the listener presents to each sender. Pair it with key_file; cert_pem and key_pem are the in-memory alternative.
• syslog.tcp.tls.cert_pem Type: string. Optional. PEM-encoded certificate content the listener presents to each sender, in place of cert_file.
• syslog.tcp.tls.cipher_suites Type: array of string. Optional. TLS cipher suites the listener accepts, in preference order. If omitted, the Collector uses the secure defaults from the Go TLS library. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go.
• syslog.tcp.tls.client_ca_file Type: string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file. If set, the listener requires each sender to present a certificate signed by that authority and rejects connections that don’t. If omitted, the listener accepts any sender that completes the TLS handshake.
• syslog.tcp.tls.client_ca_file_reload Type: boolean. Optional. Whether to reload client_ca_file when the file changes on disk, so a rotated authority takes effect without restarting the Collector. If false, the Collector reads the file once at startup.
• syslog.tcp.tls.curve_preferences Type: array of string. Optional. Elliptic curves the listener accepts for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses the Go TLS library defaults.
• syslog.tcp.tls.key_file Type: string. Optional. Path to the PEM-encoded private key file for the certificate in cert_file.
• syslog.tcp.tls.max_version Type: string. Optional. Maximum TLS protocol version the listener accepts. If omitted, the Collector uses the maximum version supported by the Go TLS library.
• syslog.tcp.tls.min_version Type: string. Optional. Minimum TLS protocol version the listener accepts. If omitted, the minimum version is TLS 1.2.
• syslog.tcp.tls.reload_interval Type: duration. Optional. How often the Collector reloads the listener’s certificate and key files, so a renewed certificate takes effect without a restart. If omitted, the Collector doesn’t reload them.
• syslog.tcp.tls.key_pem Type: string. Optional. PEM-encoded private key content for the certificate in cert_pem, in place of key_file.
• syslog.tcp.tls.tpm Type: object. Optional. Trusted platform module settings for the listener’s private key.
• syslog.tcp.tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
• syslog.tcp.tls.tpm.path Type: string. Optional. Path to the trusted platform module device or Unix domain socket, such as /dev/tpm0 or /dev/tpmrm0.
• syslog.tcp.tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
• syslog.tcp.tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
• syslog.protocol Type: string. Optional. Default: rfc5424. Wire format the sender emits. rfc5424 is the current format. rfc3164 is the older format, which the default rsyslog forwarding template, the syslog-ng network() destination, and most network appliances emit unless configured otherwise. A record that arrives in one format while the listener expects the other isn’t parsed and is forwarded as one opaque line. none skips parsing and keeps each line as the record body, for a sender that conforms to neither format. Allowed values: rfc5424, rfc3164, none.
• syslog.location Type: string. Optional. Default: UTC. Minimum length: 1. Time zone to interpret RFC 3164 timestamps in, as a name from the Internet Assigned Numbers Authority time zone database, such as America/New_York. An RFC 3164 timestamp carries no zone, so without this setting the Collector reads it as UTC, and records from a sender whose clock shows local time get a timestamp that’s off by the zone offset. Set it to the zone the senders use. RFC 5424 timestamps carry their own offset, so this setting doesn’t affect them. The time zone of the host running the Collector isn’t consulted. A name the Collector can’t load stops the listener from starting, and the integration’s listener health reports the failure.