Review a monitor’s status
The detail page leads with the monitor’s current Status, which shows the number of active alerts at each severity and whether any notifications are muted:- Passing indicates that no series tracked by the monitor is actively triggering an alert.
- If any alerts are muted, a banner describes the muting. Click More details to see links to the active muting rules that are silencing notifications.
- If a notifier or notifier connection on a matching route for a notification policy has delivery failures, a banner identifies the affected target and describes the cause. See Identify failed notification delivery for remediation steps.
Alerting signal limit
Each monitor can track up to 5,000 alerting signals. When a monitor reaches the limit, Cortex XCOR continues to process previously tracked signals but blocks new signals. Blocked signals don’t generate notifications or appear in alert history. The monitor detail page displays aSignal limit reached warning with the configured
limit and the number of blocked signals.
To resume processing new signals, reduce the number of signals that meet the monitor’s
alert conditions:
- Remove labels with many distinct values from the monitor’s signal grouping. Labels that identify pods, instances, requests, or traces can create a separate signal for each value.
- Narrow the monitor query to return only the intended time series.
- Split a broad monitor into targeted monitors. Each monitor has its own signal limit.
Query results
The Query results section visualizes the time series data returned by the monitor’s query as a time series chart. Use the following controls to adjust the visualization:- All, Alerting, or Critical: Filter which series display on the chart by alert state.
- Time range selector: Define the time span of data visualized in the chart.
- Show thresholds: Draw horizontal lines on the chart depicting the monitor’s triggering thresholds.
- Show query: Display the query that populates the chart.
- Open in explorer: Open Metrics Explorer populated with the monitor’s query.
- Analyze anomaly (DDx): Right-click or long-press a series to perform Cortex XCOR Differential Diagnosis (DDx) on it.
Series legend
The series legend lists all metrics displayed in the chart, in either list or table view. Both views show the label keys and values for each series. The Alert status column shows each series’ current alert state as a link. The link text is Critical, Warning, Muted, Resolved, or Passing. Click the link to open that series’ alert details page. A series might be Muted due to an active muting rule. For more information, see per-series muting status. Filter and interact with the series legend:- Use Search Series to filter the displayed series by text.
- Click a series color to isolate that line on the chart. Click again to clear the
selection. Use
Control+click (Command+click for macOS) to select multiple series. - In a label column, hold the pointer over a cell to reveal the Add or Remove label from filters icons. These filters apply only when the time series includes all filtered labels.
Alert history
The stacked histogram in this card groups alerts by Warning and Critical severity based on when each alert triggered. The histogram includes all signals for the monitor, regardless of the signal selected on the page. The card includes the following insights:- Fired Alerts: The number of alerts that triggered during the selected time range.
- Trend: The percentage change in triggered alerts compared with the preceding
time range of the same length. The card displays
no prior alertswhen the preceding range has no alerts, orprevious window unavailablewhen Cortex XCOR can’t retrieve that range. - Firing Now: The number of alerts that are still firing, including alerts that triggered before the selected time range.
- p50 time to resolve: The median time between an alert triggering and resolving. This value excludes alerts that are still firing.
- Quick-resolve share: The percentage of resolved alerts that resolved in five minutes or less.
- To sort, click a column header.
- To filter by event type, click the Event types dropdown and select the event types to display. Use the Search field in the dropdown to find a specific type.
- To toggle the signal scope, switch between the currently selected signal and all signals for the monitor.
- To view an alert’s JSON payload, click the > icon on the alert’s row. Click the icon to copy the payload.
There can be up to a five-minute delay between when an alert resolves and when
Cortex XCOR sends the resolution notification.
Alert event payload
The JSON payload captures the values at the time the alert triggered. The payload fields are primarily defined in the monitor data model.monitorSlug: The monitor’s unique slug.eventType: The type of event triggered. Valid alert event types are:- Alert triggered
- Alert resolved
- Notification sent (alert triggered)
- Notification failed (alert triggered)
- Notification sent (alert resolved)
- Notification failed (alert resolved)
- Alert muted
- Alert unmuted
createdAt: The time the alert triggered.signal: See signals. Contains one or morenameandvalueentries.details: Containsseverityas defined by the notification policy andnotifier.notifiervalues contain the notifiernameandslug.
Change events
If you’ve enabled change events, the query results chart can overlay associated events using category icons. To configure which change events appear:- Click Events to open the Display events panel.
- Toggle Show event markers to show or hide event markers on the chart.
- Toggle Filter by entity to limit events to those related to this monitor.
- Use the Code, Builder, or Event presets mode to configure the change events query. Event presets appears only when presets are configured for the monitor.
- Click Save.
Monitor information
The monitor detail page includes a sidebar with reference information about the monitor’s configuration:- Monitor information: The collection assigned as its Owner, the Team responsible for it, and its Notification policy.
- Datasource and update Interval.
- Additional information: Labels, Description, and templated links from the monitor’s annotations, giving on-call engineers direct access to runbooks, related dashboards, and documentation.
Muting rules that apply
The Muting rules that apply card lists active and scheduled muting rules that apply based on the monitor’s configuration. The card includes rules even when the monitor isn’t generating alerts. Each rule displays one of the following labels:- Fully mutes: Every matcher targets a label whose value is fixed for every alert the monitor can send and matches that value. During the rule’s muting windows, it mutes notifications for every alert.
- Partially mutes: Every matcher for a fixed label matches that label’s value, and at least one matcher targets a label that varies by alert, such as a signal label. During the rule’s muting windows, it mutes notifications only for alerts whose variable label values satisfy those matchers.
Active alerts list
The Active alerts list card shows all currently active alert instances for the monitor. The card header displays the total count, for example Active alerts list (3). Each entry links to the corresponding alert details page. If the monitor has no active alerts, the card displaysThis monitor does not have any active alerts.