View muting rules
Select from the following methods to view a list of muting rules and review which monitors and time series they affect.- Web
- Chronoctl
- API
To display a list of defined muting rules, in the navigation menu select
Alerting > Muting Rules.You can take the following actions:
- Use the Search muting rules box to find a specific rule by name.
- Filter the list of rules by choosing a rule status, which can be Active, Scheduled, or Expired.
-
Point to any muting rule to display additional actions:
- Copy muting rule: Available for rules that are Active, Scheduled, or Expired.
- Expire Muting Rule: Available for rules that are Active or Scheduled.
- Select any muting rule in the list to view its overview. The overview identifies the matched monitor or time series, the start and end times, the muting schedule, the creator, the total monitors and time series affected, and whether an expiration notification is configured. A recurring muting rule has an Active status between its overall start and end times, including periods outside its scheduled muting windows. Review the Schedule field to determine when the rule mutes notifications.
- All alerts are a complete list of alerts muted during the selected time period.
- Active alerts are actively triggering, but are currently muted by the muting rule.
- Resolved alerts were previously triggered and muted by the muting rule, but are no longer triggering.
View per-series muting status on a monitor
When you open a monitor in Cortex XCOR, the Series Legend displays an Alert status for each time series in the chart. When an active muting rule’s label matchers match a specific series, the legend shows Muted for that series. Other series in the same monitor can display different statuses at the same time. For example, if a muting rule matchesenvironment=staging, only series with that
label indicate Muted. Series with environment=production continue to indicate
their actual alert status, such as Critical or Warning.
Muted appears as a link to the alert page when the series has an active alert,
or as plain text when the series has no active alert but its labels match a muting
rule. Open any monitor to display the Alert status column in the Series Legend.
To review all active and scheduled rules that match a monitor, see
Muting rules that apply.
Create a muting rule
You can create a muting rule to mute an entire monitor or specific time series. To mute notifications during repeating weekly time windows, create a scheduled muting rule.- Web
- Chronoctl
- API
You can add a muting rule from the navigation menu from the list of defined monitors
or directly from the list of muting rules.Adding a muting rule from an existing monitor pre-populates the matching criteria
without having to manually enter that information. As you define a muting rule, the
Preview alerts section shows which alerts match the rule and would be muted.
Based on this preview, you can review and adjust the rule before it’s active.To create a muting rule:
-
Choose where to create the muting rule:
- To create the rule from an existing monitor:
- In the navigation menu, select Alerting > Monitors and click the monitor you want to mute.
- If you want to mute specific time series only (and not the entire monitor), choose the time series from the list in the Series Legend.
- On the same line as the monitor’s title, click Mute.
- To create a new muting rule, in the navigation menu, select Alerting > Muting Rules, and click Create muting rule.
- To create the rule from an existing monitor:
-
Define your muting rule:
- Name: Required, but doesn’t need to be unique. When creating a muting rule from a monitor, this field is prepopulated with that monitor’s name.
- What alerts should be muted?: Select A monitor, An SLO, or Time series. Define the Matcher operator and the Key and Value to match.
- When should it start?: Start muting Now or at a specific Start time.
- For how long?: The Duration or defined End time of the muting rule. Muting rules are based on the time zone of the creator, with working hours defined as 9 AM to 5 PM Monday through Friday. Creating a rule that extends outside these hours displays an informational message in the muting rule interface.
- And on what schedule?: Select Continuous to mute alerts for the entire duration of the rule. Select Scheduled to mute alerts only during repeating weekly time windows. For configuration details, see Schedule recurring muting.
-
Expiration notification (optional): Enable
Notify before this mute expires, choose Notifiers or Destinations,
and set the recipients and lead time. For limits and behavior, see
Expiration notification.
Muting rules support regular expressions, which you can use to match key values against. Refer to the section about how to match on values using regular expressions.
- Click Create.
Schedule recurring muting
A scheduled muting rule suppresses matching notifications during repeating weekly time windows. Cortex XCOR continues evaluating the monitor while the rule suppresses notifications. When a window ends, notification processing resumes according to the existing notification policy. Ending a scheduled window doesn’t reset the notification policy’s repeat interval. The rule’s start and end times define the overall recurrence period. The rule mutes only during the portions of its weekly windows that fall between the rule’s overall start and end times. The selected Internet Assigned Numbers Authority (IANA) time zone keeps each window aligned with local time when daylight saving time changes. Use the following guidelines when defining weekly windows:- Set each range with a start time before its end time. Ranges on the same day can’t overlap.
- To span midnight, end one range at
24:00and start another range at00:00on the following day. Cortex XCOR treats adjacent ranges across midnight as one occurrence. - A day without a range has no scheduled muting.
- The start of a range is included, and its end is excluded. For example, a range
from
09:00to17:00stops muting at17:00.
- Web
- Chronoctl
- API
To configure a recurring schedule when creating a muting rule:
- For And on what schedule?, select Scheduled.
- Select the Time zone for the schedule.
- Set the Start and End of the range, and select its days under Repeat every.
- Optional: Click Add range to define another weekly time window.
- Complete the remaining fields and click Create.
Match on values using regular expressions
When creating a muting rule, you can use regular expressions to include or exclude values. In Cortex XCOR, select one of these PromQL Matchers:- Exact match (
=) - Match regex (
=~)
type in the label matcher to
REGEX to match values with a regular expression.
For example, you might want to match a time series with a label named
envoy_cluster_name and include any values that begin with cluster_span_rc. You
can use Cortex XCOR or Chronoctl to define a regular expression pattern
that matches based on values such as cluster_span_rc_shared__spanhandler_proxy_r-0.
- Web
- Chronoctl
When defining your muting rule in Cortex XCOR, specify the following
criteria:
- Select the match regex (
=~) matcher. - Enter
envoy_cluster_nameas the key. - Enter
cluster_span_rc.*as the value to match on.
Expiration notification
An expiration notification is an optional, one-time message Cortex XCOR sends before a muting rule ends. The message includes how many matching alert series are currently firing and how many monitors they affect, so recipients know notifications might resume after the muting rule expires. Cortex XCOR doesn’t send a separate message at the exact moment the muting rule expires. Set expiration notifications in the drawers for creating or editing a muting rule. The setting is separate from notification policies. When you open a saved rule from the Muting Rules list, the details drawer lists Expiration notification with the lead time and recipients, or Not configured when the toggle is off.Limits
- Expiration notifications support only email, Slack, or webhook recipients. When you choose Notifiers, the picker lists notifiers that meet that requirement and excludes PagerDuty, Opsgenie, VictorOps, and discard notifiers. When you choose Destinations, add inline email, Slack, or webhook destinations the same way you define them in a notification policy route.
- Lead time must be at least five minutes and at most seven days, and shorter than the mute duration (from start to end).
- You can’t add the same notifier with the same lead time twice on one rule.
- Repeat interval and notification grouping options don’t apply. The warning sends once before the overall rule end time, not once per weekly occurrence.
Message content
The notification title uses this format:Muting rule "<name>" expires at <time>.
The body summarizes how many matching alert series are currently firing, how many
monitors they span, and that those alerts might notify after the mute expires. Counts
reflect the state when the message sends, not which alerts will page after the mute
expires. A link opens the muting rule in Cortex XCOR when a console URL is available.
Webhook payloads include eventType set to muting_rule.expiration_warning.v1.
Rule changes
Cortex XCOR sends at most one expiration warning for each muting rule:- Changing the rule name or matchers doesn’t send another warning.
- Changing the end time, recipients, or lead time cancels the pending warning and schedules a new one. Extending a mute moves the warning to the new end time.
- Deleting the rule cancels any pending warning.
- If the rule is already expired or stopped when you save, or the mute expires before the warning is due, Cortex XCOR doesn’t send the warning.
- Copying a muting rule doesn’t copy expiration notification settings. Add them on the new rule if needed.
Edit a muting rule
Edit a muting rule to change the name, duration, or expiration notification settings. The matching criteria, start time, and weekly schedule can’t change after creation.- Web
- Chronoctl
- API
To edit a muting rule:
- On the Muting Rules page, click the muting rule you want to edit.
- On the Muting Rules Details page, click Edit. The Edit Muting Rule drawer opens and displays a message stating when the selected muting rule duration ends.
-
Make one or more changes:
- To change the rule’s name, update the Name field.
- Expiration notification: Use the same fields as when you create a muting rule. See Expiration notification for limits and behavior.
- To change the duration by a relative time period, select Extend by or Shorten by. Enter a number in the Time period field, and then choose Hours, Minutes, or Days from the menu. The new scheduled time appears next to the duration boxes.
- To set a specific end time, select End time, and then set a custom time.
- Review the read-only Schedule section. To use different weekly windows, create a replacement muting rule.
- Click Save.
Stop a muting rule
Use Cortex XCOR or Chronoctl to stop a muting rule, which deactivates the rule and changes its status to Expired.- Web
- Chronoctl
To stop a muting rule:
- On the Muting Rules page, take one of the following actions:
- Hold the pointer over the muting rule you want to expire and click Expire Muting Rule.
- Click a muting rule to open the Muting Rules Details page, and then click Expire Muting Rule.
- In the confirmation dialog, click Confirm (Expire).
Delete a muting rule
Select from the following methods to delete a muting rule.- Chronoctl
- API
To permanently delete a muting rule with Chronoctl, use this command:Replace
SLUG with the rule’s slug, such as 3021f691-a0bf-4bdd-a4c8-f8468e5b272a.