destinations that reference
notifier connections,
the default approach for new credentials and routes. The same API still accepts
notifier_slugs that reference separate
notifiers until you finish moving those routes. You can’t set both on the same
notifier list. While you migrate, follow the notifier resource migration process.
Optionally, you can use signals to create groups of
notifications. Signals determine how to group alerts, which affects how many
notifications Cortex XCOR sends.
To customize the title and description of notifications generated by a monitor,
see Notification templates.
Identify failed notification delivery
Cortex XCOR reports repeated delivery failures so that you can identify alerts that aren’t reaching their destinations. A delivery failure can appear in the following locations:- A Delivery failing badge marks an affected entry in the Notifiers list or the Notifier connections tab. Hold the pointer over the badge to view the cause.
- The details view for an affected notifier or notifier connection displays one or more banners with the cause, the number of affected monitors, and how long the failures have occurred.
- An affected monitor’s detail page displays banners for failing notifiers or notifier connections on routes that match the monitor.
- An SLO detail page displays banners for failing notification targets when burn rate alerting is enabled. Because Cortex XCOR evaluates the notification policy at the policy level for an SLO, these banners can include targets from overrides that don’t match the SLO.
- For a configuration problem, review the diagnostic message, then correct invalid credentials or settings. If the banner identifies a notifier that sends through a notifier connection, inspect both configurations.
- If the destination rejects notifications, Cortex XCOR retries delivery until the destination recovers.
- If the destination rate limits notifications, route fewer notifications through the affected notifier or connection, or request a higher rate limit from the receiving service.
- For an internal delivery problem, Cortex XCOR retries delivery. Contact Cortex XCOR Support if the failure continues.
- In the navigation menu, select Alerting > Monitors.
- Select a monitor.
- Click Alert history in the actions menu.
Monitor failed notification delivery
Thechrono_notification_delivery_failing_targets gauge reports one time series
with a value of 1 for each notifier or notifier connection that has a
Delivery failing status. Each time series has the following labels:
target_type: The type of notification target. The value isENTITY_NOTICE_TARGET_TYPE_NOTIFIERorENTITY_NOTICE_TARGET_TYPE_EXTERNAL_CONNECTION.target_slug: The slug of the failing notifier or notifier connection.
0. Cortex XCOR updates the metric about every five
minutes. A time series can remain for up to two hours after the delivery problem is
resolved.
Each time series contributes to your metrics usage. The number of time series is
limited to the number of notifiers and notifier connections with a Delivery failing
status.
target_slug so that each failing notification target generates a
separate alert that includes its slug. Don’t configure a missing data condition for
this monitor. The metric is absent when no notification targets are failing, so a
missing data condition would also alert when delivery is healthy.
To count failing notification targets by type, use the following query: