Notifiers

When an alert triggers, Palo Alto Networks Cortex XCOR sends a notification based on a notifier, which is an endpoint you specify that defines where to deliver active alerts and who to notify. When creating a notification policy, you select a notifier that specifies where to route the alert. A single alert results in one notification, which can be of either warn or critical severity. If there are multiple time series, the alert is de-duplicated and activated only once. However, the alert indicates which time series are violating the condition. Each notification policy triggers the sending of new notifications periodically, with a default period of 60 minutes. Special global notifiers receive and send notifications for every alert generated by Cortex XCOR. For Slack, PagerDuty, webhooks, VictorOps, or OpsGenie, the Cortex XCOR API uses inline destinations that reference notifier connections, or legacy notifier_slugs as in Create a notifier and View notifiers. You can’t set both on the same notifier list. For rules, migration steps, and field-level destination fields, see Notifier resource migration and Notifier connections and route destinations. Choose a notifier based on where you want to deliver alerts. Cortex XCOR supports the following notifiers:

Discard

Create a notifier to discard all notifications.

Email

Create an email notifier to send monitor alerts to a specific email address.

incident.io

Configure an alert source in incident.io and then create a webhook notifier in Cortex XCOR.

OpsGenie

Create a notifier in OpsGenie.

PagerDuty

Create a service in PagerDuty and then create the notifier in Cortex XCOR.

Slack

Generate an incoming webhook in Slack and then create a Slack notifier in Cortex XCOR.

VictorOps

Create an API key in VictorOps and then create a VictorOps notifier in Cortex XCOR.

Webhook

Create a webhook notifier with a URL that specifies the endpoint to send HTTP POST requests to.

Prerequisites

Some notifier configurations require specific information, such as a service key, an API key, or support intervention. The required information depends on the type of notifier that you’re creating. Refer to each notifier’s configuration to determine the necessary information.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

View notifiers

You can view available notifiers in Cortex XCOR, or return a list of notifiers using Chronoctl. You can view notifiers created by Terraform in Cortex XCOR, but you can’t modify them.

Web
Chronoctl
API

To view a notifier:

In the navigation menu, select Alerting > Notifiers.
Search for the notifier you want to view and select it from the list.

The Edit Notifier page displays the notifier definition.If an entry has recent delivery failures, the list marks it with a Delivery failing badge. Hold the pointer over the badge to view the causes. Select the notifier to view banners with diagnostic messages, the number of affected monitors, and the failure duration. For remediation steps, see Identify failed notification delivery.

To view available notifiers with Chronoctl, use one of the following commands:

To return a complete list of notifier resource definitions as YAML documents, use this command:
chronoctl notifiers list
To return specific resources by name or slug, pass a comma-delimited list to the --names or --slugs options respectively. For example:
chronoctl notifiers list --names LIST
Replace LIST with a list of notifier names, such as "Infra Webhook,Email Notifier". Each notifier has a name field in the resource definition, which you include in the list.
To return a single resource definition by slug, use the notifiers read command:
chronoctl notifiers read SLUG
Replace SLUG with the notifier’s slug, such as email-notifier.

These commands return a Notifier resource, which has the following format:

Chronoctl example

api_version: v1/config
kind: Notifier
spec:
  slug: email-notifier
  name: Email Notifier
  created_at: "2022-11-17T22:56:59.000Z"
  updated_at: "2022-11-17T22:56:59.000Z"
  email:
    to: your-address@example.com
    html: <b>Alert triggering</b>
    text: Alert triggering

To complete this action with the Cortex XCOR API, use the ListNotifiers endpoint.Because the Cortex XCOR API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Cortex XCOR instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Create a notifier

Notifiers use a custom definition format in YAML. You can use Cortex XCOR, Chronoctl, or the Terraform provider to create and manage them. Notifiers support variables and templating. After creating a notifier, you can select it when defining a notification policy. The following steps apply to creating notifiers. See each notifier page for examples of creating that specific notifier type.

Web
Chronoctl
Terraform
API

To create a notifier:

In the navigation menu, select Alerting > Notifiers.
Click Create notifier.
Enter a descriptive name for the notifier.
Select the type of notifier you want to create.
Enter the required information for the notifier.
Optional: Select Notify when resolved to send a resolved alert notification.
Click Save.

To create one or more notifiers with Chronoctl, create a YAML file to define your notifiers and apply the configuration to your instance. Use the lowercase name of the service as a YAML key to set the service to use.Refer to each notifier type for a full example of a definition. You can also generate an example that contains multiple notifier types with Chronoctl by using this command:

chronoctl notifiers scaffold

To apply a definition, use this command:

chronoctl apply -f FILE_NAME.yaml

Replace FILE_NAME with the name of your notifier YAML file.

Create an alert notifier with Terraform by using the chronosphere_{channel}_alert_notifier type followed by a name in a resource declaration.See each notifier type for a full definition example.

Add the definition to a Terraform file.
Run this command to create the resource:
terraform apply

To complete this action with the Cortex XCOR API, use the CreateNotifier endpoint.Because the Cortex XCOR API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Cortex XCOR instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

To create global notifiers, contact Cortex XCOR Support.

Edit a notifier

Select from the following methods to edit a notifier.

Web
Chronoctl
Terraform
API

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To edit a notifier:

In the navigation menu, select Alerting > Notifiers.
Search for the notifier you want to view and select it from the list.
In the Edit Notifier page, make changes to the notifier definition.
Click Save.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To edit a notifier with Chronoctl:

Get the resource definition for the notifier you want to edit:
- To return a complete list of notifier resource definitions as YAML documents, use the notifiers list command:
  chronoctl notifiers list
- To return a single resource definition by slug, use this command:
  chronoctl notifiers read SLUG
  Replace SLUG with the notifier’s slug, such as email-notifier.
Modify the YAML file for the notifier you want to update.
Apply your changes:
chronoctl apply -f FILE_NAME.yaml
Replace FILE_NAME with the name of your notifier YAML file.

Chronoctl updates the notifier’s properties if it has the same slug.

To edit a notifier using Terraform:

Modify the notifier resource’s existing properties.
Run terraform apply to apply the changes.
terraform apply

To complete this action with the Cortex XCOR API, use the UpdateNotifier endpoint.Because the Cortex XCOR API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Cortex XCOR instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

To edit global notifiers, contact Cortex XCOR Support.

Delete a notifier

Select from the following methods to delete a notifier.

Web
Chronoctl
Terraform
API

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To delete a notifier:

In the navigation menu, select Alerting > Notifiers.
Search for the notifier you want to delete and select it from the list.
In the Edit Notifier page, click the delete icon .
In the confirmation dialog, click Delete.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To delete a notifier with Chronoctl, run this command:

chronoctl notifiers delete SLUG

Replace SLUG with the slug of the notifier you want to delete.

To delete a resource that’s managed by Terraform:

Edit your Terraform configuration file to remove the pre-existing resource definition.
Run this command to remove the resource from Cortex XCOR:
terraform apply

To complete this action with the Cortex XCOR API, use the DeleteNotifier endpoint.Because the Cortex XCOR API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Cortex XCOR instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

To delete global notifiers, contact Cortex XCOR Support.

Notifications for alerts in third-party services

By default, Cortex XCOR sends a notification when an alert for a monitor resolves in third-party services, such as PagerDuty and VictorOps. You can choose to not send notifications by configuring the specified parameter in the Chronoctl YAML definition or Terraform resource for your notifier.

By default, Cortex XCOR doesn’t send notifications when an alert resolves. To send resolved notifications, select Notify when resolved when creating a notifier. For notifiers created using Chronoctl, Terraform, or the Cortex XCOR API, use the skip_resolved or send_resolved parameter.

Chronoctl
Terraform

To skip notifications when alerts for a monitor resolve, set the value of the skip_resolved property to true in your notifier definition.For example, the following definition skips sending notifications for resolved alerts for an Opsgenie notifier:

Chronoctl example

api_version: v1/config
kind: Notifier
spec:
  name: test-opsgenie
  slug: test-opsgenie
  skip_resolved: true
  ops_genie:
    # ...

To skip notifications when alerts for a monitor resolve, set the value of the send_resolved key to false. If you don’t explicitly set a value for this key, Cortex XCOR sends notifications for resolved alerts by default. The send_resolved key is a top-level key.For example, the following resource skips sending notifications for resolved alerts for an Opsgenie notifier:

Terraform example

resource "chronosphere_opsgenie_alert_notifier" "opsgenie_notifier" {
  name = "Opsgenie Notifier"
  # ...
  send_resolved = false
  # ...
}

Use variables in notifiers

Notification content supports Go templates and Prometheus Alertmanager variables. Notifiers support many features and variables demonstrated in Alertmanager notification examples.

Available variables

You can reference Prometheus Alertmanager variables with the {{.VARIABLE_NAME }} syntax. Variable references work in the notifier’s definition and the notification’s contents. Notifiers can access monitor labels by using variables with the {{ .CommonLabels.LABEL }} pattern, and from the alerting metric with the {{ .Labels.LABEL }} pattern. In both patterns, replace LABEL with the label’s name. Similarly, you can reference monitor-defined annotations by using the {{ .CommonAnnotations.ANNOTATION }} pattern, replacing ANNOTATION with the monitor annotation’s name.

If your monitor query includes a label that generates different annotation values, then annotations won’t be included in CommonAnnotations or CommonLabels.

See the Alertmanager documentation for a reference list of alerting variables and templating functions. For a simpler way to customize the title and description of notifications using monitor-specific variables, see Notification templates.

Variable usage examples

This Slack notifier’s channel value takes advantage of monitor-defined labels to post a notification to a channel based on a triggering metric’s label:

Chronoctl
Terraform

Chronoctl example

api_version: v1/config
kind: Notifier
spec:
  name: slack-notifier
  slug: slack-notifier
  slack:
    # ...
    channel: "metric-notification-{{ .CommonLabels.<LABEL> }}"
    # ...

Terraform example

resource "chronosphere_slack_alert_notifier" "slack_notifier" {
  name = "Slack Notifier"
  # ...
  channel = "metric-notification-{{ .CommonLabels.<LABEL> }}"
  # ...
}

You can also reference annotations defined in the monitor:

Chronoctl
Terraform

Chronoctl example

api_version: v1/config
kind: Notifier
spec:
  name: slack-notifier
  slug: slack-notifier
  slack:
    # ...
    text: "Summary: {{ .CommonAnnotations.summary }}"
    # ...

Terraform example

resource "chronosphere_slack_alert_notifier" "slack_notifier" {
  name = "Slack Notifier"
  # ...
  text = "Summary: {{ .CommonAnnotations.summary }}"
  # ...
}

You can also use multiple labels in notifiers. The following example uses multiple labels for an OpsGenie notifier:

Chronoctl

Chronoctl example

api_version: v1/config
kind: Notifier
spec:
  ops_genie:
    priority: '{{ .CommonLabels.Priority }}'
---
api_version: v1/config
kind: Monitor
spec:
  labels:
    Priority: P2

The priority in the outgoing notification is correctly templated with priority P2.

Global notifiers

Instead of using a notification policy to define when a notification is sent, global notifiers receive and send notifications for every alert generated across Cortex XCOR. Since these notifiers can generate a constant feed of many notifications, they must be created and managed exclusively by Cortex XCOR Support. You can’t edit or delete global notifiers on your own, and Cortex XCOR designates this with a banner in notifier editing interfaces. In the Notifiers list, Cortex XCOR designates global notifiers with a Global chip displayed with their type.

Cortex XCOR

Palo Alto Networks Telemetry Pipeline

More information

Discard

Email

incident.io

OpsGenie

PagerDuty

Slack

VictorOps

Webhook

Prerequisites

View notifiers

Create a notifier

Edit a notifier

Delete a notifier

Notifications for alerts in third-party services

Use variables in notifiers

Available variables

Variable usage examples

Global notifiers

Discard

Email

incident.io

OpsGenie

PagerDuty

Slack

VictorOps

Webhook

​Prerequisites

​View notifiers

​Create a notifier

​Edit a notifier

​Delete a notifier

​Notifications for alerts in third-party services

​Use variables in notifiers

​Available variables

​Variable usage examples

​Global notifiers

Prerequisites

View notifiers

Create a notifier

Edit a notifier

Delete a notifier

Notifications for alerts in third-party services

Use variables in notifiers

Available variables

Variable usage examples

Global notifiers