Use annotations with monitors

Annotations provide runbook summaries, descriptions, and other text context for on-call engineers diagnosing issues. Use the separate Links section in the monitor editor to add templated URLs to dashboards, Logs Explorer, and external resources. Palo Alto Networks Cortex XCOR also supports link-valued annotations through Terraform, Chronoctl, or the Cortex XCOR API. For cross-telemetry linking patterns, see Create links to related information. Annotation values support two interchangeable forms: shorthand such as $value and $labels.LABEL, and the equivalent Go template form such as {{ .Value }} and {{ .Labels.LABEL }}. Use {{ .CommonLabels.LABEL }} to reference a monitor label, and {{ .Labels.LABEL }} to reference a label from the alerting metric. Replace LABEL with the label name.

Labels referenced in a variable must be present in the alerting time series. If a label is absent, the variable renders as empty in notifications.

Cortex XCOR provides these annotation variables:

Shorthand	Go template	Description
`$value`	`{{ .Value }}`	The numeric query result value at the time the alert triggered.
`$labels.LABEL`	`{{ .Labels.LABEL }}`	A label from the alerting time series. Replace `LABEL` with the label name.
`$externalLabels.LABEL`	`{{ .ExternalLabels.LABEL }}`	A system-level external label configured in Cortex XCOR. Replace `LABEL` with the label name.
`$externalURL`	`{{ .ExternalURL }}`	The base URL of your Cortex XCOR instance.
`$startTime`	`{{ .StartTime }}`	The time the alert started firing.
`$endTime`	`{{ .EndTime }}`	The time the alert resolved.

Use shorthand syntax to reference label names that contain dots, such as label.name. You can also use the humanize template function to format numeric values. For example, {{ $value | humanize }} renders 1500000 as 1.5M. For a reference list of alerting variables and templating functions, see the Alertmanager documentation. Annotation values support Markdown formatting, including bold text (**text**), inline code (`code`), named links ([label](url)), and plain HTTP URLs. Annotation values can also span multiple lines using the multiline annotation input. Each paragraph break renders as a new line in the monitor details view.

Add annotations to a monitor

Web
Chronoctl
Terraform

To add annotations to a monitor:

Create a monitor.

In the Annotations section, add a description for your annotation in the Key field, and text or links in the Value field. For example, you might add the following key-value pairs as annotations:

Key	Value
summary	Instance `{{ $labels.instance }}` is down
description	Container `{{ $labels.namespace }}`/`{{ $labels.pod }}`/`{{ $labels.container }}` terminated with `{{ $labels.reason }}`.
value	Current value: `{{ $value }}`
runbook	`http://default-runbook`

With Markdown formatting enabled, annotation values support richer formatting. For example, a description value might look like:

**Pod down**: `{{ $labels.namespace }}/{{ $labels.pod }}` terminated
with reason `{{ $labels.reason }}`. See the
[runbook](https://runbooks.example.com/pod-down) for remediation steps.

To add annotations to a monitor with Chronoctl:

Define the resource definition for your monitor, and add an annotations section:

Chronoctl example

api_version: v1/config
kind: Monitor
spec:
  # Required name of the monitor. Can be modified after the monitor is created.
  name: Instance is down
  # PromQL query. If set, you can't set graphite_query.
  prometheus_query: up{env="prod", instance="kubernetes", exported_job="JOB123"} ==0
  # Annotations are visible in notifications generated by this monitor.
  # You can template annotations with labels from notifications.
  annotations:
    summary: 'Instance {{ $labels.instance }} is down'
    description: 'Container {{ $labels.namespace }}/{{ $labels.pod }}/{{ $labels.container }} terminated with {{ $labels.reason }}.'
    value: 'Current value: {{ $value }}'
    runbook: 'http://default-runbook'

Apply the changes:
chronoctl apply -f FILE_NAME
Replace FILE_NAME with the name of your monitor YAML file.

To add annotations to a monitor with Terraform:

Create a monitor with Terraform by using the chronosphere_monitor type followed by a name in a resource declaration:

Terraform example

resource "chronosphere_monitor" "instance_down" {
  name = "Instance down"

  # Reference to the collection the alert belongs to
  collection_id = chronosphere_collection.infra.id

  # Override the notification policy.
  # By default, uses the policy from the previously stated collection_id.
  notification_policy_id = chronosphere_collection.infra_testing.id

  # Arbitrary set of labels to assign to the alert
  labels = {
    "priority" = "sev-1"
  }

  # Arbitrary set of annotations to include in alert notifications
  annotations = {
    summary     = "Instance {{ $labels.instance }} is down"
    description = "Container {{ $labels.namespace }}/{{ $labels.pod }}/{{ $labels.container }} terminated with {{ $labels.reason }}."
    value       = "Current value: {{ $value }}"
    runbook     = "http://default-runbook"
  }
# ...
}

In the annotations section, define the annotations for your monitor.
Run terraform apply to create the monitor resource:
terraform apply

Cortex XCOR

Palo Alto Networks Telemetry Pipeline

More information

Add annotations to a monitor

​Add annotations to a monitor

Add annotations to a monitor