> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Respond to incidents
> A workflow guide for on-call engineers who receive, triage, investigate, and resolve alerts in Palo Alto Networks Cortex XCOR.
export const AiMessageIcon = props => ;
When an alert triggers, time matters. This guide walks through the end-to-end
workflow for on-call engineers: from receiving a notification, to triaging scope and
impact, to investigating root causes across telemetry types, to silencing noise and
documenting resolution.
```mermaid theme={null}
graph LR
accTitle: Responder workflow stages
accDescr: A five-step flow from receiving an alert, to assessing severity, investigating root cause, reducing noise, and collaborating to resolve.
A[Receive alert] --> B[Assess severity & scope] --> C[Investigate root cause]
C --> D[Reduce noise] --> E[Collaborate & resolve]
click A href "#step-1-receive-and-review-the-alert"
click B href "#step-2-assess-severity-and-scope"
click C href "#step-3-investigate-root-cause"
click D href "#step-4-reduce-noise-while-you-work"
click E href "#step-5-collaborate-and-resolve"
classDef greenFill fill:#60E684,stroke:#000,stroke-width:2px,color:#000000
class A,B,C,D,E greenFill;
```
## Step 1: Receive and review the alert
Palo Alto Networks Cortex XCOR routes alert notifications through the channels
your team has configured, such as PagerDuty, OpsGenie, Slack, email, or a webhook.
When a notification arrives, it links back to the triggering alert in Cortex XCOR.
If you're already in the product, view your
[personal home page](/observe/personal-homepage) for recently triggered alerts,
recent comments and notes. A separate **Favorites** panel lists resources you've
added as favorites. Global search (`Control+K` or `Command+K` on macOS) also jumps
directly to a monitor, dashboard, or service by name.
To find active alerts directly in the product:
1. In the navigation menu, select **Alerting > Alerts**.
2. Review the **Status** column to distinguish **Critical**, **Warning**,
**Muted**, and **Resolved** alerts.
3. Click an alert's title to open its
[alert details](/investigate/alerts/alert-details) page.
The alert details page displays the following information:
* The alert's current status and the triggering
[signals](/investigate/alerts/notifications/signals)
* A time series chart of the query that triggered the alert, with optional
threshold overlays
* The conditions that triggered, including operator, threshold, and sustain duration
* Any [change events](/observe/enable-events/use-events) that correlate with
the alert window
* The alert's owner [collection](/administer/collections) and responsible
[team](/administer/accounts-teams/teams)
* [Annotations](/investigate/alerts/monitors/monitor-annotations) defined on the
monitor, such as runbook links and related dashboard links
Compare the **Stored data** and **Evaluated data** tabs before forming a
hypothesis. **Stored data** shows the values currently in the database, while
**Evaluated data** shows the values recorded when the alerting engine evaluated
the monitor. A late-arriving data notice identifies cases where ingestion delays
changed the stored result after evaluation.
Annotations are the primary way to surface context for incident response, such as
runbooks, related dashboards, data links to traces, and contact routing. If these
links are present, start there. The people who built the monitor placed them to
accelerate exactly this kind of triage.
## Step 2: Assess severity and scope
Before running queries, establish the scope to understand the surface area you're
dealing with.
### Identify the affected service and team
The **Alert information** sidebar on the alert details page lists the owning
[collection](/administer/collections) and
[team](/administer/accounts-teams/teams) responsible for the alert's source.
If the alert is about a service you don't own, use this to identify who to
contact.
Click **Source monitor** or **Source SLO** to open the entity that defined the
alert. From there, review the full monitor configuration, its notification
policy, and its signal grouping.
### Understand blast radius with the dependency map
If the alert originates from a known service, open the
[service page](/observe/services/service-pages) to see upstream and downstream
dependencies:
1. In the navigation menu, select
**Go to Admin > System
Overview > Services**,
then click the service name.
2. In the **Dependency map**, examine the arrows on edges between services.
Single, double, and triple arrows indicate increasing trend magnitude for
duration, requests, and errors.
3. Click a service in the map to see per-service totals for requests, errors,
leaf errors, median duration, and outlier duration.
The dependency map shows upstream and downstream relationships without
requiring prior knowledge of the topology.
### Scan for related alerts
Return to **Alerting >Alerts** and scan the
list for other alerts that triggered in the same time window. Alerts occurring across
multiple services suggest a broad infrastructure or network issue rather than a
code-level regression in a single service.
Adjust the time range on the alert details page to widen or narrow the window
and confirm whether the alert is ongoing or already resolving.
## Step 3: Investigate root cause
Finding the root cause of an issue can be complicated, so Cortex XCOR
provides numerous tools to help target the problem and resolve it.
### Start with Cortex XCOR Differential Diagnosis (DDx) on metrics
[DDx](/investigate/analyze/differential-diagnosis)
scans all label-value combinations for a metric to find which ones correlate
most strongly with the original query. Use DDx to convert a noisy alert signal
into a specific hypothesis without manually comparing dozens of series.
From an alert details page:
1. In the **Alert details** panel, click **Analyze anomaly (DDx)**. On narrower
screens, open the three vertical dots menu
first.
2. On the **Metrics Differential Diagnosis (DDx)** page, review the
**Correlation summary** for the label-value combinations with the strongest
correlation to the query.
3. Use those labels as starting filters for deeper investigation in
[Metrics Explorer](/investigate/querying/metrics/explorer).
From a service dependency map:
1. Click a service node or edge, then click the
three vertical dots icon next to the trend statistic you want to investigate.
2. Select **Differential Diagnosis**.
### Apply DDx to traces
If the alert involves latency or error-rate spikes and your service emits
traces, run
[DDx](/investigate/analyze/differential-diagnosis/traces) in Trace
Explorer. DDx scans all `tag:value` pairs across spans to identify which
combinations are unusually correlated with errors or high latency, such as a
specific `deployment.version` or `host.region`.
1. In Trace Explorer, define a search that captures the problematic traffic.
2. Click the **Differential Diagnosis (DDx)** tab.
3. Select a comparison period from **Compare to past**, such as **1 hour prior**,
to compare errors, successes, and durations for selected tags.
### Use Metrics Explorer to drill into queries
Click **Open in explorer** from the alert details page to open
[Metrics Explorer](/investigate/querying/metrics/explorer) pre-populated with
the triggering query. From there:
* Adjust label filters to isolate specific hosts, pods, or regions
* Use the [Query Builder](/investigate/querying/metrics/query-builder) to
modify the query without writing PromQL from scratch
* Toggle the time range to compare current behavior to a prior window
### Check related dashboards
If the monitor's [annotations](/investigate/alerts/monitors/monitor-annotations) link
to a dashboard, open it for a pre-built operational view of the affected
service. Dashboards combine multiple panels, including time series, gauges, tables,
and service topology, that the team curated for exactly this kind of
investigation.
Dashboards are also available through:
* The service page's
[connected resources](/observe/services/extending-services#view-connected-resources)
* Your [personal home page](/observe/personal-homepage), which lists recent and
favorite dashboards
* Global search (`Control+K` or `Command+K` on macOS), filtered to
**Dashboards**
From a dashboard panel with a Prometheus query, use the three vertical dots menu to
open the query in Metrics Explorer for analysis, or
[add the panel to a notebook](/navigate/notebooks) to preserve it as evidence.
For more information, see [Dashboards](/observe/dashboards).
### Compare to a previous time period
Use the [**Compare**](/navigate/time-ranges#compare-current-data-to-past-data)
option in the time range selector to overlay data from a prior period, for
example, the same time window one week ago. This reveals whether the current
behavior is anomalous or matches a recurring pattern, such as a weekly
traffic cycle.
On service pages, drag a region in any chart to synchronize all panels to that
time window, making cross-signal correlation faster.
### Correlate across telemetry types
Cortex XCOR connects metrics, traces, and logs. To carry context
forward when moving between them:
* From a service page, click **Explore trace data** in the dependency map to
open [Trace Explorer](/investigate/querying/traces) pre-scoped to that
service.
* From Trace Explorer, identify leaf error spans. These are errors with no
failing child span, which can indicate a potential source of a failure
rather than a propagated error.
* Use [pinned scopes](/navigate/pinned-scopes) to carry label filters such as
`environment=production` across Metrics Explorer, supported service-page
panels, and dashboards without re-entering them on each page.
If your service page includes a **View logs** link, open
[Logs Explorer](/investigate/querying/query-logs) scoped to the same service.
This approach is effective when the metric alert is caused by an
app-level error that only surfaces in log output.
### Check change events for correlated deployments
[Change events](/observe/enable-events/use-events) appear as vertical markers
on time series charts across service pages, dashboards, and Metrics Explorer.
Look for markers that align with the start of the anomaly. A deployment,
config change, or feature flag flip often correlates directly with the alert.
To search across all recent changes, open
[Changes Explorer](/investigate/querying/filter-events) and filter by time
range, service, or event source. Change events enabled on
[dashboards](/observe/enable-events/dashboards) and
[service pages](/observe/enable-events/services) also display inline without
additional configuration.
### Generate queries with natural language
If you're unfamiliar with the affected service's metrics or don't know the
exact PromQL or log query syntax, use
[natural language queries](/investigate/querying/natural-language) to
describe what you're looking for. Click **Edit with AI** in Metrics Explorer or
Logs Explorer (or press `Control+I` or `Command+I` on macOS) and enter a prompt
such as:
> error rate for checkout service in us-east-1 last hour
Cortex XCOR generates a query using
[semantic metric search](/investigate/querying/natural-language#semantic-metric-search)
to match relevant metrics by intent rather than exact name.
### Ask Operator
This feature is in Early Access (EA), and might not be visible in your app. To learn
more about this program and the features it contains, see the
[Early access](/early-access) page.
When you need an answer rather than a query, or when a question spans several
telemetry types, ask [Operator](/navigate/operator). Click
** Ask Operator** in the page header or press A.
Because Operator reads the page you're on,
you can open it from an alert or dashboard and ask about that context directly:
Why did this alert trigger, and what changed in the 30 minutes beforehand?
Operator runs the searches and queries it needs, displays each step, and
summarizes what it found. Reach for it in this step when you want to correlate
metrics with logs, traces, and change events in one pass instead of pivoting between
explorers manually.
This feature isn't available to all Palo Alto Networks Cortex XCOR users and
might not be visible in your app. For information about enabling this feature in your
environment, contact [Cortex XCOR Support](/support).
For longer analysis, ask it to start an investigation. The investigation runs in the
background while you keep triaging, and Operator reports the result when it
finishes. See
[Start an investigation](/navigate/operator/capabilities#start-an-investigation).
To get better answers, see
[Write effective prompts](/overview/generative-ai#write-effective-prompts).
## Step 4: Reduce noise while you work
Alerts that are triggering while you investigate can generate repeated notifications.
Muting the right alerts without over-muting lets you work without distraction.
### Create a targeted muting rule
From the alert details page, click **Mute alert** to open a muting rule pre-populated
with matchers for the alert's source and signal labels. This approach derives the
scope from the specific alert rather than relying on manual entry.
Before saving, the **Preview alerts** panel shows exactly which alerts the rule will
silence. Review this list to confirm you're not accidentally muting alerts from
unrelated services.
Review the pre-populated matchers and remove any that don't belong in the scope.
To mute only a specific pod or environment, add or retain its label matcher. Use
regular expression matching when you need to mute a pattern of related series.
Set a duration that matches your expected resolution window. Extend or expire the
rule early as needed.
After the incident resolves, expire the rule immediately from
**Alerting > Muting Rules** rather than
letting it run to its scheduled end time.
### Verify notification routing
If you're unsure whether notifications are reaching the right people, open the
source monitor and click **Test notifications** to send a synthetic alert
through the same routing logic as a real alert. This confirms that notifiers
and templates are working as expected without waiting for another real event
to fire.
## Step 5: Collaborate and resolve
Use notebooks, comments, and resolution notes to share investigation context with
other responders and create a permanent record of what happened and how you fixed it.
### Gather evidence in a notebook
As you investigate, add charts, query results, and dashboard panels to a
[notebook](/navigate/notebooks) to build a running evidence file. Click the
**Notebook** icon in the page header or right sidebar to open one. From any panel
or explorer result, click the three-dots menu and select **Add to notebook**, or
drag panels directly from a dashboard into the open notebook.
Notebooks support:
* Panels from dashboards, Metrics Explorer, Logs Explorer, DDx, alert details,
SLO charts, service pages, monitors, and the metrics catalog
* A dedicated **Add to notebook** action in Logs Explorer that adds both a log
volume histogram and the current results visualization
* Copying a URL from any resource page and pasting it into the notebook to embed a
link card
* Per-panel time range overrides, so you can compare the same chart at different
points in time
* [Snapshots](/navigate/notebooks#take-a-snapshot) to freeze a panel's data for
long-term reference
* [Panel editing](/navigate/notebooks#edit-panel-contents) to refine queries
without leaving your evidence file
* [Version history](/navigate/notebooks#view-version-history) to review or restore
earlier states of the notebook
* [Operator editing](/navigate/notebooks#edit-a-notebook-with-operator), so
you can ask it to summarize what you've gathered or add a chart while you work
This feature isn't available to all Palo Alto Networks Cortex XCOR users and
might not be visible in your app. For information about enabling this feature in your
environment, contact [Cortex XCOR Support](/support).
* Completed investigation reports (**Save as notebook**). See
[Save an investigation report](/navigate/notebooks#save-an-investigation-report).
Share a notebook by clicking **Copy URL** to give other responders a single link to
your full investigation context. If another responder edits a shared notebook,
Cortex XCOR prompts you to reload so both sides stay in sync.
### Add comments for active collaboration
[Comments](/navigate/comments) let you annotate monitors, metrics, and change events
with notes visible to anyone viewing that resource. From the monitor's page, click
**Add comment** to leave notes for other responders working the same incident.
Comments also persist on metrics in Metrics Explorer and on change events in Changes
Explorer. Use them to record institutional knowledge such as
**"this metric spikes every Tuesday during batch processing"** so the next responder
doesn't retrace the same investigation.
### Document the resolution
When the alert resolves, add a
[resolution note](/investigate/alerts/alert-details#document-an-alerts-resolution)
on the alert details page. Resolution notes:
* Accept Markdown, so you can include code snippets, links, and structured text
* Associate with specific signals so notes for different affected services stay
organized
* Persist after the alert closes, creating a searchable record for
post-incident reviews
To add a resolution note:
1. On the alert details page, click **+ Add** in the **Resolution notes**
section.
2. Enter a description of what you found and what action you took.
3. Click **Create**.
### Annotate related change events
If a deployment or configuration change correlates with the alert,
[create a change event](/observe/enable-events/use-events#create-change-events)
from the monitor page. Change events appear in time series charts across
Cortex XCOR, making the correlation visible to anyone who views
that time window in the future.
## What to do after the incident
The incident response workflow doesn't end at resolution. After closing the incident,
consider taking the following actions to help other users remediate future issues
with less investigation:
* Improve monitor signal quality: if the alert triggered but required significant
manual triage, [edit the monitor's annotations](/investigate/alerts/monitors/monitor-annotations)
to add runbook links, dashboard links, or data links to relevant traces for the
next responder.
* Adjust thresholds: if the alert was noisy, triggering frequently on non-issues,
review the monitor's
[conditions and sustain duration](/investigate/alerts/monitors/data-model) to
better reflect real risk.
* Review the
[alert history](/investigate/alerts/monitors/monitor-details#alert-history)
for the monitor to see whether this alert has triggered repeatedly, which
might indicate a systemic issue worth addressing at the code or infrastructure level.
* [Favorite](/navigate/favorites) the dashboards and monitors you used during the
incident. Favorites appear on your personal home page and surface first in global
search, reducing navigation time during future incidents.
* Add [change events](/observe/enable-events/use-events#create-change-events) to mark
the deployment or configuration change that caused the issue, if one hasn't already
been created. These markers persist on time series charts for future responders.
## Related topics
- [Case studies](/overview/case-studies/case-studies.md)
- [Cortex XCOR guides](/overview/guides.md)
- [Alert Response & Troubleshooting](/academy/courses/alert-response-and-troubleshooting.md)
- [Collections](/administer/collections.md)
- [Manage telemetry data budgets](/overview/guides/manage-budgets.md)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.