> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Case study: responding to incidents

> A narrative walkthrough of an on-call engineer investigating and resolving a production incident using Palo Alto Networks Cortex XCOR.

When an alert triggers, time matters. This case study follows an on-call engineer
through a real-world incident, from the initial page to resolution, demonstrating how
Palo Alto Networks Cortex XCOR tools connect to form a complete investigation
workflow.

For a step-by-step reference version of this case study, see the
[Incident response guide](/overview/guides/incident-response).

## The summons

The phone blared at 2:17 AM.

Not the polite chirp of a Slack mention or the quiet nudge of an email. This was the
PagerDuty alarm, jarring Alex Reeves out of sleep. Knowing she was on call, she
reached for the phone with the muscle memory of someone who has done this a hundred
times before, squinting at the screen:

> **CRITICAL: checkout-service - Error rate exceeded 5% threshold for 3 minutes**
>
> *Monitor: checkout-error-budget-burn*
> *Collection: Commerce Platform*
> *Team: Payments*

Alex was on her feet before she finished reading. Laptop open, password entered,
Cortex XCOR loading in the dark.

## The first thread

Alex's [personal home page](/observe/personal-homepage) confirmed the bad news: the
**Recently triggered alerts** panel showed not one, but three alerts triggering in
the last 10 minutes. The checkout error monitor was the oldest, and the others
belonged to `payment-gateway` and `order-service`. Three fires, all at once.

Alex clicked the checkout alert title, and the
[alert details page](/investigate/alerts/alert-details) showed the situation.

The time series chart was unambiguous. At 2:14 AM, the error rate for
`checkout-service` had jumped from its baseline of 0.3% to 7.2%, blowing past the 5%
threshold. The condition showed `> 5% for sustain duration 3m`, confirming this
wasn't a flapping alert. The [signals](/investigate/alerts/notifications/signals)
grouped the alert by `environment=production, region=us-east-1`, while `us-west-2`
remained calm. Whatever this was, it was regional.

Alex compared the **Stored data** and **Evaluated data** tabs. Both showed the same
spike, and no late-arriving data notice appeared. The alerting engine had evaluated
the same values that were currently stored.

The **Alert information** sidebar included a runbook titled "Checkout Error Rate
Playbook." In
[**Additional information**](/investigate/alerts/alert-details#view-an-alerts-related-information),
Alex found a link to the **Commerce Checkout Dashboard**. The dashboard link would
provide context. But first, she needed to understand scope.

## Mapping the blast radius

Alex navigated to the [service page](/observe/services/service-pages) for
`checkout-service`. The **Dependency map** showed `frontend` upstream, calling
into `checkout-service`, which fanned out to `payment-gateway`,
`inventory-service`, and `order-service` downstream. The arrows on the edge
between `checkout-service` and `payment-gateway` changed based on the selected
trend. She selected **Errors** from **Trends**. The edge showed a triple upward
indicator, the strongest increase.

She clicked the `payment-gateway` node. The overlay showed:

* Requests: 1,240 (normal)
* Errors: 312 (elevated, normally \~8)
* Outlier duration (P99): 4,200 ms at the ninety-ninth percentile, compared
  with a normal 180 ms

Something was deeply wrong between checkout and payments. But was
`payment-gateway` the cause, or was the issue elsewhere?

She set a [pinned scope](/navigate/pinned-scopes) for
`environment=production` and `region=us-east-1`, so supported pages would carry that
context without her having to re-enter it.

## Cortex XCOR Differential Diagnosis (DDx)

Back on the alert details page, Alex clicked **Analyze anomaly (DDx)** on the error
rate chart. [DDx](/investigate/analyze/differential-diagnosis)
scanned every label-value combination for
`checkout_http_requests_total{status=~"5.."}` and compared each result with the
original query.

The **Correlation summary** listed `endpoint=/v2/process-payment` and
`deployment.version=v3.41.0` among the strongest correlations. The `pod` label had
weaker correlations because the errors affected every pod in the deployment.

The `/v2/process-payment` endpoint was the epicenter. Alex also noted
`deployment.version=v3.41.0`. That was new. She hadn't seen that version before
tonight. A deployment, rolled out while she slept.

Alex added the DDx panel to her [notebook](/navigate/notebooks) by clicking the
three-dots icon and selecting **Add to notebook**, naming it
`2 AM Checkout Incident`. The evidence trail had begun.

## Into the metrics

Alex clicked **Open in explorer** to carry the alert query into
[Metrics Explorer](/investigate/querying/metrics/explorer). The chart appeared
pre-populated:

```text wrap theme={null}
sum(rate(checkout_http_requests_total{status=~"5..", environment="production", region="us-east-1"}[5m]))
/
sum(rate(checkout_http_requests_total{environment="production", region="us-east-1"}[5m]))
```

Alex added a filter: `endpoint="/v2/process-payment"`. The error rate jumped to 23%
for that single endpoint, which was far worse than the aggregate. She used the
[Query Builder](/investigate/querying/metrics/query-builder) to split by `status`
code and found the culprit: HTTP `503 Service Unavailable` status codes
dominated, with a smattering of HTTP `504 Gateway Timeout` status codes.

She added this chart to her notebook, the second piece of evidence.

## The dashboard reveals the pattern

Following the annotation link from the monitor, Alex opened the
**Commerce Checkout Dashboard**. The [dashboard](/observe/dashboards) showed
panels for request rates, error breakdowns, latency percentiles, and an
**Upstream Dependencies** panel showing the health of every service that
`checkout-service` calls.

The `payment-gateway` panel was solid red. Latency had spiked from 180 ms to
over four seconds at 2:14 AM, and the error panel showed `payment-gateway`
returning HTTP `503 Service Unavailable` status codes to every caller. But
on the dashboard she noticed something the alert details hadn't shown: a
vertical [change event](/observe/enable-events/use-events) marker at
2:12 AM, two minutes before the errors began.

She held her pointer over the marker:

> **Deploy: checkout-service v3.41.0**
> *Source: ArgoCD*
> *Author: deploy-bot*
> *Region: us-east-1*

There it was. A deployment at 2:12 AM. The errors followed two minutes later at
exactly the time it takes for pods to roll through their readiness probes. Alex added
the dashboard's error panel to her notebook, capturing a
[snapshot](/navigate/notebooks#take-a-snapshot) so the data would persist even after
the time window scrolled past.

## Following the trace

Alex needed to understand *why* `v3.41.0` was causing HTTP 503 errors. She returned to the
service page and clicked **Explore trace data** in the dependency map, opening
[Trace Explorer](/investigate/querying/traces) pre-scoped to `checkout-service`.

She defined her search:

* Service: `checkout-service`
* Status: Error
* Tag: `deployment.version = v3.41.0`

Results flooded in, showing hundreds of failed traces in the last 15 minutes.
She switched to the **Span statistics** tab and grouped by service and operation. The
[leaf errors](/investigate/querying/traces/features) told the story: while
`checkout-service` showed errors on `POST /v2/process-payment`, the leaf errors (the
deepest failing spans with no failing children) clustered on `payment-gateway` at the
operation `grpc.PaymentProcessor/ChargeCard`.

Alex clicked into a single trace. The waterfall rendered the request's path:
`frontend` called `checkout-service`, which called `payment-gateway.ChargeCard`. The
`ChargeCard` span was red, 4,100 ms long, and terminated with:

> `error.message: "connection refused: payment-provider.internal:443"`

The checkout service's `v3.41.0` deployment had changed the payment provider
URL, and the new URL wasn't resolving in `us-east-1`.

She ran [trace DDx](/investigate/analyze/differential-diagnosis/traces) against the
failing spans and selected **1 hour prior** from **Compare to past**. Error spans in
the current window were concentrated on `deployment.version=v3.41.0` and
`net.peer.name=payment-provider.internal`. The second tag didn't appear in the
comparison period.

## The logs confirm it

To confirm the DNS hypothesis, Alex opened
[Logs Explorer](/investigate/querying/query-logs) from the service page's
**View logs** link. Filtered to `service=checkout-service` and `severity=ERROR`, the
logs appeared:

```text theme={null}
2026-06-02T06:15:42Z [ERROR] checkout-service/payment_client.go:87
  dial tcp: lookup payment-provider.internal on 10.96.0.10:53:
  no such host
```

This was repeated hundreds of times. The `v3.41.0` deployment had
introduced a new payment client that pointed to `payment-provider.internal`,
a hostname that existed in the staging DNS but not in production's
`us-east-1` zone. The old endpoint, `payment-api.prod.internal`, worked
fine. This was a configuration error baked into the release.

Alex added the log line to her notebook with a markdown note:

> "Root cause: v3.41.0 references staging DNS hostname `payment-provider.internal`
> which doesn't resolve in prod us-east-1."

## Silencing the noise

With a root cause identified, Alex needed breathing room. PagerDuty had paged her
twice more during the investigation. She returned to the alert details page and
clicked **Mute alert**, creating a [muting rule](/investigate/alerts/muting-rules)
pre-populated with the source monitor and signal labels. She set a 60-minute
duration. The **Preview alerts** panel confirmed it would silence only the checkout
error alert, not the downstream payment-gateway alerts that her colleague on the
Payments team might need to see.

## The fix and the record

Alex knew the fix: rollback `checkout-service` to `v3.40.2`. She triggered the
rollback through their deployment pipeline, then turned to documentation.

On the monitor page, she clicked **Add comment** to leave a
[comment](/navigate/comments) for her Payments team colleague:

> "Root cause identified: v3.41.0 introduced `payment-provider.internal`
> hostname which doesn't exist in prod us-east-1 DNS. Rolled back to v3.40.2.
> The payment-gateway errors are downstream of this and should resolve within 5
> minutes of rollback completing."

She created a [change event](/observe/enable-events/use-events#create-change-events)
to mark the rollback

> **Rollback: checkout-service v3.40.2**
> *Reason: v3.41.0 DNS configuration error causing HTTP 503 errors*

Within four minutes of the rollback completing, the error rate dropped back to
0.3%. The alert resolved itself. Alex added a
[resolution note](/investigate/alerts/alert-details#document-an-alerts-resolution)
on the alert details page:

> "**Root cause:** checkout-service v3.41.0 deployment at 2:12 AM introduced a
> reference to `payment-provider.internal`, a hostname that resolves only in
> staging DNS. Production us-east-1 pods couldn't reach the payment provider,
> causing HTTP 503 status codes on `/v2/process-payment`. Rolled back to v3.40.2 at 2:48 AM.
> Error rate returned to baseline by 2:52 AM."

She shared her notebook URL in the incident Slack channel, expired the muting rule
from **Alerting > Muting Rules**, and added the Commerce Checkout Dashboard as a
[favorite](/navigate/favorites) so it would surface faster next time.

## After

At 3:01 AM, Alex closed her laptop. Forty-four minutes from page to resolution.
Every step documented, every chart preserved in the notebook, the deployment
marked with a change event that would warn the next engineer who looked at that
time window.

In the morning, the team would find her trail. The comment on the monitor, the
resolution note on the alert, the notebook with its chain of evidence from DDx
through Trace Explorer to the log line. The team would know *what* happened, and
*how* she found it. The deployment pipeline would gain a DNS validation step before
the next release reached production.


## Related topics

- [Case studies](/overview/case-studies/case-studies.md)
- [Case study: protecting a telemetry data budget](/overview/case-studies/manage-budgets.md)
- [Palo Alto Networks Cortex XCOR](/overview.md)
- [Respond to incidents](/overview/guides/incident-response.md)
- [Service level objectives](/observe/slo.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.