> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Case study: creating a service-health dashboard

> A narrative walkthrough of an engineer building a reusable service-health dashboard with variables, panels, and events in Palo Alto Networks Cortex XCOR.

A dashboard built for one incident often fails the next one. This case study follows an
engineer on the Commerce Platform team as he replaces four competing service dashboards
with one reusable [standard dashboard](https://docs-xcor.paloaltonetworks.com/observe/dashboards) in Palo Alto Networks
Cortex XCOR. The walkthrough shows how dashboard variables, panel types chosen for
each data shape, Prometheus Query Language (PromQL) queries, change events, and
dashboard annotations fit together to give a team one shared view of service health.

For a step-by-step reference version of this case study, see the
[Create dashboards for service health guide](https://docs-xcor.paloaltonetworks.com/overview/guides/create-dashboards.md).

> [!NOTE]
>
> Prefer a guided lab? Take
> [Building Dynamic Dashboards](https://docs-xcor.paloaltonetworks.com/academy/courses/building-dynamic-dashboards.md) to
> practice panel types, queries, variables, and dashboard layout.

## Four dashboards and no single answer

The action item came out of a Friday postmortem, though the problem was much older.

Marcus Chen took the Monday morning rotation for the Commerce Platform team. Three weeks
had passed since the 2:17 AM outage in the
[incident response case study](https://docs-xcor.paloaltonetworks.com/overview/case-studies/respond-to-incidents.md), and a week
had passed since the log spike in the
[budget protection case study](https://docs-xcor.paloaltonetworks.com/overview/case-studies/manage-budgets.md). Before his shift
started, he opened **Dashboards** and filtered the table by the `Commerce Platform`
[collection](https://docs-xcor.paloaltonetworks.com/administer/collections) to see what an on-call engineer would find.

Four dashboards displayed in the results:

| Dashboard name | Owner | Problem |
| - | - | - |
| `Commerce Checkout Dashboard` | `checkout-service` | Hardcoded to `checkout-service` in `production` |
| `Checkout Triage - Copy` | `Commerce Platform` | Cloned during an incident, using 1-minute rate windows |
| `Payment Gateway Overview` | `payment-gateway` | Tracked average latency instead of histogram buckets |
| `Order Service HTTP` | `order-service` | Plotted 40 endpoints on one line chart |

Every dashboard had been built by an engineer trying to help their teammates, and
together they made triage slower. A responder investigating an issue that crossed from
`checkout-service` into `payment-gateway` had to open two dashboards with different
layouts, different time ranges, and different definitions of error rate. Worse, none of
the panels could be reused for `order-service` without copying the dashboard and
rewriting every query by hand.

Marcus wanted one standard dashboard that worked across all three services in the
collection. Before touching a query, he wrote down the six questions a responder needed
the dashboard to answer in the first minute:

* Is the service receiving traffic?
* Are requests succeeding?
* Is latency within an expected range?
* Which endpoint, instance, or other dimension needs attention?
* Did a deployment or other change coincide with the behavior?
* Where can the viewer continue investigating an issue?

## Normalizing the telemetry before building

A reusable dashboard depends on consistent telemetry across the services it covers.
Before creating the dashboard, Marcus opened
[**Metrics Explorer**](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/explorer.md) and checked whether
`checkout-service`, `payment-gateway`, and `order-service` exposed the same HTTP
request metrics and labels.

All three services emitted `http_requests_total` and
`http_request_duration_seconds_bucket`, and all three carried `environment`, `status`,
and `endpoint` labels. However, an older gRPC adapter in `order-service` still emitted
its service identifier on a `svc` label instead of `service` for a subset of internal
routes. If Marcus built a dashboard variable on the `service` label alone, those routes
would never appear when a responder selected `order-service`.

Rather than waiting on a release to rename the label in service code, Marcus checked
the [**Attributes Catalog**](https://docs-xcor.paloaltonetworks.com/investigate/system-library/attributes-catalog.md) and verified that
a [derived label](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/derived-telemetry/derived-labels.md)
(also called a **mapped label**) already mapped `svc`
into `service` at query time. He also checked the
[**Metrics Catalog**](https://docs-xcor.paloaltonetworks.com/investigate/system-library/metrics-catalog.md) for the retry counter
created during the log budget cleanup, `checkout_payment_retry_total`. That review showed
which metrics belonged to every service and which belonged only to the checkout service.

With the label schema confirmed, Marcus created the dashboard:

1. In the navigation menu, he selected **Dashboards**, clicked **+ Add**, and selected
   **New dashboard**.
2. In the **Add dashboard** dialog, he entered `Commerce service health` in the
   **Dashboard name** field.
3. He set **Owner** to the `Commerce Platform` collection so the team that maintained
   the three services owned the dashboard together, and clicked **Save**.

He set the default [time range](https://docs-xcor.paloaltonetworks.com/navigate/time-ranges.md) to **Last 30 minutes**. Thirty
minutes was short enough to make an active incident stand out and wide enough to show
the baseline before a deploy.

Next, he planned the visual hierarchy with three
[panel groups](https://docs-xcor.paloaltonetworks.com/observe/dashboards/customization/layouts.md) so a responder who knew the
services, but hadn't memorized the dashboard, could read from top to bottom:

* **Overview**: Current request rate, error rate, and availability at a glance
* **Trends**: Request volume and error behavior over time
* **Diagnostics**: Latency distribution across buckets and a ranked table of endpoints

He clicked **Edit** on the dashboard, clicked **Panel Group** to create each group, and
kept all three groups set to **Open** by default.

## Turning one dashboard into three with variables

Hardcoded selectors were the reason the team had four dashboards instead of one. Marcus
added [dashboard variables](https://docs-xcor.paloaltonetworks.com/observe/dashboards/customization/dashboard-variables.md)
before building any panels so every query would share the same filtering pattern from
the start.

He clicked **Variables** to open the **Dashboard variables** side panel, clicked
**Add variable**, and configured two list variables:

* `service`: He set **Type** to **List** and **Source** to
  **\[Prometheus] Label values**, selected the `service` label, and added a series
  selector of `http_requests_total` so the dropdown listed only services that emitted
  HTTP request metrics in the tenant.
* `environment`: He set **Type** to **List** and **Source** to
  **\[Prometheus] Label values**, selected the `environment` label, and scoped it with
  `http_requests_total{service=~"$service"}`.

On both variables, he enabled **Select multiple values** and **Include "All" option**,
and set **Refresh** to **On time range change** so the dropdown options reflected the
active series in the selected window. Because a viewer could select more than one
service or choose **All**, every panel query would use regular expression matchers:

```text theme={null}
{service=~"$service", environment=~"$environment"}
```

Naming the variables `service` and `environment` to match the underlying Prometheus
label names gave the team a second shortcut. When a responder pinned a tooltip on any
[time series chart](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/time-series-chart.md#filter-a-dashboard-by-a-series-value),
the series action menu included **Filter service → `checkout-service`** and
**Filter environment → `production`**. Clicking either action narrowed the entire
dashboard from a single spiking line without scrolling back to the header.

Marcus considered adding an [interval variable](https://docs-xcor.paloaltonetworks.com/observe/dashboards/customization/intervals.md)
and decided against it. Cortex XCOR calculates query intervals from the time range and
panel resolution automatically. A manual interval variable lets a viewer pick a step
smaller than the 15-second scrape interval, which produces empty gaps in rate
calculations. Instead, he used `$__rate_interval` in every counter query so the
lookback window scaled with the dashboard time range and scrape interval.

After clicking **Apply** and saving the dashboard, the **Saved View** menu appeared in
the header. Marcus selected `service = checkout-service` and `environment = production`,
opened the **Saved View** menu, and clicked **Create new view** to save a view named
`Checkout - Production`. He repeated the step for `Payment Gateway - Production` and
`Order Service - Production` so on-call engineers could switch services in two clicks.

## Matching each panel type to the shape of the data

Each question on Marcus's list returned a different shape of data: a single current
ratio, a set of continuous rates over time, a histogram bucket distribution, and a
high-cardinality list of endpoint labels. Choosing the wrong visualization had made the
team's old dashboards hard to read. Marcus chose a distinct
[panel type](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels#panel-types) for each data type.

The following table summarizes how he matched each question and data type to a panel
type:

| Question | Data shape | Panel type | Group |
| - | - | - | - |
| What's the current error rate? | Single scalar percentage reduced from counter rates | [Stat chart](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/stat-chart.md) | **Overview** |
| When did request or error rates change, and for how long? | Continuous rate series grouped by status over time | [Time series chart](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/time-series-chart.md) | **Trends** |
| Is latency clustering normally or growing a slow tail? | Cumulative histogram buckets (`le`) over time | [Heat map](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/heat-map.md) | **Diagnostics** |
| Which specific endpoint handles the most traffic right now? | High-cardinality categorical label (`endpoint`) with exact values | [Table](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/table.md) | **Diagnostics** |

### A stat chart for a single current health ratio

In the **Overview** group, a responder needed to know immediately whether the selected
service was failing requests. That answer is a single current number, the percentage of
HTTP `5xx` responses, rather than a multi-line graph that forces the viewer to read a
Y-axis under stress.

Marcus added a panel named `Error rate (%)` to the **Overview** group, selected
**Stat Chart** from the **Type** dropdown, selected the **Metrics (Prometheus)**
datasource, and entered the [PromQL](https://docs-xcor.paloaltonetworks.com/investigate/querying/promql) ratio query:

```text wrap theme={null}
100 *
sum(rate(http_requests_total{service=~"$service", environment=~"$environment", status=~"5.."}[$__rate_interval]))
/
sum(rate(http_requests_total{service=~"$service", environment=~"$environment"}[$__rate_interval]))
```

In the query **Options**, he set **Query type** to **Instant** so the stat tile
evaluated the latest rate window. In the **Settings** tab, he enabled **Sparkline** to
show the recent direction of the metric beneath the number. He then set the unit in
**Unit and formatting** to **Percent (0-100)** with two decimal places.

In the **Thresholds** section, he added thresholds at `1` (orange) and `5` (red),
matching the team's warning and critical alert conditions. To ensure color wasn't the
only indication of health, he expanded each threshold row and entered `Healthy` in the
base **Message** field, `Elevated (Warning >= 1%)` on the `1` threshold, and
`Breaching SLO (Critical >= 5%)` on the `5` threshold. He added two companion stat
charts alongside it in the **Overview** group for current request rate and P99 latency.

### A time series chart for continuous rates over time

Once a responder knew the error rate was elevated, the next question was temporal: when
did the change start, how fast did it climb, and did successful traffic drop at the
same time? Continuous rate series indexed by time belong on a
[time series chart](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/time-series-chart.md), where the X-axis
aligns the timing, duration, and direction of every status class across the dashboard's
time window.

In the **Trends** group, Marcus created a panel named `Request rate by status`, kept
the default **Time Series Chart** type, and entered the following PromQL query:

```text wrap theme={null}
sum by (status) (
  rate(http_requests_total{service=~"$service", environment=~"$environment"}[$__rate_interval])
)
```

Grouping `by (status)` returned a manageable number of series (`200`, `400`, `404`,
`500`, `503`, `504`) without flooding the chart. He set the series naming pattern to
`{{status}}` and set the Y-axis unit to requests per second. He also enabled the
**Legend** in **Table** mode with **Last \*** and **Max** values visible so viewers
could compare the current rate against the peak in the window. In the **Tooltip**
section, he set **Mode** to **Nearby** and **Sort** to **Descending** so holding the
pointer over a spike listed the highest-volume status codes first.

### A heat map for histogram bucket distributions

On the old `Payment Gateway Overview` dashboard, a single average latency line had
masked partial outages whenever a small fraction of requests timed out at four seconds
while fast cache hits pulled the mean down. Even a single percentile line like P95 can
hide whether latency forms one stable band or splits into two distinct clusters.

Because `http_request_duration_seconds_bucket` is a cumulative Prometheus histogram
where each series carries an upper bound in the `le` label, its data represents density
across latency buckets over time. A [heat map](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/heat-map.md) is
purpose-built for histogram bucket data: it plots time on the X-axis, bucket bounds on
the Y-axis, and request density as color intensity in each cell.

In the **Diagnostics** group, Marcus created a panel named
`Request latency distribution`, changed **Type** to **Heat Map**, and entered:

```text wrap theme={null}
sum by (le) (
  rate(http_request_duration_seconds_bucket{
    service=~"$service",
    environment=~"$environment"
  }[$__rate_interval])
)
```

In the **Settings** tab under **Y axis**, he set **Unit** to **Time** and selected
**Seconds (s)**. He kept the **Legend** enabled on the right side of the panel so the
relationship between color intensity and request count remained explicit without
relying on color perception alone. On the preview for `checkout-service`, a dense band
sat between `50 ms` and `200 ms`, while a faint upper band near `4 s` immediately
showed when downstream payment calls stalled.

### A table for high-cardinality endpoint rankings

The old `Order Service HTTP` dashboard had failed for the opposite reason: it plotted
`sum by (endpoint)` on a time series chart. Because the three services exposed dozens
of REST and gRPC endpoints, the chart rendered a tangle of overlapping lines and a
scrolling legend that hid the exact endpoint names.

When a dimension has many distinct string values and a responder needs to rank them by
exact numbers, a [table](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/table.md) fits the data type far
better than a line chart. Each series becomes one sortable row, and full label strings
remain readable in their own column.

Marcus added a panel named `Top endpoints by request rate` in the **Diagnostics**
group, selected **Table** from the **Type** dropdown, and entered:

```text wrap theme={null}
topk(
  10,
  sum by (endpoint) (
    rate(http_requests_total{
      service=~"$service",
      environment=~"$environment"
    }[$__rate_interval])
  )
)
```

In the query **Options**, he set **Query type** to **Instant** so each endpoint
returned one current value rather than a time series of points, and set the query
**Alias** to `Requests per second`. In the **Settings** tab, he kept **Time series
format** set to **Default**, set **Sort by** to `Requests per second`, and set **Sort
order** to **Descending**. On the **Table columns** tab, he set the **Display Name**
for `endpoint` to `Endpoint` and hid the timestamp column.

To give responders errors alongside traffic in the same view, he clicked **Add query**
in the table panel and added a second instant query for `status=~"5.."` aggregated
`by (endpoint)` with the alias `Errors per second`. Because both queries used the
**Default** table format and returned the same `endpoint` label set, Cortex XCOR merged
them into one row per endpoint with separate columns for `Requests per second` and
`Errors per second`.

## Overlaying change events and dashboard annotations

A chart shows when a metric moved, but a responder still needs to know what changed at
that timestamp and where to go next.

First, Marcus edited each panel to add a clear **Description** stating what the query
measured, what normal behavior looked like, and how missing data behaved. For example,
the description on `Error rate (%)` noted that the query returns no data when zero `5xx`
series exist in a quiet development environment. On the `Error rate (%)` and
`Top endpoints by request rate` panels, he opened the **Links** tab, clicked
**Add link**, and added [panel links](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels#add-links-to-a-panel)
to the team's checkout and payment runbooks and to Trace Explorer URLs that embedded
`$service` and `$environment`. For dashboard-to-dashboard links, he enabled
**Include current time range** and **Include current variable values** so clicking a
link preserved the active filter context for the responder.

Next, he added [change events](https://docs-xcor.paloaltonetworks.com/observe/enable-events/dashboards.md#standard-dashboards)
so deployments would appear directly on the time series panels:

1. On the dashboard, he clicked **More actions** and selected **Events** to open the
   **Dashboard events** side panel.

2. He clicked **Add event**, named the event layer `Production deploys`, and selected
   **Timeline events** from the **Datasource** menu.

3. In the filter box, he entered a change event query scoped to production deployments:

   ```text theme={null}
   category = "deploys" AND labels.environment = "production"
   ```

4. He clicked **Add**, clicked **Apply** in the **Dashboard events** side panel, and
   clicked **Save** on the dashboard.

Vertical deploy markers immediately appeared across the `Request rate by status` panel.
Clicking a marker pinned it and exposed **Open in Changes Explorer**, which opened the
event's **Summary** tab in [Changes Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/filter-events.md).
Because change events are stored platform-wide, the same deploy markers appeared on
service pages and in Changes Explorer for every team in Cortex XCOR.

Mid-afternoon, while Marcus was watching `checkout-service` on the new dashboard, a
brief two-minute bump appeared in the `500 ms` to `1 s` buckets on the heat map and in
the `200` status series at 2:40 PM. No deploy marker accompanied it. He checked with
the engineer running a canary test and learned she had manually increased the payment
client connection pool size in `us-east-1` to test connection reuse. That manual tuning
step didn't emit a deployment event and mattered only to people reading this dashboard.

To record that context without polluting platform-wide change events, Marcus created a
[dashboard annotation](https://docs-xcor.paloaltonetworks.com/observe/dashboards/annotations.md):

1. On the `Request rate by status` time series chart, he clicked the data point at
   2:40 PM to [pin](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/time-series-chart.md#pin-a-point-in-time)
   the tooltip.
2. In the pinned tooltip, he clicked **Create annotation**.
3. In the **Create annotation** side panel, he entered
   `Increased payment client connection pool to 200 during canary test` in the
   **Title** field and confirmed the **Timestamp** defaulted to 2:40 PM.
4. He clicked **Add Label** to attach `service = checkout-service` and
   `environment = production`, and clicked **Create**.

The annotation marker appeared on the dashboard's time series panels through the
built-in Annotations event layer. Anyone on the Commerce Platform team who looked at
the 2:40 PM bump would see the explanation immediately in the **Annotation** side
panel, while Changes Explorer stayed focused on system-wide events.

## Testing the workflow with a responder

Before retiring the old dashboards, Marcus asked Alex Reeves, who had worked the 2:17 AM
checkout outage, to review the `Commerce service health` dashboard.

Alex opened the dashboard, switched the **Saved View** menu from
`Checkout - Production` to `Payment Gateway - Production`, and then selected **All** in
the `service` dropdown. She verified each item on their review checklist:

* The **Overview** stat charts answered whether the selected service was healthy in
  under five seconds, with text messages on every threshold state.
* Selecting multiple services or **All** in the `service` and `environment` variables
  updated every panel cleanly, and pinning a series on `Request rate by status` offered
  the one-click **Filter** action for matching variables.
* The **Trends** time series chart, **Diagnostics** heat map, and endpoint **Table**
  moved from summary to timing to exact dimension values without visual clutter.
* Production deploy change events and the 2:40 PM dashboard annotation explained spikes
  in place without obscuring the lines.
* Clicking **Inspect panel** on each chart confirmed that query durations stayed fast
  at both the default 30-minute window and a 7-day comparison range.

With the review complete, Marcus connected the dashboard to the rest of the team's
observability workflow:

1. He clicked **More actions**, selected **Connections**, and
   [connected the dashboard](https://docs-xcor.paloaltonetworks.com/observe/dashboards#connect-a-dashboard-to-a-service) to
   `checkout-service`, `payment-gateway`, and `order-service` so it appeared on all
   three service pages even though the `Commerce Platform` collection owned it.
2. He updated the [annotations](https://docs-xcor.paloaltonetworks.com/investigate/alerts/monitors/monitor-annotations.md) on
   the team's monitors, including `checkout-error-budget-burn`, to point to
   `Commerce service health` with the service variable pre-populated in the URL.
3. He clicked **More actions**, selected
   [**Code config**](https://docs-xcor.paloaltonetworks.com/tooling/gitops.md#use-the-code-config-tool), and exported the
   `chronosphere_dashboard` resource to the team's
   [Terraform](https://docs-xcor.paloaltonetworks.com/tooling/infrastructure/terraform) repository so future edits went
   through code review alongside their monitors and collections.

Finally, he deleted `Checkout Triage - Copy`, `Payment Gateway Overview`, and
`Order Service HTTP` so no stale dashboard could compete as the source of truth.

## After

Two days later, a staging load test triggered a latency warning on `order-service`. The
engineer on call clicked the dashboard link in the monitor notification and landed on
`Commerce service health` already scoped to `order-service` and `staging`.

The **Overview** stat chart showed error rate holding at `0.12%` (`Healthy`), while the
P99 latency stat tile showed `1.8 s`. Scrolling to **Diagnostics**, the heat map showed
requests bifurcating into a fast `40 ms` band and a secondary `1.8 s` band starting at
11:05 AM, right on top of a staging deploy change event. The endpoint table immediately
next to the heat map ranked `/v1/orders/validate-inventory` at the top of the latency
and traffic columns. The responder pinned the 11:05 AM deploy marker, opened it in
Changes Explorer, and identified the inventory query change in under four minutes. The
investigation required no dashboard cloning and no manual PromQL rewrites, because the
entire team now worked from one shared view.


## Related topics

- [Case studies](https://docs-xcor.paloaltonetworks.com/overview/case-studies/case-studies.md)
- [Create dashboards for service health](https://docs-xcor.paloaltonetworks.com/overview/guides/create-dashboards.md)
- [Case study: responding to incidents](https://docs-xcor.paloaltonetworks.com/overview/case-studies/respond-to-incidents.md)
- [Send Google Cloud Personalized Service Health events to Cortex XCOR](https://docs-xcor.paloaltonetworks.com/ingest/third-party/google-cloud.md)
- [Cortex XCOR guides](https://docs-xcor.paloaltonetworks.com/overview/guides.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.