> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create dashboards for service health

> Build a reusable Cortex XCOR standard dashboard with variables, PromQL panels, annotations, and change events.

A service-health dashboard gives a team a shared view of system behavior before,
during, and after an incident. Use this guide to help design a reusable
[standard dashboard](https://docs-xcor.paloaltonetworks.com/observe/dashboards) in Palo Alto Networks Cortex XCOR, from
choosing the questions it must answer to adding variables, Prometheus Query Language
(PromQL) panels, and event context.

Before creating a new dashboard, search for an existing dashboard that covers your
use case. Reusing or extending an existing dashboard prevents several dashboards from
competing as the source of truth.

> [!NOTE]
>
> Prefer a guided lab? Take
> [Building Dynamic Dashboards](https://docs-xcor.paloaltonetworks.com/academy/courses/building-dynamic-dashboards.md) to
> practice panel types, queries, variables, and dashboard layout.

```mermaid theme={null}
graph LR
    accTitle: Dashboard creation workflow stages
    accDescr: A six-step flow from defining the dashboard purpose to creating the dashboard, adding variables and panels, adding event context, and reviewing the finished dashboard.
    A[Define the purpose] --> B[Create dashboard] --> C[Add variables]
    C --> D[Build panels] --> E[Add context] --> F[Review and share]
    click A href "#step-1-define-the-dashboards-purpose"
    click B href "#step-2-create-a-standard-dashboard"
    click C href "#step-3-add-variables"
    click D href "#step-4-build-panels-for-the-data"
    click E href "#step-5-add-context-and-investigation-paths"
    click F href "#step-6-review-and-share-the-dashboard"

classDef greenFill fill:#238A4D,stroke:#000,stroke-width:2px,color:#FFFFFF
class A,B,C,D,E,F greenFill;
```

## Step 1: Define the dashboard's purpose

Start with the decisions the dashboard must support. A service-health dashboard
should help a viewer answer these questions:

* Is the service receiving traffic?
* Are requests succeeding?
* Is latency within an expected range?
* Which endpoint, instance, or other dimension needs attention?
* Did a deployment or other change coincide with the behavior?
* Where can the viewer continue investigating an issue?

Use these questions to limit the dashboard's scope.

### Identify an owner and audience

Identify the team who owns the service and is responsible for maintaining it.
Ownership makes the dashboard discoverable from the corresponding resource and
identifies the team that should update queries when instrumentation changes.

Design the default view for a responder who knows the service but not the dashboard.
Put a quick health summary first, trends next, and diagnostic detail last. This
visual hierarchy lets a viewer assess current health before examining the cause.

## Step 2: Create a standard dashboard

Follow [Create a new dashboard](https://docs-xcor.paloaltonetworks.com/observe/dashboards#create-a-new-dashboard) to create
a standard dashboard and assign its owner. Use a name that identifies the service and
purpose, such as `Checkout service health`.

Assign the dashboard to the [service](https://docs-xcor.paloaltonetworks.com/observe/services) or
[collection](https://docs-xcor.paloaltonetworks.com/administer/collections) the dashboard relates to.

Set a default [time range](https://docs-xcor.paloaltonetworks.com/navigate/time-ranges.md) that matches how the team uses the
dashboard. A shorter range emphasizes active incidents, while a longer range reveals
recurring patterns. Dashboard viewers can change the range and use comparison mode to
overlay historical data.

### Plan the panel groups

Organize related panels into [panel groups](https://docs-xcor.paloaltonetworks.com/observe/dashboards/customization/layouts.md).
For a service-health dashboard, use a structure like:

* **Overview**: Current request rate, error rate, and latency
* **Trends**: Request and error behavior over time
* **Diagnostics**: Latency distribution and the endpoints generating the most traffic
  or errors

Keep each panel focused on one question. A chart with many unrelated queries takes
longer to interpret than several panels with specific names.

## Step 3: Add variables

[Dashboard variables](https://docs-xcor.paloaltonetworks.com/observe/dashboards/customization/dashboard-variables.md) turn one
dashboard into reusable views of several services and environments. Add the variables
before building panels so each query follows the same filtering pattern.

For this dashboard, create these list variables:

* `service`: Populate the list from values of the `service` label.
* `environment`: Populate the list from values of the `environment` label.

Use Prometheus label values as the source so each list reflects the data available in
the tenant. When a variable supports multiple values or an **All** selection, use a
regular expression matcher in PromQL:

```text theme={null}
{service=~"$service", environment=~"$environment"}
```

Matching variable names to label names also lets viewers filter the dashboard from a
[time series tooltip](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/time-series-chart.md#filter-a-dashboard-by-a-series-value).

Use an [interval variable](https://docs-xcor.paloaltonetworks.com/observe/dashboards/customization/intervals.md) only when
viewers need to control query or render intervals. Cortex XCOR calculates an interval
from the time range and panel resolution by default. Avoid interval options smaller
than the underlying data resolution.

## Step 4: Build panels for the data

Create each panel by following
[Create panels in a dashboard](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels#create-panels-in-a-dashboard).
Select the **Metrics (Prometheus)** datasource and add the example
[PromQL](https://docs-xcor.paloaltonetworks.com/investigate/querying/promql) expressions in this section. Replace metric
names and labels when the service uses a different instrumentation schema.

The examples use `$__rate_interval` for cumulative counters so the range scales with
the dashboard time range and accounts for the scrape interval.

### Show the current error rate with a stat chart

Use a [stat chart](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/stat-chart.md) for a value that a viewer
must assess at a glance. An error-rate percentage is a single current health
indicator, so a large value with a threshold and short sparkline communicates the
current state faster than a full chart in the overview group.

```text wrap theme={null}
100 *
sum(rate(http_requests_total{service=~"$service", environment=~"$environment", status=~"5.."}[$__rate_interval]))
/
sum(rate(http_requests_total{service=~"$service", environment=~"$environment"}[$__rate_interval]))
```

Set the unit to percent and choose an instant query, or reduce the selected range to
its last numeric value. Add thresholds that match the team's service expectations and
label the threshold states so color isn't the only indication of health.

### Show request trends with a time series chart

Use a [time series chart](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/time-series-chart.md) when the timing,
duration, and direction of a change matter. The following query shows request rate over
time and separates the results by status:

```text wrap theme={null}
sum by (status) (
  rate(http_requests_total{service=~"$service", environment=~"$environment"}[$__rate_interval])
)
```

Name each series from its `status` label, set the unit to requests per second, and
keep the legend visible. If the query returns too many series, aggregate statuses into
meaningful groups or split the question into another panel.

### Show latency distribution with a heat map

Use a [heat map](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/heat-map.md) for histogram data. A percentile
line summarizes latency, but a heat map shows whether requests form one stable
distribution or several clusters and whether the slow tail is growing.

```text wrap theme={null}
sum by (le) (
  rate(http_request_duration_seconds_bucket{
    service=~"$service",
    environment=~"$environment"
  }[$__rate_interval])
)
```

Set the Y-axis unit to seconds. Keep the legend visible so the bucket density remains
interpretable without relying on color intensity alone.

### Rank endpoints with a table

Use a [table](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels/table.md) when labels and exact values matter
more than shape over time. A sortable endpoint ranking helps a responder identify
where to investigate next.

```text wrap theme={null}
topk(
  10,
  sum by (endpoint) (
    rate(http_requests_total{
      service=~"$service",
      environment=~"$environment"
    }[$__rate_interval])
  )
)
```

Use an instant query so each endpoint has one current value. Display the `endpoint`
label as a column, name the value column `Requests per second`, and sort it in
descending order.

## Step 5: Add context and investigation paths

Queries show what happened. Add enough context for another responder to determine
whether the behavior matters and what to inspect next.

### Explain panels and thresholds

Give every panel a descriptive name, unit, and description. Use the description to
state what the query measures, what a normal result looks like, and any important
scope or missing-data behavior. Add thresholds when the team has an explicit
operational boundary. Don't invent a threshold only to add color to a panel.

Add [panel links](https://docs-xcor.paloaltonetworks.com/observe/dashboards/panels#add-links-to-a-panel) to related
dashboards, runbooks, or telemetry views. Include the current time range and variable
values when the destination supports them so the viewer keeps the investigation
context.

### Display change events

[Enable change events on the dashboard](https://docs-xcor.paloaltonetworks.com/observe/enable-events/dashboards.md#standard-dashboards)
to correlate telemetry with deployments, configuration updates, and other changes.
Filter the event layer to the dashboard's purpose. For example, the following filter
shows production deployments:

```text theme={null}
category = "deploys" AND labels.environment = "production"
```

Change events appear across Cortex XCOR and can open in
[Changes Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/filter-events.md). Use them for activity that
should remain available outside this dashboard.

### Record dashboard-specific context

Use a [dashboard annotation](https://docs-xcor.paloaltonetworks.com/observe/dashboards/annotations.md) to mark a point that
matters only to this dashboard, such as the start of an incident or a manual
configuration change. Annotations remain attached to the dashboard and can include
labels and comments.

Choose a change event instead when the occurrence should appear on other dashboards,
service pages, or in Changes Explorer.

## Step 6: Review and share the dashboard

Review the completed dashboard with someone who didn't build it. Confirm the
dashboard meets these criteria:

* The title, owner, and first panel group make the purpose clear.
* The default time range fits the expected workflow.
* The overview group answers whether the service is healthy.
* Each variable updates every panel where it applies, including the **All** and
  multiple-value selections.
* Panel names, descriptions, legends, units, and threshold labels explain the data.
* The layout moves from summary to trends to diagnostic detail.
* Change events and annotations add context without obscuring the telemetry.
* Panel links preserve enough context to continue an investigation.
* Queries return within an acceptable time at the default and longest expected time
  ranges.

Save the dashboard, then connect it to other services that need it by following
[Connect a dashboard to a service](https://docs-xcor.paloaltonetworks.com/observe/dashboards#connect-a-dashboard-to-a-service).
Add the dashboard to relevant monitor
[annotations](https://docs-xcor.paloaltonetworks.com/investigate/alerts/monitors/monitor-annotations.md) so responders can open
it from an alert.

## Maintain the dashboard

Maintain the dashboard as a living service resource:

* Remove panels that no longer support a decision or investigation.
* Update descriptions and thresholds when service expectations change.
* Review slow or high-volume queries and adjust their aggregation or
  [resolution](https://docs-xcor.paloaltonetworks.com/observe/dashboards/customization/intervals.md).
* Consolidate dashboards that answer the same questions.
* Export the dashboard with
  [Code config](https://docs-xcor.paloaltonetworks.com/tooling/gitops.md#use-the-code-config-tool) when the team manages
  observability resources through code review.


## Related topics

- [Cortex XCOR guides](https://docs-xcor.paloaltonetworks.com/overview/guides.md)
- [Send Google Cloud Personalized Service Health events to Cortex XCOR](https://docs-xcor.paloaltonetworks.com/ingest/third-party/google-cloud.md)
- [Get started with Cortex XCOR](https://docs-xcor.paloaltonetworks.com/overview/get-started.md)
- [Cortex XCOR-managed dashboards](https://docs-xcor.paloaltonetworks.com/observe/dashboards/managed-dashboards.md)
- [Observe your telemetry data](https://docs-xcor.paloaltonetworks.com/observe.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.