Create dashboards for service health

A service-health dashboard gives a team a shared view of system behavior before, during, and after an incident. Use this guide to help design a reusable standard dashboard in Palo Alto Networks Cortex XCOR, from choosing the questions it must answer to adding variables, Prometheus Query Language (PromQL) panels, and event context. Before creating a new dashboard, search for an existing dashboard that covers your use case. Reusing or extending an existing dashboard prevents several dashboards from competing as the source of truth.

Prefer a guided lab? Take Building Dynamic Dashboards to practice panel types, queries, variables, and dashboard layout.

Step 1: Define the dashboard’s purpose

Start with the decisions the dashboard must support. A service-health dashboard should help a viewer answer these questions:

Is the service receiving traffic?
Are requests succeeding?
Is latency within an expected range?
Which endpoint, instance, or other dimension needs attention?
Did a deployment or other change coincide with the behavior?
Where can the viewer continue investigating an issue?

Use these questions to limit the dashboard’s scope.

Identify an owner and audience

Identify the team who owns the service and is responsible for maintaining it. Ownership makes the dashboard discoverable from the corresponding resource and identifies the team that should update queries when instrumentation changes. Design the default view for a responder who knows the service but not the dashboard. Put a quick health summary first, trends next, and diagnostic detail last. This visual hierarchy lets a viewer assess current health before examining the cause.

Step 2: Create a standard dashboard

Follow Create a new dashboard to create a standard dashboard and assign its owner. Use a name that identifies the service and purpose, such as Checkout service health. Assign the dashboard to the service or collection the dashboard relates to. Set a default time range that matches how the team uses the dashboard. A shorter range emphasizes active incidents, while a longer range reveals recurring patterns. Dashboard viewers can change the range and use comparison mode to overlay historical data.

Plan the panel groups

Organize related panels into panel groups. For a service-health dashboard, use a structure like:

Overview: Current request rate, error rate, and latency
Trends: Request and error behavior over time
Diagnostics: Latency distribution and the endpoints generating the most traffic or errors

Keep each panel focused on one question. A chart with many unrelated queries takes longer to interpret than several panels with specific names.

Step 3: Add variables

Dashboard variables turn one dashboard into reusable views of several services and environments. Add the variables before building panels so each query follows the same filtering pattern. For this dashboard, create these list variables:

service: Populate the list from values of the service label.
environment: Populate the list from values of the environment label.

Use Prometheus label values as the source so each list reflects the data available in the tenant. When a variable supports multiple values or an All selection, use a regular expression matcher in PromQL:

{service=~"$service", environment=~"$environment"}

Matching variable names to label names also lets viewers filter the dashboard from a time series tooltip. Use an interval variable only when viewers need to control query or render intervals. Cortex XCOR calculates an interval from the time range and panel resolution by default. Avoid interval options smaller than the underlying data resolution.

Step 4: Build panels for the data

Create each panel by following Create panels in a dashboard. Select the Metrics (Prometheus) datasource and add the example PromQL expressions in this section. Replace metric names and labels when the service uses a different instrumentation schema. The examples use $__rate_interval for cumulative counters so the range scales with the dashboard time range and accounts for the scrape interval.

Show the current error rate with a stat chart

Use a stat chart for a value that a viewer must assess at a glance. An error-rate percentage is a single current health indicator, so a large value with a threshold and short sparkline communicates the current state faster than a full chart in the overview group.

100 *
sum(rate(http_requests_total{service=~"$service", environment=~"$environment", status=~"5.."}[$__rate_interval]))
/
sum(rate(http_requests_total{service=~"$service", environment=~"$environment"}[$__rate_interval]))

Set the unit to percent and choose an instant query, or reduce the selected range to its last numeric value. Add thresholds that match the team’s service expectations and label the threshold states so color isn’t the only indication of health.

Show request trends with a time series chart

Use a time series chart when the timing, duration, and direction of a change matter. The following query shows request rate over time and separates the results by status:

sum by (status) (
  rate(http_requests_total{service=~"$service", environment=~"$environment"}[$__rate_interval])
)

Name each series from its status label, set the unit to requests per second, and keep the legend visible. If the query returns too many series, aggregate statuses into meaningful groups or split the question into another panel.

Show latency distribution with a heat map

Use a heat map for histogram data. A percentile line summarizes latency, but a heat map shows whether requests form one stable distribution or several clusters and whether the slow tail is growing.

sum by (le) (
  rate(http_request_duration_seconds_bucket{
    service=~"$service",
    environment=~"$environment"
  }[$__rate_interval])
)

Set the Y-axis unit to seconds. Keep the legend visible so the bucket density remains interpretable without relying on color intensity alone.

Rank endpoints with a table

Use a table when labels and exact values matter more than shape over time. A sortable endpoint ranking helps a responder identify where to investigate next.

topk(
  10,
  sum by (endpoint) (
    rate(http_requests_total{
      service=~"$service",
      environment=~"$environment"
    }[$__rate_interval])
  )
)

Use an instant query so each endpoint has one current value. Display the endpoint label as a column, name the value column Requests per second, and sort it in descending order.

Step 5: Add context and investigation paths

Queries show what happened. Add enough context for another responder to determine whether the behavior matters and what to inspect next.

Explain panels and thresholds

Give every panel a descriptive name, unit, and description. Use the description to state what the query measures, what a normal result looks like, and any important scope or missing-data behavior. Add thresholds when the team has an explicit operational boundary. Don’t invent a threshold only to add color to a panel. Add panel links to related dashboards, runbooks, or telemetry views. Include the current time range and variable values when the destination supports them so the viewer keeps the investigation context.

Display change events

Enable change events on the dashboard to correlate telemetry with deployments, configuration updates, and other changes. Filter the event layer to the dashboard’s purpose. For example, the following filter shows production deployments:

category = "deploys" AND labels.environment = "production"

Change events appear across Cortex XCOR and can open in Changes Explorer. Use them for activity that should remain available outside this dashboard.

Record dashboard-specific context

Use a dashboard annotation to mark a point that matters only to this dashboard, such as the start of an incident or a manual configuration change. Annotations remain attached to the dashboard and can include labels and comments. Choose a change event instead when the occurrence should appear on other dashboards, service pages, or in Changes Explorer. Review the completed dashboard with someone who didn’t build it. Confirm the dashboard meets these criteria:

The title, owner, and first panel group make the purpose clear.
The default time range fits the expected workflow.
The overview group answers whether the service is healthy.
Each variable updates every panel where it applies, including the All and multiple-value selections.
Panel names, descriptions, legends, units, and threshold labels explain the data.
The layout moves from summary to trends to diagnostic detail.
Change events and annotations add context without obscuring the telemetry.
Panel links preserve enough context to continue an investigation.
Queries return within an acceptable time at the default and longest expected time ranges.

Save the dashboard, then connect it to other services that need it by following Connect a dashboard to a service. Add the dashboard to relevant monitor annotations so responders can open it from an alert.

Maintain the dashboard

Maintain the dashboard as a living service resource:

Remove panels that no longer support a decision or investigation.
Update descriptions and thresholds when service expectations change.
Review slow or high-volume queries and adjust their aggregation or resolution.
Consolidate dashboards that answer the same questions.
Export the dashboard with Code config when the team manages observability resources through code review.

Cortex XCOR

Palo Alto Networks Telemetry Pipeline

More information

Step 1: Define the dashboard’s purpose

Identify an owner and audience

Step 2: Create a standard dashboard

Plan the panel groups

Step 3: Add variables

Step 4: Build panels for the data

Show the current error rate with a stat chart

Show request trends with a time series chart

Show latency distribution with a heat map

Rank endpoints with a table

Step 5: Add context and investigation paths

Explain panels and thresholds

Display change events

Record dashboard-specific context

Maintain the dashboard

​Step 1: Define the dashboard’s purpose

​Identify an owner and audience

​Step 2: Create a standard dashboard

​Plan the panel groups

​Step 3: Add variables

​Step 4: Build panels for the data

​Show the current error rate with a stat chart

​Show request trends with a time series chart

​Show latency distribution with a heat map

​Rank endpoints with a table

​Step 5: Add context and investigation paths

​Explain panels and thresholds

​Display change events

​Record dashboard-specific context

​Step 6: Review and share the dashboard

​Maintain the dashboard

Step 1: Define the dashboard’s purpose

Identify an owner and audience

Step 2: Create a standard dashboard

Plan the panel groups

Step 3: Add variables

Step 4: Build panels for the data

Show the current error rate with a stat chart

Show request trends with a time series chart

Show latency distribution with a heat map

Rank endpoints with a table

Step 5: Add context and investigation paths

Explain panels and thresholds

Display change events

Record dashboard-specific context

Step 6: Review and share the dashboard

Maintain the dashboard