Skip to main content
A dashboard built for one incident often fails the next one. This case study follows an engineer on the Commerce Platform team as he replaces four competing service dashboards with one reusable standard dashboard in Palo Alto Networks Cortex XCOR. The walkthrough shows how dashboard variables, panel types chosen for each data shape, Prometheus Query Language (PromQL) queries, change events, and dashboard annotations fit together to give a team one shared view of service health. For a step-by-step reference version of this case study, see the Create dashboards for service health guide.
Prefer a guided lab? Take Building Dynamic Dashboards to practice panel types, queries, variables, and dashboard layout.

Four dashboards and no single answer

The action item came out of a Friday postmortem, though the problem was much older. Marcus Chen took the Monday morning rotation for the Commerce Platform team. Three weeks had passed since the 2:17 AM outage in the incident response case study, and a week had passed since the log spike in the budget protection case study. Before his shift started, he opened Dashboards and filtered the table by the Commerce Platform collection to see what an on-call engineer would find. Four dashboards displayed in the results: Every dashboard had been built by an engineer trying to help their teammates, and together they made triage slower. A responder investigating an issue that crossed from checkout-service into payment-gateway had to open two dashboards with different layouts, different time ranges, and different definitions of error rate. Worse, none of the panels could be reused for order-service without copying the dashboard and rewriting every query by hand. Marcus wanted one standard dashboard that worked across all three services in the collection. Before touching a query, he wrote down the six questions a responder needed the dashboard to answer in the first minute:
  • Is the service receiving traffic?
  • Are requests succeeding?
  • Is latency within an expected range?
  • Which endpoint, instance, or other dimension needs attention?
  • Did a deployment or other change coincide with the behavior?
  • Where can the viewer continue investigating an issue?

Normalizing the telemetry before building

A reusable dashboard depends on consistent telemetry across the services it covers. Before creating the dashboard, Marcus opened Metrics Explorer and checked whether checkout-service, payment-gateway, and order-service exposed the same HTTP request metrics and labels. All three services emitted http_requests_total and http_request_duration_seconds_bucket, and all three carried environment, status, and endpoint labels. However, an older gRPC adapter in order-service still emitted its service identifier on a svc label instead of service for a subset of internal routes. If Marcus built a dashboard variable on the service label alone, those routes would never appear when a responder selected order-service. Rather than waiting on a release to rename the label in service code, Marcus checked the Attributes Catalog and verified that a derived label (also called a mapped label) already mapped svc into service at query time. He also checked the Metrics Catalog for the retry counter created during the log budget cleanup, checkout_payment_retry_total. That review showed which metrics belonged to every service and which belonged only to the checkout service. With the label schema confirmed, Marcus created the dashboard:
  1. In the navigation menu, he selected Dashboards, clicked + Add, and selected New dashboard.
  2. In the Add dashboard dialog, he entered Commerce service health in the Dashboard name field.
  3. He set Owner to the Commerce Platform collection so the team that maintained the three services owned the dashboard together, and clicked Save.
He set the default time range to Last 30 minutes. Thirty minutes was short enough to make an active incident stand out and wide enough to show the baseline before a deploy. Next, he planned the visual hierarchy with three panel groups so a responder who knew the services, but hadn’t memorized the dashboard, could read from top to bottom:
  • Overview: Current request rate, error rate, and availability at a glance
  • Trends: Request volume and error behavior over time
  • Diagnostics: Latency distribution across buckets and a ranked table of endpoints
He clicked Edit on the dashboard, clicked Panel Group to create each group, and kept all three groups set to Open by default.

Turning one dashboard into three with variables

Hardcoded selectors were the reason the team had four dashboards instead of one. Marcus added dashboard variables before building any panels so every query would share the same filtering pattern from the start. He clicked Variables to open the Dashboard variables side panel, clicked Add variable, and configured two list variables:
  • service: He set Type to List and Source to [Prometheus] Label values, selected the service label, and added a series selector of http_requests_total so the dropdown listed only services that emitted HTTP request metrics in the tenant.
  • environment: He set Type to List and Source to [Prometheus] Label values, selected the environment label, and scoped it with http_requests_total{service=~"$service"}.
On both variables, he enabled Select multiple values and Include “All” option, and set Refresh to On time range change so the dropdown options reflected the active series in the selected window. Because a viewer could select more than one service or choose All, every panel query would use regular expression matchers:
Naming the variables service and environment to match the underlying Prometheus label names gave the team a second shortcut. When a responder pinned a tooltip on any time series chart, the series action menu included Filter service → checkout-service and Filter environment → production. Clicking either action narrowed the entire dashboard from a single spiking line without scrolling back to the header. Marcus considered adding an interval variable and decided against it. Cortex XCOR calculates query intervals from the time range and panel resolution automatically. A manual interval variable lets a viewer pick a step smaller than the 15-second scrape interval, which produces empty gaps in rate calculations. Instead, he used $__rate_interval in every counter query so the lookback window scaled with the dashboard time range and scrape interval. After clicking Apply and saving the dashboard, the Saved View menu appeared in the header. Marcus selected service = checkout-service and environment = production, opened the Saved View menu, and clicked Create new view to save a view named Checkout - Production. He repeated the step for Payment Gateway - Production and Order Service - Production so on-call engineers could switch services in two clicks.

Matching each panel type to the shape of the data

Each question on Marcus’s list returned a different shape of data: a single current ratio, a set of continuous rates over time, a histogram bucket distribution, and a high-cardinality list of endpoint labels. Choosing the wrong visualization had made the team’s old dashboards hard to read. Marcus chose a distinct panel type for each data type. The following table summarizes how he matched each question and data type to a panel type:

A stat chart for a single current health ratio

In the Overview group, a responder needed to know immediately whether the selected service was failing requests. That answer is a single current number, the percentage of HTTP 5xx responses, rather than a multi-line graph that forces the viewer to read a Y-axis under stress. Marcus added a panel named Error rate (%) to the Overview group, selected Stat Chart from the Type dropdown, selected the Metrics (Prometheus) datasource, and entered the PromQL ratio query:
In the query Options, he set Query type to Instant so the stat tile evaluated the latest rate window. In the Settings tab, he enabled Sparkline to show the recent direction of the metric beneath the number. He then set the unit in Unit and formatting to Percent (0-100) with two decimal places. In the Thresholds section, he added thresholds at 1 (orange) and 5 (red), matching the team’s warning and critical alert conditions. To ensure color wasn’t the only indication of health, he expanded each threshold row and entered Healthy in the base Message field, Elevated (Warning >= 1%) on the 1 threshold, and Breaching SLO (Critical >= 5%) on the 5 threshold. He added two companion stat charts alongside it in the Overview group for current request rate and P99 latency.

A time series chart for continuous rates over time

Once a responder knew the error rate was elevated, the next question was temporal: when did the change start, how fast did it climb, and did successful traffic drop at the same time? Continuous rate series indexed by time belong on a time series chart, where the X-axis aligns the timing, duration, and direction of every status class across the dashboard’s time window. In the Trends group, Marcus created a panel named Request rate by status, kept the default Time Series Chart type, and entered the following PromQL query:
Grouping by (status) returned a manageable number of series (200, 400, 404, 500, 503, 504) without flooding the chart. He set the series naming pattern to {{status}} and set the Y-axis unit to requests per second. He also enabled the Legend in Table mode with Last * and Max values visible so viewers could compare the current rate against the peak in the window. In the Tooltip section, he set Mode to Nearby and Sort to Descending so holding the pointer over a spike listed the highest-volume status codes first.

A heat map for histogram bucket distributions

On the old Payment Gateway Overview dashboard, a single average latency line had masked partial outages whenever a small fraction of requests timed out at four seconds while fast cache hits pulled the mean down. Even a single percentile line like P95 can hide whether latency forms one stable band or splits into two distinct clusters. Because http_request_duration_seconds_bucket is a cumulative Prometheus histogram where each series carries an upper bound in the le label, its data represents density across latency buckets over time. A heat map is purpose-built for histogram bucket data: it plots time on the X-axis, bucket bounds on the Y-axis, and request density as color intensity in each cell. In the Diagnostics group, Marcus created a panel named Request latency distribution, changed Type to Heat Map, and entered:
In the Settings tab under Y axis, he set Unit to Time and selected Seconds (s). He kept the Legend enabled on the right side of the panel so the relationship between color intensity and request count remained explicit without relying on color perception alone. On the preview for checkout-service, a dense band sat between 50 ms and 200 ms, while a faint upper band near 4 s immediately showed when downstream payment calls stalled.

A table for high-cardinality endpoint rankings

The old Order Service HTTP dashboard had failed for the opposite reason: it plotted sum by (endpoint) on a time series chart. Because the three services exposed dozens of REST and gRPC endpoints, the chart rendered a tangle of overlapping lines and a scrolling legend that hid the exact endpoint names. When a dimension has many distinct string values and a responder needs to rank them by exact numbers, a table fits the data type far better than a line chart. Each series becomes one sortable row, and full label strings remain readable in their own column. Marcus added a panel named Top endpoints by request rate in the Diagnostics group, selected Table from the Type dropdown, and entered:
In the query Options, he set Query type to Instant so each endpoint returned one current value rather than a time series of points, and set the query Alias to Requests per second. In the Settings tab, he kept Time series format set to Default, set Sort by to Requests per second, and set Sort order to Descending. On the Table columns tab, he set the Display Name for endpoint to Endpoint and hid the timestamp column. To give responders errors alongside traffic in the same view, he clicked Add query in the table panel and added a second instant query for status=~"5.." aggregated by (endpoint) with the alias Errors per second. Because both queries used the Default table format and returned the same endpoint label set, Cortex XCOR merged them into one row per endpoint with separate columns for Requests per second and Errors per second.

Overlaying change events and dashboard annotations

A chart shows when a metric moved, but a responder still needs to know what changed at that timestamp and where to go next. First, Marcus edited each panel to add a clear Description stating what the query measured, what normal behavior looked like, and how missing data behaved. For example, the description on Error rate (%) noted that the query returns no data when zero 5xx series exist in a quiet development environment. On the Error rate (%) and Top endpoints by request rate panels, he opened the Links tab, clicked Add link, and added panel links to the team’s checkout and payment runbooks and to Trace Explorer URLs that embedded $service and $environment. For dashboard-to-dashboard links, he enabled Include current time range and Include current variable values so clicking a link preserved the active filter context for the responder. Next, he added change events so deployments would appear directly on the time series panels:
  1. On the dashboard, he clicked More actions and selected Events to open the Dashboard events side panel.
  2. He clicked Add event, named the event layer Production deploys, and selected Timeline events from the Datasource menu.
  3. In the filter box, he entered a change event query scoped to production deployments:
  4. He clicked Add, clicked Apply in the Dashboard events side panel, and clicked Save on the dashboard.
Vertical deploy markers immediately appeared across the Request rate by status panel. Clicking a marker pinned it and exposed Open in Changes Explorer, which opened the event’s Summary tab in Changes Explorer. Because change events are stored platform-wide, the same deploy markers appeared on service pages and in Changes Explorer for every team in Cortex XCOR. Mid-afternoon, while Marcus was watching checkout-service on the new dashboard, a brief two-minute bump appeared in the 500 ms to 1 s buckets on the heat map and in the 200 status series at 2:40 PM. No deploy marker accompanied it. He checked with the engineer running a canary test and learned she had manually increased the payment client connection pool size in us-east-1 to test connection reuse. That manual tuning step didn’t emit a deployment event and mattered only to people reading this dashboard. To record that context without polluting platform-wide change events, Marcus created a dashboard annotation:
  1. On the Request rate by status time series chart, he clicked the data point at 2:40 PM to pin the tooltip.
  2. In the pinned tooltip, he clicked Create annotation.
  3. In the Create annotation side panel, he entered Increased payment client connection pool to 200 during canary test in the Title field and confirmed the Timestamp defaulted to 2:40 PM.
  4. He clicked Add Label to attach service = checkout-service and environment = production, and clicked Create.
The annotation marker appeared on the dashboard’s time series panels through the built-in Annotations event layer. Anyone on the Commerce Platform team who looked at the 2:40 PM bump would see the explanation immediately in the Annotation side panel, while Changes Explorer stayed focused on system-wide events.

Testing the workflow with a responder

Before retiring the old dashboards, Marcus asked Alex Reeves, who had worked the 2:17 AM checkout outage, to review the Commerce service health dashboard. Alex opened the dashboard, switched the Saved View menu from Checkout - Production to Payment Gateway - Production, and then selected All in the service dropdown. She verified each item on their review checklist:
  • The Overview stat charts answered whether the selected service was healthy in under five seconds, with text messages on every threshold state.
  • Selecting multiple services or All in the service and environment variables updated every panel cleanly, and pinning a series on Request rate by status offered the one-click Filter action for matching variables.
  • The Trends time series chart, Diagnostics heat map, and endpoint Table moved from summary to timing to exact dimension values without visual clutter.
  • Production deploy change events and the 2:40 PM dashboard annotation explained spikes in place without obscuring the lines.
  • Clicking Inspect panel on each chart confirmed that query durations stayed fast at both the default 30-minute window and a 7-day comparison range.
With the review complete, Marcus connected the dashboard to the rest of the team’s observability workflow:
  1. He clicked More actions, selected Connections, and connected the dashboard to checkout-service, payment-gateway, and order-service so it appeared on all three service pages even though the Commerce Platform collection owned it.
  2. He updated the annotations on the team’s monitors, including checkout-error-budget-burn, to point to Commerce service health with the service variable pre-populated in the URL.
  3. He clicked More actions, selected Code config, and exported the chronosphere_dashboard resource to the team’s Terraform repository so future edits went through code review alongside their monitors and collections.
Finally, he deleted Checkout Triage - Copy, Payment Gateway Overview, and Order Service HTTP so no stale dashboard could compete as the source of truth.

After

Two days later, a staging load test triggered a latency warning on order-service. The engineer on call clicked the dashboard link in the monitor notification and landed on Commerce service health already scoped to order-service and staging. The Overview stat chart showed error rate holding at 0.12% (Healthy), while the P99 latency stat tile showed 1.8 s. Scrolling to Diagnostics, the heat map showed requests bifurcating into a fast 40 ms band and a secondary 1.8 s band starting at 11:05 AM, right on top of a staging deploy change event. The endpoint table immediately next to the heat map ranked /v1/orders/validate-inventory at the top of the latency and traffic columns. The responder pinned the 11:05 AM deploy marker, opened it in Changes Explorer, and identified the inventory query change in under four minutes. The investigation required no dashboard cloning and no manual PromQL rewrites, because the entire team now worked from one shared view.