Prefer a guided lab? Take
Building Dynamic Dashboards to
practice panel types, queries, variables, and dashboard layout.
Four dashboards and no single answer
The action item came out of a Friday postmortem, though the problem was much older. Marcus Chen took the Monday morning rotation for the Commerce Platform team. Three weeks had passed since the 2:17 AM outage in the incident response case study, and a week had passed since the log spike in the budget protection case study. Before his shift started, he opened Dashboards and filtered the table by theCommerce Platform
collection to see what an on-call engineer would find.
Four dashboards displayed in the results:
Every dashboard had been built by an engineer trying to help their teammates, and
together they made triage slower. A responder investigating an issue that crossed from
checkout-service into payment-gateway had to open two dashboards with different
layouts, different time ranges, and different definitions of error rate. Worse, none of
the panels could be reused for order-service without copying the dashboard and
rewriting every query by hand.
Marcus wanted one standard dashboard that worked across all three services in the
collection. Before touching a query, he wrote down the six questions a responder needed
the dashboard to answer in the first minute:
- Is the service receiving traffic?
- Are requests succeeding?
- Is latency within an expected range?
- Which endpoint, instance, or other dimension needs attention?
- Did a deployment or other change coincide with the behavior?
- Where can the viewer continue investigating an issue?
Normalizing the telemetry before building
A reusable dashboard depends on consistent telemetry across the services it covers. Before creating the dashboard, Marcus opened Metrics Explorer and checked whethercheckout-service, payment-gateway, and order-service exposed the same HTTP
request metrics and labels.
All three services emitted http_requests_total and
http_request_duration_seconds_bucket, and all three carried environment, status,
and endpoint labels. However, an older gRPC adapter in order-service still emitted
its service identifier on a svc label instead of service for a subset of internal
routes. If Marcus built a dashboard variable on the service label alone, those routes
would never appear when a responder selected order-service.
Rather than waiting on a release to rename the label in service code, Marcus checked
the Attributes Catalog and verified that
a derived label
(also called a mapped label) already mapped svc
into service at query time. He also checked the
Metrics Catalog for the retry counter
created during the log budget cleanup, checkout_payment_retry_total. That review showed
which metrics belonged to every service and which belonged only to the checkout service.
With the label schema confirmed, Marcus created the dashboard:
- In the navigation menu, he selected Dashboards, clicked + Add, and selected New dashboard.
- In the Add dashboard dialog, he entered
Commerce service healthin the Dashboard name field. - He set Owner to the
Commerce Platformcollection so the team that maintained the three services owned the dashboard together, and clicked Save.
- Overview: Current request rate, error rate, and availability at a glance
- Trends: Request volume and error behavior over time
- Diagnostics: Latency distribution across buckets and a ranked table of endpoints
Turning one dashboard into three with variables
Hardcoded selectors were the reason the team had four dashboards instead of one. Marcus added dashboard variables before building any panels so every query would share the same filtering pattern from the start. He clicked Variables to open the Dashboard variables side panel, clicked Add variable, and configured two list variables:service: He set Type to List and Source to [Prometheus] Label values, selected theservicelabel, and added a series selector ofhttp_requests_totalso the dropdown listed only services that emitted HTTP request metrics in the tenant.environment: He set Type to List and Source to [Prometheus] Label values, selected theenvironmentlabel, and scoped it withhttp_requests_total{service=~"$service"}.
service and environment to match the underlying Prometheus
label names gave the team a second shortcut. When a responder pinned a tooltip on any
time series chart,
the series action menu included Filter service → checkout-service and
Filter environment → production. Clicking either action narrowed the entire
dashboard from a single spiking line without scrolling back to the header.
Marcus considered adding an interval variable
and decided against it. Cortex XCOR calculates query intervals from the time range and
panel resolution automatically. A manual interval variable lets a viewer pick a step
smaller than the 15-second scrape interval, which produces empty gaps in rate
calculations. Instead, he used $__rate_interval in every counter query so the
lookback window scaled with the dashboard time range and scrape interval.
After clicking Apply and saving the dashboard, the Saved View menu appeared in
the header. Marcus selected service = checkout-service and environment = production,
opened the Saved View menu, and clicked Create new view to save a view named
Checkout - Production. He repeated the step for Payment Gateway - Production and
Order Service - Production so on-call engineers could switch services in two clicks.
Matching each panel type to the shape of the data
Each question on Marcus’s list returned a different shape of data: a single current ratio, a set of continuous rates over time, a histogram bucket distribution, and a high-cardinality list of endpoint labels. Choosing the wrong visualization had made the team’s old dashboards hard to read. Marcus chose a distinct panel type for each data type. The following table summarizes how he matched each question and data type to a panel type:A stat chart for a single current health ratio
In the Overview group, a responder needed to know immediately whether the selected service was failing requests. That answer is a single current number, the percentage of HTTP5xx responses, rather than a multi-line graph that forces the viewer to read a
Y-axis under stress.
Marcus added a panel named Error rate (%) to the Overview group, selected
Stat Chart from the Type dropdown, selected the Metrics (Prometheus)
datasource, and entered the PromQL ratio query:
1 (orange) and 5 (red),
matching the team’s warning and critical alert conditions. To ensure color wasn’t the
only indication of health, he expanded each threshold row and entered Healthy in the
base Message field, Elevated (Warning >= 1%) on the 1 threshold, and
Breaching SLO (Critical >= 5%) on the 5 threshold. He added two companion stat
charts alongside it in the Overview group for current request rate and P99 latency.
A time series chart for continuous rates over time
Once a responder knew the error rate was elevated, the next question was temporal: when did the change start, how fast did it climb, and did successful traffic drop at the same time? Continuous rate series indexed by time belong on a time series chart, where the X-axis aligns the timing, duration, and direction of every status class across the dashboard’s time window. In the Trends group, Marcus created a panel namedRequest rate by status, kept
the default Time Series Chart type, and entered the following PromQL query:
by (status) returned a manageable number of series (200, 400, 404,
500, 503, 504) without flooding the chart. He set the series naming pattern to
{{status}} and set the Y-axis unit to requests per second. He also enabled the
Legend in Table mode with Last * and Max values visible so viewers
could compare the current rate against the peak in the window. In the Tooltip
section, he set Mode to Nearby and Sort to Descending so holding the
pointer over a spike listed the highest-volume status codes first.
A heat map for histogram bucket distributions
On the oldPayment Gateway Overview dashboard, a single average latency line had
masked partial outages whenever a small fraction of requests timed out at four seconds
while fast cache hits pulled the mean down. Even a single percentile line like P95 can
hide whether latency forms one stable band or splits into two distinct clusters.
Because http_request_duration_seconds_bucket is a cumulative Prometheus histogram
where each series carries an upper bound in the le label, its data represents density
across latency buckets over time. A heat map is
purpose-built for histogram bucket data: it plots time on the X-axis, bucket bounds on
the Y-axis, and request density as color intensity in each cell.
In the Diagnostics group, Marcus created a panel named
Request latency distribution, changed Type to Heat Map, and entered:
checkout-service, a dense band
sat between 50 ms and 200 ms, while a faint upper band near 4 s immediately
showed when downstream payment calls stalled.
A table for high-cardinality endpoint rankings
The oldOrder Service HTTP dashboard had failed for the opposite reason: it plotted
sum by (endpoint) on a time series chart. Because the three services exposed dozens
of REST and gRPC endpoints, the chart rendered a tangle of overlapping lines and a
scrolling legend that hid the exact endpoint names.
When a dimension has many distinct string values and a responder needs to rank them by
exact numbers, a table fits the data type far
better than a line chart. Each series becomes one sortable row, and full label strings
remain readable in their own column.
Marcus added a panel named Top endpoints by request rate in the Diagnostics
group, selected Table from the Type dropdown, and entered:
Requests per second. In the Settings tab, he kept Time series
format set to Default, set Sort by to Requests per second, and set Sort
order to Descending. On the Table columns tab, he set the Display Name
for endpoint to Endpoint and hid the timestamp column.
To give responders errors alongside traffic in the same view, he clicked Add query
in the table panel and added a second instant query for status=~"5.." aggregated
by (endpoint) with the alias Errors per second. Because both queries used the
Default table format and returned the same endpoint label set, Cortex XCOR merged
them into one row per endpoint with separate columns for Requests per second and
Errors per second.
Overlaying change events and dashboard annotations
A chart shows when a metric moved, but a responder still needs to know what changed at that timestamp and where to go next. First, Marcus edited each panel to add a clear Description stating what the query measured, what normal behavior looked like, and how missing data behaved. For example, the description onError rate (%) noted that the query returns no data when zero 5xx
series exist in a quiet development environment. On the Error rate (%) and
Top endpoints by request rate panels, he opened the Links tab, clicked
Add link, and added panel links
to the team’s checkout and payment runbooks and to Trace Explorer URLs that embedded
$service and $environment. For dashboard-to-dashboard links, he enabled
Include current time range and Include current variable values so clicking a
link preserved the active filter context for the responder.
Next, he added change events
so deployments would appear directly on the time series panels:
- On the dashboard, he clicked More actions and selected Events to open the Dashboard events side panel.
-
He clicked Add event, named the event layer
Production deploys, and selected Timeline events from the Datasource menu. -
In the filter box, he entered a change event query scoped to production deployments:
- He clicked Add, clicked Apply in the Dashboard events side panel, and clicked Save on the dashboard.
Request rate by status panel.
Clicking a marker pinned it and exposed Open in Changes Explorer, which opened the
event’s Summary tab in Changes Explorer.
Because change events are stored platform-wide, the same deploy markers appeared on
service pages and in Changes Explorer for every team in Cortex XCOR.
Mid-afternoon, while Marcus was watching checkout-service on the new dashboard, a
brief two-minute bump appeared in the 500 ms to 1 s buckets on the heat map and in
the 200 status series at 2:40 PM. No deploy marker accompanied it. He checked with
the engineer running a canary test and learned she had manually increased the payment
client connection pool size in us-east-1 to test connection reuse. That manual tuning
step didn’t emit a deployment event and mattered only to people reading this dashboard.
To record that context without polluting platform-wide change events, Marcus created a
dashboard annotation:
- On the
Request rate by statustime series chart, he clicked the data point at 2:40 PM to pin the tooltip. - In the pinned tooltip, he clicked Create annotation.
- In the Create annotation side panel, he entered
Increased payment client connection pool to 200 during canary testin the Title field and confirmed the Timestamp defaulted to 2:40 PM. - He clicked Add Label to attach
service = checkout-serviceandenvironment = production, and clicked Create.
Testing the workflow with a responder
Before retiring the old dashboards, Marcus asked Alex Reeves, who had worked the 2:17 AM checkout outage, to review theCommerce service health dashboard.
Alex opened the dashboard, switched the Saved View menu from
Checkout - Production to Payment Gateway - Production, and then selected All in
the service dropdown. She verified each item on their review checklist:
- The Overview stat charts answered whether the selected service was healthy in under five seconds, with text messages on every threshold state.
- Selecting multiple services or All in the
serviceandenvironmentvariables updated every panel cleanly, and pinning a series onRequest rate by statusoffered the one-click Filter action for matching variables. - The Trends time series chart, Diagnostics heat map, and endpoint Table moved from summary to timing to exact dimension values without visual clutter.
- Production deploy change events and the 2:40 PM dashboard annotation explained spikes in place without obscuring the lines.
- Clicking Inspect panel on each chart confirmed that query durations stayed fast at both the default 30-minute window and a 7-day comparison range.
- He clicked More actions, selected Connections, and
connected the dashboard to
checkout-service,payment-gateway, andorder-serviceso it appeared on all three service pages even though theCommerce Platformcollection owned it. - He updated the annotations on
the team’s monitors, including
checkout-error-budget-burn, to point toCommerce service healthwith the service variable pre-populated in the URL. - He clicked More actions, selected
Code config, and exported the
chronosphere_dashboardresource to the team’s Terraform repository so future edits went through code review alongside their monitors and collections.
Checkout Triage - Copy, Payment Gateway Overview, and
Order Service HTTP so no stale dashboard could compete as the source of truth.
After
Two days later, a staging load test triggered a latency warning onorder-service. The
engineer on call clicked the dashboard link in the monitor notification and landed on
Commerce service health already scoped to order-service and staging.
The Overview stat chart showed error rate holding at 0.12% (Healthy), while the
P99 latency stat tile showed 1.8 s. Scrolling to Diagnostics, the heat map showed
requests bifurcating into a fast 40 ms band and a secondary 1.8 s band starting at
11:05 AM, right on top of a staging deploy change event. The endpoint table immediately
next to the heat map ranked /v1/orders/validate-inventory at the top of the latency
and traffic columns. The responder pinned the 11:05 AM deploy marker, opened it in
Changes Explorer, and identified the inventory query change in under four minutes. The
investigation required no dashboard cloning and no manual PromQL rewrites, because the
entire team now worked from one shared view.