> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Analyze live traffic metrics

> Use Live Telemetry Analyzer to view incoming metrics, troubleshoot ingest spikes, and identify metrics to reduce.

export const MicroscopeIcon = props => <svg viewBox="0 0 24 24" {...props} className="inline-block h-5 w-5" aria-label="Microscope" style={{
  fill: "#28a561"
}}>
    <path d="M9.46,6.28L11.05,9C8.47,9.26 6.5,11.41 6.5,14A5,5 0 0,0 11.5,19C13.55,19 15.31,17.77 16.08,16H13.5V14H21.5V16H19.25C18.84,17.57 17.97,18.96 16.79,20H19.5V22H3.5V20H6.21C4.55,18.53 3.5,16.39 3.5,14C3.5,10.37 5.96,7.2 9.46,6.28M12.74,2.07L13.5,3.37L14.36,2.87L17.86,8.93L14.39,10.93L10.89,4.87L11.76,4.37L11,3.07L12.74,2.07Z" />
  </svg>;

The Live Telemetry Analyzer provides a real-time view of incoming metrics grouped by
label, and their relative frequency. This view helps you understand how often your
applications emit metrics, troubleshoot spikes in ingest rates, and ensure that the
[Chronosphere Collector](/ingest/metrics-traces/collector) is aware of particular
metrics. Use the metrics telemetry analyzer as a first step in identifying
opportunities to reduce the overall volume of metrics.

<Note>
  Palo Alto Networks Cortex XCOR adaptively adjusts the metrics sampling rate
  based on the current workload. This behavior means that not all metrics are
  immediately visible in the Live Telemetry Analyzer.
</Note>

## Capture and analyze live profiling data

To use the metrics telemetry analyzer to capture live profiling data:

1. In the navigation menu, click **<Icon icon="shield-user" /> Go to Admin** and
   then select
   **<MicroscopeIcon /> Analyzers <span aria-label="and then">></span> Live Telemetry**.

2. Click either the **Metrics** or **Traces** tab, depending on the data you want to
   profile.

3. Click **Capture live data** to begin gathering statistics for data that
   Cortex XCOR accepted for matching. Click **Capture live data** again
   to pause the capture. Live Telemetry Analyzer automatically pauses the capture
   after three minutes.

   The **Labels** table contains these columns:

   * **Label Keys:** Label keys in the captured metrics.
   * **Unique Values:** Number of unique values for the respective label key.
   * **Appears In:** The percentage of metrics you're viewing that have the matching
     label key.

   The **Label values** table contains columns for the selected grouping labels and
   these columns:

   * **Avg. DPPS:** Average data points per second (DPPS), calculated over the
     previous 15 seconds.
   * **Current DPPS:** Current data points per second.

   You can make changes to the [groupings and filters](#group-and-filter-metrics)
   while profiling.

4. To modify the displayed data, select an option from the **Data phase** menu to
   show data in [different stages](#ingestion-stages-and-phases) along the pathway
   from ingestion to persistence.

   For example, select **Rejected by drop rule** to view all data that Cortex XCOR
   dropped because of a configured drop rule.

5. To display data for a specific pool only, select a pool from the **Pool** menu to
   choose a specific [metric pool](/control/shaping/shape-metrics/quotas/quotas-ui).

   Use the **Priority** menu to narrow the filter to a specific pool priority.

To filter on specific labels, start typing a label name in the **Add label filter**
field, choose from an autocomplete list of labels, and then add a value to filter
for a specific label.

Live Telemetry Analyzer displays the first 500 label keys and label values and
supports up to ten grouping labels.

### Analyze metrics

When analyzing traffic, use the following methods to help narrow your analysis and
find the information you need:

* Review the metric names that generate the most data points per second
  (**Avg. DPPS** or **Current DPPS**). If those metrics are unfamiliar to you or are
  expensive, these might be candidates to
  [roll up](/control/shaping/shape-metrics/rules/rollup)
  or [drop](/control/shaping/shape-metrics/rules/drop-rules).
* Ensure your drop and rollup rules are working as expected by reviewing your
  rolled up metrics, or ensuring that a dropped metric no longer displays.
* [Group metrics](#group-and-filter-metrics) by job to identify the specific scrape
  jobs generating the most metrics. Filter for each job, and analyze the job's
  individual metrics to find opportunities for reduction. Metrics from the same job
  are often used together, letting you investigate metrics from a single job more
  quickly.
* Review metrics isolated to single environments. For example, metrics available only
  in development or production environment metrics. These are likely to have different
  metric workload shapes from each other.

### Ingestion stages and phases

Cortex XCOR profiles metrics in the following stages, which display in
Live Telemetry Analyzer, in the **Data phase** menu:

**Ingestion:** Metrics sent directly from the Chronosphere Collector. Ingestion
includes these phases:

* **Received:** Not selectable.
* **Rejected by drop rule:** Metrics dropped due to
  [drop rules](/control/shaping/shape-metrics/rules/drop-rules). This option is
  relevant only for the Ingestion phase.
* **Rejected by ingest limit:** Metrics that dropped due to exceeding the ingestion
  rate limit.
* **Accepted for matching:** Metrics that aren't dropped prior to ingestion.

**Aggregation:** When available, this stage contains metrics matching existing
aggregation rules. Select
**Matched for aggregation** to focus on metrics that match an
[aggregation rule](/control/shaping/shape-metrics/rules/rollup) before persisting data. To view
samples of that traffic, see
[Inspect matched data for aggregation](#inspect-matched-data-for-aggregation).

**Persistence:** Metrics sent to the database. This stage includes aggregated
metrics and the following phases:

* **Rejected by persist limit:** Metrics not sent to permanent storage due
  to persistence limits.
* **Accepted for storage:** Metrics sent to storage.
* **Stored:** Not selectable.

### Special request metadata

The Live Telemetry Analyzer generates rows for the following special non-label
request metadata. This special non-label request metadata is available in the Live
Telemetry Analyzer and for matching in rollup rules, but isn't stored.

The following label keys display for all incoming metrics:

* `__metric_type__` displays on the incoming metric's
  [Cortex XCOR metric type](/control/shaping/shape-metrics/types#cortex-xcor-types).
  Valid values are `cumulative_counter`, `cumulative_exponential_histogram`,
  `delta_counter`, `delta_exponential_histogram`, `gauge`, or `measurement`. This
  is the recommended method for determining an incoming metric's type.
* `__metric_source__` displays on the incoming metric's [source format](/control/shaping/shape-metrics/types#supported-formats).
  Valid values are `carbon`, `chrono_azure`, `chrono_cloudflare`, `chrono_gcp`,
  `chrono_integration`, `cloudwatch_metric_stream`, `dogstatsd`, `open_metrics`,
  `open_telemetry`, `prometheus`, `signalfx`, `statsd`, `tagged_statsd`,
  or `wavefront`.

When ingesting data with Prometheus, the following label keys display:

* `__m3_prom_type__` displays the incoming metric's
  [Prometheus metric type](/control/shaping/shape-metrics/types#prometheus).
  Valid values are `counter`, `gauge`, `histogram`, `gauge_histogram`, `summary`,
  `info`, `state_set`, or `quantile`.

When ingesting data with OpenTelemetry, the following label keys display:

* `__otel_type__` displays the incoming metric's
  [OpenTelemetry metric type](/control/shaping/shape-metrics/types#opentelemetry).
  Valid values are `sum`, `monotonic_sum`, `gauge`, `histogram`, `exp_histogram`, or
  `summary`.
* `__otel_temporality__` displays the incoming metric's
  [OpenTelemetry temporality](/control/shaping/shape-metrics/types#opentelemetry). Valid values are
  `delta` or `cumulative`.
* DEPRECATED: `__m3_type__` displays on the incoming metric's legacy M3 type, if any.
  Valid values are `counter`, `gauge`, or `timer`.

## Inspect matched data for aggregation

In Live Telemetry Analyzer, the aggregation stage provides deeper visibility into how
data is matched to aggregation rules for improved debugging and validation.

Live Telemetry Analyzer adds dimensions such as `__rollup_name__` and
`__rollup_rule_slug__` in this phase so you can relate samples back to the series
being matched that go into rules. This behavior is similar
to how you use `__rollup_rule_slug__` under **Accepted for storage** when
[identifying rules that generate metrics](#identify-rules-that-generate-metrics).

The **Label values** tab summarizes how often each label value appears. When
**Matched for aggregation** is selected in the **Data phase** dropdown, the
**Samples** tab streams individual data points. Each row shows metric names, label
sets, values, timestamps, and metadata. You can use this stream to tune or verify
[aggregation rules](/control/shaping/shape-metrics/rules/rollup).

You can still use [groupings and filters](#group-and-filter-metrics) from the
**Labels** and **Label values** tables while **Matched for aggregation** is selected.

To capture and review samples:

1. In the navigation menu, click **<Icon icon="shield-user" /> Go to Admin** and
   then select
   **<MicroscopeIcon /> Analyzers <span aria-label="and then">></span> Live Telemetry**.
2. Select the **Metrics** tab, and click **Capture live data**.
3. In the **Data phase** menu, under **Aggregation**, select **Matched for aggregation**.
4. Click the **Samples** tab. The tab label includes a count of captured samples.
5. In the **Filter then press Enter** field, enter `key=value` pairs,
   `key!=value` pairs for exclusions, or a bare token to match any field. Press
   **Enter** to apply the filter.
6. Scan the **Timestamp**, **Value**, and **Metric** table columns, plus any dynamic
   metadata columns populated for your traffic. Click a row to open a detailed view
   with labels and metadata in a separate pane.
7. Click **<Icon icon="pause" /> Enable or disable sample streaming** to pause
   updates and hold the current buffer, or resume streaming when you want updated
   samples.
8. To discard captured samples and reset the list, click
   **<Icon icon="trash" /> Drain current samples**.

Live Telemetry Analyzer buffers up to 200 samples.

## Group and filter metrics

The initial view displays two tables, which list labels for captured metrics. The
**Labels** table lists the label keys returned during the capture.

Use the **Search** text box to find a specific label. The **Search** text box filters
as you type, reducing the label list displayed. Live Telemetry Analyzer uses
[glob syntax](/investigate/querying/glob-syntax).

Select the checkbox next to any label to filter the **Label Values** table by the
selected value.

The right table shows the **Label Values**. Click a label value to add it to
the **Add Label Filter** text box.

Filter both tables by adding label `key:value` pairs to the **Add label filter**
field by selecting values in the **Label values** table, or type in the field. Typing
in the field displays a **Label** and **Value** text box. The **Label** field displays
a matching list of label keys as you type. Select an option from the list at any
time. Click the <Icon icon="check" /> check icon when finished. Click any label
value to edit it. The **Value** field supports
[glob syntax](/investigate/querying/glob-syntax).

<Note>
  Cortex XCOR glob syntax doesn't support using two asterisks where one of
  them is in the middle of a string. For example, `*k8s*staging` isn't valid.
</Note>

Click the arrow in any of the columns to sort by that data to help interpret the
results. For example, a high total percentage in the **Appears In** column with low
unique values gives you a high-level breakdown of where to attribute metrics. You can
also sort by the **Unique Values** column, which helps identify high-cardinality
labels.

Consider the following metrics as an example:

```text theme={null}
sign_up{location="placeA"}
sign_up{location="placeB"}
login{version="v0.1.0"}
```

By default, the **Label values** table groups results by `__name__` and
`__metric_type__`. The following example focuses on the labels supplied with the
metrics and excludes the [special request metadata](#special-request-metadata).

With these metrics, the Live Telemetry Analyzer generates three label rows based on
`__name__`, `location`, and `version`. Because every metric has a
`__name__` label, the percentage for that label is 100%. There are only two unique
values for `__name__`, which are `sign_up` and `login`, causing the **Unique Values**
column to display `2`. Only two metrics have the `location` label, which is `66%`,
and there are two unique values for this label (`placeA` and `placeB`). The same
applies for `version`.

| Label Keys | Unique Values | Appears In |
| - | - | - |
| `__name__` | 2 | 100% |
| `location` | 2 | 66% |
| `version` | 1 | 33% |

## Identify rules that generate metrics

When using the Live Telemetry Analyzer, you can view which metrics were rejected by a
drop rule or impacted by an aggregation rule. You can also view the specific rollup
rule or drop rule that caused a metric to be aggregated or dropped. Use this
information to help understand why metrics are missing, and why results are formatted
in a particular way. You can also click the rule name to go directly to the rule in
Cortex XCOR.

The Live Telemetry Analyzer displays all matching aggregation rules, which helps to
identify duplicates.

To understand which drop rules are dropping certain metrics:

1. In the **Data phase** menu, select **Rejected by drop rule**.
2. Under the **Labels** section, select the `__drop_rule_slug__` label.

   The drop rule slug names display in the **Label values** table.
3. Click the **<Icon icon="square-arrow-out-up-right" />** arrow icon to navigate directly to the
   drop rule that caused the metrics to be dropped.

To understand which aggregation rules are producing aggregated metrics:

1. In the **Data phase** menu, select **Accepted for storage**.
2. Under the **Labels** section, select the `__rollup_rule_slug__` label.

   The aggregation rule slug names display in the **Label values** table.
3. Click the **<Icon icon="square-arrow-out-up-right" />** arrow icon to navigate directly to the
   aggregation rule that produced the aggregated metric.

## Troubleshoot missing metrics

If metrics don't display when running the Live Telemetry Analyzer:

* Examine the filters to ensure they're not dropping the metrics you're searching
  for.
* Review the **Collectors** dashboard and ensure metrics are being scraped by the
  [Collectors](/ingest/metrics-traces/collector).
* See [metric limits](/administer/limits-licensing/limits/metric-limits) for more
  information.


## Related topics

- [Analyze live traffic traces](/investigate/analyze/trace-analyzer.md)
- [Analyze metrics usage](/investigate/analyze/usage.md)
- [Analyze data](/investigate/analyze.md)
- [Verify the Collector is scraping metrics](/ingest/metrics-traces/collector/monitor/verify-metrics.md)
- [Review aggregation rules](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.