> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Identify issues behind suspicious tracing trends

> Learn how to use Differential Diagnosis in Trace Explorer to identify tag:value pairs most closely correlated with suspicious behavior.

When an incident occurs, you can use Trace Explorer to
[search and filter](/investigate/querying/traces) trace data to identify
trends in requests, errors, or latencies in services and related operations. After
identifying a trend, you want to understand why it's happening.

Palo Alto Networks Cortex XCOR includes
[Differential Diagnosis (DDx)](/investigate/querying/traces/features#differential-diagnosis),
which lets you identify trends and immediately scan through all related tags and values
to pinpoint the exact `tag:value` pairs most closely correlated with suspicious behavior.

You can compare `tag:value` pairs to illustrate how data is changing, which helps
determine whether data is usually correlated, indicating normal operations, or
unusually correlated, which can indicate underlying causes of errors or latency.

Comparing data over time shows how the data is changing. It distinguishes data that's
always clustered in errors or slow requests from data that's *recently* clustered in
errors or slow requests. This comparison
indicates a possible source of errors or spikes in latency. For example, a developer
who's debugging errors can compare current data against data immediately before a
deploy to determine what's changed.

Administrators can [promote top tags](#promote-top-tags) in your Cortex XCOR tenant so
those tags always display in DDx results.

<Tip>
  Learn more about DDx, see
  [Troubleshoot microservices issues faster with Differential Diagnosis](https://chronosphere.io/learn/troubleshoot-faster-with-ddx/)
  on the Palo Alto Networks blog.
</Tip>

## Access and use DDx

You can access DDx insights from different areas of Trace
Explorer. The method you choose depends on your entry point and what data you're
investigating:

* **Span statistics**: Use this tab to locate interesting or alarming trends in errors or
  latencies. After identifying a service or service and operation with increasing
  trends, use DDx to understand what issues are causing the trend.

* **Differential Diagnosis (DDx)**: Use this tab to determine the health of the service or
  service and operation your team owns. Go directly to this tab to determine whether
  errors, successes, and latencies are distributed across environments, regions,
  build versions, or other important contexts.

* **Trace list**: Use this tab to explore a specific trace from an alert or a related
  log. Locate spans with very long durations or considerable errors. Then use
  DDx to analyze similar spans for a common trend that might
  explain unexpected behavior.

<Note>
  You can query traces directly [from a dashboard](/observe/dashboards/query-traces),
  and then complete DDx on the defined query. Cortex XCOR
  applies the context from your query to Trace Explorer.
</Note>

To access and use DDx:

1. In the navigation menu select
   **<Icon icon="compass" /> Explorers <span aria-label="and then">></span> Trace Explorer**.

2. Choose the time period to display results for, which defaults to the last 15
   minutes.

3. Use the **Query Builder** to
   [define a search](/investigate/querying/traces#define-a-search), and then
   click **<Icon icon="refresh-cw" /> Run**.

   If the time range for your query is broad, such as the last five days, Trace
   Explorer loads results progressively. As results are available, you can click
   **Display more results** to view partial results.

   When all results are available, click **Display complete results** to view the
   complete list of results.

4. Use one of the following options to access DDx insights:

   * On the **Span statistics** tab, click a service or service and operation, and then
     click **Differential Diagnosis** in the resulting dialog.

   * Click the **Differential Diagnosis (DDx)** tab, and then select a service or service
     and operation.

     <Note>
       A shortcut to view traces related to your DDx query is to
       click **View traces** on the **Differential Diagnosis (DDx)** tab. The resulting
       dialog displays individual traces matching your query in the same view presented
       in the **Trace list** tab.

       Click **Update Trace Explorer query** to apply the differential diagnosis
       criteria you specified to the **Query Builder** in Trace Explorer.
     </Note>

   * Click the **Trace list** tab. Click the name of a trace in the **Trace** column to
     open the trace details page. Select a span, and then click **Differential Diagnosis**.

     The **Differential Diagnosis (DDx)** tab displays insights for the criteria you
     selected.

5. Select one or more tags to display the distribution of those `tag:value` pairs
   across [several metrics](/investigate/querying/traces/features#differential-diagnosis-metrics)
   simultaneously.

6. To compare trends for selected tags against a point in the past, make a selection
   from the **Compare to past** dropdown. For example, selecting **1 hour prior**
   updates each of the charts with statistics for the selected tags compared to the
   values one hour ago.

7. To view trends for your data, click **Over time**. The visualizations change to
   line graphs, which display errors, successes, and duration over time for each of
   the selected tags in the chosen time period. Duration for each tag is represented
   in microseconds (µs).

   Use this feature in conjunction with the **Compare to past** dropdown to identify
   anomalies, including where and when they started. You can also
   [overlay change events](/observe/enable-events/traces) on the **Over time**
   panels to help identify what changed in your environment, and whether those events
   correlate with changes in errors, successes, or duration.

8. To sync the order of all the bars in other charts to a specific chart, use the
   **Chart sorting** dropdown. For example, if you choose **Sync to error spans**,
   each of the charts update to reflect the ordering of the **Error spans** chart.
   This capability lets you compare the same tag across different heuristics.

## Promote top tags

Administrators can promote tags to display in the **Top tags** section of the
**Differential Diagnosis (DDx)** tab. These tags always display in DDx
results for all users in your Cortex XCOR tenant.

You must have administrative privileges to complete this task.

<Note>
  You can also promote top tags from the
  [Live Telemetry Analyzer](/investigate/analyze/trace-analyzer#promote-tags-for-cortex-xcor-differential-diagnosis-ddx).
</Note>

To promote tags to top tags:

1. In the navigation menu, click **<Icon icon="shield-user" /> Go to Admin** and
   then select
   **<Icon icon="compass" /> Explorers <span aria-label="and then">></span> Trace Explorer**.

2. Click the **Differential Diagnosis (DDx)** tab.

3. In the **Find tags** field, enter the name of the tag you want to promote to a top
   tag.

4. Hold the pointer over the tag, click the <Icon icon="ellipsis-vertical" /> three
   vertical dots icon, and then click **Add to top tags**.

   The selected tag moves to the **Top tags** section, and is automatically included
   in DDx results.

5. Complete the previous step to promote additional tags.

To remove a tag from the **Top tags** section, hold the pointer over the tag, click
the <Icon icon="ellipsis-vertical" /> three vertical dots icon, and then click
**Remove from top tags**.


## Related topics

- [Identify issues behind suspicious trends](/investigate/analyze/differential-diagnosis.md)
- [Identify issues behind suspicious metrics trends](/investigate/analyze/differential-diagnosis/metrics.md)
- [Service pages](/observe/services/service-pages.md)
- [Ingest, query, and control your trace data](/overview/types/traces.md)
- [Trace Explorer features overview](/investigate/querying/traces/features.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.