Optimize your telemetry data

Your app generates a lot of data, but not all of it’s actionable. Storing every byte is costly, and can quickly overwhelm your capacity limit, license limit, or both. Aside from financial cost, there’s computing cost associated with generating, processing, and temporarily storing your telemetry data. Reducing telemetry data volume keeps costs predictable and manageable as your business scales, and helps teams manage their data effectively by reducing the amount of available information. When you optimize your telemetry data, on-call engineers can find the data they need to solve problems faster. To take control of your telemetry data, Palo Alto Networks Cortex XCOR provides multiple tools you can use to identify and eliminate data that isn’t used and doesn’t deliver value to your business. See Data optimization concepts to learn about the mechanisms you can use to control your telemetry data. The following diagram shows how the stages of the control workflow connect, and each stage links to its section on this page. The workflow is a loop rather than a one-time setup, because partitions and budgets reveal what your data consumes, which sends you back to your shaping rules. For a guide that defines this workflow, including how to set thresholds that alert or drop data, see Manage telemetry data budgets.

​

Reduce volume

After sending data to Cortex XCOR, use the Live Telemetry Analyzer to identify opportunities to reduce the overall volume of telemetry data. If you’re sending Prometheus metrics, you can also use the to identify obsolete or unnecessary data, and understand the impact of a proposed shaping rule to users of that data.

​

Shape and sample your data

Use the Cortex XCOR Control Plane to shape and sample your telemetry data and reduce the amount of data you persist. After implementing partitions and budgets to manage consumption, you can revisit optimization rules to transform, reshape, retain, or exclude telemetry data.

​

Review rules

After creating shaping rules, use the various tools that Cortex XCOR provides to review the impact. If your shaping rules reduced too much data or not enough, make changes to ensure the rules achieve the intended outcome.

​

Create partitions

To help manage telemetry data consumption, create partitions, which are slices of your data with distinct owners. Partitions provide a consistent structure for attributing usage and costs to the appropriate owners in your organization so they can isolate and control independent their parts of the business.

​

Analyze consumption

After defining partitions, use the Consumption page to view persisted log data consumption against license capacity. This page helps to detect unwanted spikes and understand what’s driving growth across your telemetry data. Use this information to review shaping rules and make changes that impact your budgets. Manage license consumption by reviewing licensing dashboards to identify usage trends and avoid exceeding your organization’s licensing limits. Understand telemetry data limits for each telemetry type to ensure a performant experience when ingesting, querying, and creating resources.

​

Create budgets

After creating partitions and analyzing consumption, create and apply budgets, which are shaping policies you assign to a partition to safeguard against runaway usage and overspending. Budgets help track and manage growth to ensure your license consumption remains predictable.