Applying optimization rules
Optimization rules run after both parsers and field mappings run. Disabled rules are skipped. Enable rules to have Cortex XCOR evaluate them. Cortex XCOR applies optimization rules serially and in order. Each rule operates on the output of the rule before it. Effects of previous rules apply immediately to the next rule in the sequence. If a rule matches the same condition as a previous rule, the subsequent rule operates only on the outcome of the previous rule. For example, if the first rule samples logs by 50% whereservice = "auth" and the second rule
also includes this condition, the second rule evaluates only the remaining 50% of the
original data volume.
Cortex XCOR includes the following optimization rules:
- Drop logs that match a specific filter to remove unnecessary or low-value data.
- Drop fields from logs that match specified conditions, which helps reduce the size of individual logs. If a log contains no data after fields are dropped, you can choose to drop the entire log.
- Sample logs to retain a certain percentage of matching logs and drop the remainder, which can reduce volume and maintain visibility.
- Emit metric converts log data to metric data. Use this optimization rule when you want to reduce the volume of logs, retain data for a longer period, and increase the speed of querying and filtering for metrics instead of logs.
- Replace field identifies a specified field in your log data and uses a regular expression to replace any matches based on the selected replacement mode. Use this optimization rule to truncate long fields and preserve your log structure.
- Parse field formats and splits unstructured fields into multiple fields, conditionally parses logs with similar fields based on a source field, and remaps fields.
View optimization rules
Select from one of the following methods to view optimization rules.- Web
- Chronoctl
- Terraform
- API
To view optimization rules:
- In the navigation menu, click Go to Admin and then select Optimization > Logs Ingest. All defined optimization rules display in the Optimization rules section.
- To view only optimization rules, from the View dropdown, select Optimization rules.
- To view the code definition for all configured optimization rules, click the Code config tab. The Logs optimization rules section includes all defined optimization rules.
Create optimization rules
Use optimization rules to drop certain logs at ingestion, or fields that you don’t want to persist. For example, dropping log data from a specific cluster, test data in a development environment, or unnecessary individual fields across your logs. When creating optimization rules, be sure you understand how Cortex XCOR applies optimization rules. Use one of the following methods to create optimization rules for log data:- Web
- Chronoctl
- Terraform
- API
Define optimization rules in Cortex XCOR, and then
use the Code config tool to apply the rule definition.
- In the navigation menu, click Go to Admin and then select Optimization > Logs Ingest.
- In the Optimization rules section, click Create.
- In the Create optimization rule page, enter a name for the optimization rule.
-
From the Action menu, select the type of rule to create:
- Drop logs: Completely drop logs that match the specified filter.
- Drop fields: Remove specific fields from logs that match the specified filter.
- Sample logs: Retain a certain percentage of matching logs and drop the remainder.
- Emit metric: Convert log data to metric data.
- Replace field: Replace context within a log field using a regular expression.
- Parse field: Structure, format, or separate fields out of a source field.
-
Enter a filter to return log data for the optimization rule, and then press
Ctrl+Enter(Command+Returnon macOS) to submit the filter. This rule applies only to logs that match this filter at the time the log data was ingested. -
Review the returned data from the preview filter and make changes as necessary.
Optimization rules are applied during ingestion, so the preview filter results might differ from a filter submitted in Logs Explorer after ingestion. By default, including a dot (
.) in a filter indicates nesting within a field.In some instances, fields with a key that uses dot notation within the name, such asresource.type, must be surrounded by double quotes and brackets in the optimization rule filter.If a warning message displays indicating thatThis filter is applied at ingestionand the filter returns no results, wrap the key in double quotes (") and brackets ([]). For example, the following filter contains theresource.typekey, which requires surrounding the field in double quotes and brackets for this particular service: -
Enter additional information for the specified action:
- Drop logs
- Drop fields
- Sample logs
- Emit metric
- Replace fields
- Parse fields
Review the filter and make changes as needed. - Click Save to save the optimization rule definition.
- Click the Code config tab and use the Code config tool to apply the definition.
Chronoctl example
The following example defines an optimization rules file for Chronoctl, and contains the following rules:- The first rule samples 10% of logs that match the defined filter.
- The second rule drops all logs for the
nginxservice of severityINFOwhere thehttp.Request.statusequals200. - The third rule drops all
kubernetesfields that match the defined filter. The regular expression indicates that any fields matchingkubernetes.[FIELD]are dropped from all logs. - The fourth rule uses a mapped value to search for the
error_codefield, and replaces values based on the provided key-value pairs. If the rule matches onINTERNAL_SERVER_ERROR, it replaces that value with500. If the rule matches onNOT_FOUND, it replaces the value with401. - The fifth rule emits a counter metric from NGINX error logs. Each matching log
increments
nginx_errors_totalby one. Labels on the metric come from theserviceanderror_codelog fields. - The sixth rule parses the
raw_requestfield on matching NGINX access logs using a Grok pattern. The parsed fields are written to therequestfield.
Chronoctl example
Terraform example
The following example defines an optimization rules resource for Terraform, and contains the following rules:- The first rule samples 10% of logs that match the defined filter.
- The second rule drops all logs for the
nginxservice of severityINFOwhere thehttp.Request.statusequals200. - The third rule drops all
kubernetesfields that match the defined filter. The regular expression indicates that any fields matchingkubernetes.[FIELD]are dropped from all logs. - The fourth rule uses a mapped value to search for the
error_codefield, and replaces values based on the provided key-value pairs. If the rule matches onINTERNAL_SERVER_ERROR, it replaces that value with500. If the rule matches onNOT_FOUND, it replaces the value with401. - The fifth rule emits a counter metric from NGINX error logs. Each matching log
increments
nginx_errors_totalby one. Labels on the metric come from theserviceanderror_codelog fields. - The sixth rule parses the
raw_requestfield on matching NGINX access logs using a Grok pattern. The parsed fields are written to therequestfield.
Terraform example
Edit optimization rules
When creating or editing optimization rules, you can use the Code config tool to view code representations of the rules. The displayed code also responds to changes you make in the Visual editor tab. Select from the following methods to edit optimization rules.- Web
- Chronoctl
- Terraform
- API
- In the navigation menu, click Go to Admin and then select Optimization > Logs Ingest.
- In the row of the optimization rule you want to edit, click the three vertical dots icon and then click Edit optimization rule.
- In the Edit optimization rule drawer, make changes to your optimization rule, and then click Save.
- Click the Code config tab and use the Code config tool to apply the definition.
Reorder optimization rules
Optimization rules are applied in descending order. To change the order optimization rules are applied, reorder rules. Disabled rules are skipped.- Web
- Chronoctl
- Terraform
- API
- In the navigation menu, click Go to Admin and then select Optimization > Logs Ingest.
- In the Optimization rules section, click Re-order rules.
- In the Edit optimization rules drawer, in the row of the optimization rule you want to move, click the drag indicator icon and move the optimization rule to a different order position.
- Click Save.
- Click the Code config tab and use the Code config tool to apply the definition.
Enable or disable optimization rules
Each rule has a mode, which can be enabled or disabled. By default, any created rule is enabled. Any rule that’s enabled is applied. Disabled rules are skipped.- Web
- Chronoctl
- Terraform
- API
- In the navigation menu, click Go to Admin and then select Optimization > Logs Ingest.
- In the row of the optimization rule you want to enable or disable, click the toggle in the Enabled column.
- Click Save.
- Click the Code config tab and use the Code config tool to apply the definition.
Delete optimization rules
Select from the following methods to delete optimization rules.- Web
- Chronoctl
- Terraform
- API
- In the navigation menu, click Go to Admin and then select Optimization > Logs Ingest.
- In the row of the optimization rule you want to delete, click the three vertical dots icon and then click Delete optimization rule.
- Click the Code config tab and use the Code config tool to apply the definition.