Supported telemetry types
The BlazeMeter API Monitoring integration supports these telemetry types:Compatibility
The BlazeMeter API Monitoring integration works with:- BlazeMeter API Monitoring tests, through the API Monitoring API at
https://api.runscope.com. - Every bucket the access token’s user can see, unless you list the buckets to collect or to skip. Buckets added later are picked up automatically, unless you list the buckets to collect.
- Runs from cloud locations and from Radar agents.
- Every test in the buckets one configuration collects.
Prerequisites
The BlazeMeter API Monitoring integration requires:- A BlazeMeter API Monitoring access token. Creating an API application in BlazeMeter API Monitoring issues an access token for your own account. For details, see Authentication in the BlazeMeter API Monitoring API documentation.
- The
api:read,test:read, andteam:readscopes on the token. BlazeMeter doesn’t document which scopes the access token it issues for your own account carries.
Configure the integration
- In BlazeMeter API Monitoring, sign in as the dedicated user, create an API application, and copy the access token that BlazeMeter issues for your own account.
- Decide which buckets to collect. Leave both bucket lists empty to collect every bucket the token can see. To collect only some buckets, list their keys in Buckets to collect. To collect all but some, list their keys in Buckets to skip. Use one list or the other, not both. A bucket key is the identifier in the bucket’s API URLs, not its name.
- In XCOR, open the Integrations Hub and select BlazeMeter API Monitoring.
- On the Configurations tab, add a configuration, enter a name, the access token, and your bucket list, and save.
Save validation
When you save, XCOR checks the token’s form and verifies the token by listing one bucket in BlazeMeter API Monitoring. A failure returns one of these messages:Invalid value for field `access_token`: must not contain whitespace or control characters: the pasted token includes a space or a line break. XCOR refuses it before calling BlazeMeter API Monitoring.BlazeMeter API Monitoring access_token is not valid: BlazeMeter API Monitoring rejected the token. Check that you copied the whole token and that it hasn’t been revoked.BlazeMeter API Monitoring refused the credential check (HTTP <code>): BlazeMeter API Monitoring refused the request for a reason other than the token.BlazeMeter API Monitoring credential verification temporarily unavailable (HTTP <code>): BlazeMeter API Monitoring is throttling or unavailable. Try again later.BlazeMeter API Monitoring credential verification got an unexpected answer (HTTP <code>), orBlazeMeter API Monitoring credential verification failed: <reason>: XCOR couldn’t complete the check. Try again later, and contact Cortex XCOR Support if it persists.
Validate metrics ingestion
- Allow up to about 20 minutes after saving for the first metrics to appear. XCOR starts collecting within about 10 minutes of saving, and runs that finish before then might not be collected. After that, make sure a test in a collected bucket finishes a run, and look for it about 7 minutes later.
- On the Configurations tab, check the configuration’s status. OK means the token works and can see every bucket in Buckets to collect. Error comes with a message, which Troubleshooting explains.
-
In Metrics Explorer,
count the runs collected in the last 30 minutes. Each run writes one data point, stamped at the
run’s finish time, so use a range rather than the current instant:
A result means data is flowing. The newest data point is about 5 minutes old or older, so for tests that run less often than every 20 minutes, widen the range. To filter for one bucket, add its key, for example
RunscopeTestRun_test_failed{bucket_key="<your bucket key>"}.
Dashboards
The integration includes an Overview dashboard that shows failing tests, failed runs and pass rate by bucket, passed and failed runs by test, response times by test and request step, failed assertions, the share of request steps collected without step detail, and collection volume. Open it from the Assets tab on the BlazeMeter API Monitoring integration page in the Integrations Hub. For more about bundled dashboards, see Use bundled assets.Metric semantics
The 13RunscopeTestRun_* metrics hold one value for each run, and the 12 RunscopeTestRequest_*
metrics hold one value for each request step of a run. They’re the metric names Runscope sends to
Datadog, so queries ported from Datadog keep their metric names and label names, except
agent_expired and the requested URL tag on request metrics, which the integration doesn’t
carry. Label values are normalized the way Datadog
normalizes Runscope’s tags, so a ported filter on bucket, test_name, or environment matches
the same value. One value differs by design: run metrics’ status reads pass or fail where
Runscope sends passed or failed, so change a ported filter such as status:passed to
status="pass". The names contain no dots, so plain selectors such as
RunscopeTestRun_test_failed{bucket="payments"} work.
Every series carries bucket, test_name, bucket_key, and test_uuid. It can also carry
team, environment, and environment_uuid. A run from a cloud location
carries region and region_name, the location’s name, such as us_virginia, and a run from a
Radar agent carries agent. Run metrics add
status, which reads pass or fail. Request metrics add request_method, note, and
step_index, which records the step’s position in the run as a value from 1 to 100. Runscope’s
Datadog metrics have no step_index, so aggregate it away. The Metrics tab lists every label.
XCOR normalizes provider label values the way Datadog normalizes tag values, including
bucket_key and test_uuid, so filter on the lowercased form. Each value is lowercased, every
character other than a letter, digit, -, :, ., or / becomes _, repeated underscores
collapse to one, and a trailing underscore is removed. A leading underscore stays, as it does in
Datadog. For example, a bucket named Payments API (v2) reads payments_api_v2, and a step note
that starts [Orders] Create reads _orders_create. Every series also carries
{"xcor.integration.name"="blazemeterapimonitoring"}.
Because a series has a data point only when a run finishes:
- Wrap every query in a range function, such as
sum_over_time,avg_over_time, ormax_over_time, over a range at least as long as the time between runs plus about 10 minutes. A plain selector rarely shows data. - Count failed runs with
sum(sum_over_time(RunscopeTestRun_test_failed[30m])). Each run writesRunscopeTestRun_test_failedonce, as 1 for a failed run and 0 for a passed run, so the failure ratio issum(sum_over_time(RunscopeTestRun_test_failed[1h])) / sum(count_over_time(RunscopeTestRun_test_failed[1h])). - The average of
RunscopeTestRun_test_failedacross series isn’t the failure ratio. Each value ofstatusis its own series, so the average weighs series, not runs. Datadog shows the same average for the same query. - Each data point arrives about 5 to 7 minutes after its run finishes. A query or monitor over the last 5 minutes can come back empty even when a run failed in that time, so use a range of at least 10 minutes.
- After long-term downsampling, about five days after ingest, XCOR keeps the last data point of each series in each five-minute interval. Over older ranges, run and failure counts undercount a test that finishes more than one run with the same result, location, and environment within five minutes, and failure ratios can read higher or lower than they were, usually higher for a test that mostly passes.
Troubleshooting
The following table lists symptoms and how to resolve them:Limitations
- Only runs that pass or fail are collected, once they finish. Runs that are still running or queued, canceled runs, and runs that take longer than 15 minutes from start to finish aren’t collected.
- Only a test’s own HTTP request steps are collected, up to entry 100 in a run. Requests
that a subtest step makes aren’t collected. If a collected request step occurs after entry 100,
RunscopeTestRun_total_response_timeis absent. - Response times might not match the values Runscope sent to Datadog for the same run. XCOR takes a step’s response time from the step’s detail when BlazeMeter API Monitoring returns it, and otherwise adds up the step’s timings, so the source can vary from run to run. Without the step’s detail, a step that received no response can’t be told apart, so its response time is the time it waited.
- Runscope’s per-run Datadog events, its request URL tag, and its data iteration metrics aren’t collected. The integration collects no logs, traces, or events, and uses no webhooks.
- Run history from before XCOR starts collecting isn’t collected. After an interruption on the XCOR side, runs that finished more than about an hour before collecting resumed aren’t collected. Runs that finish while BlazeMeter API Monitoring rejects the token, or throttles it past the waits XCOR allows, aren’t collected later. Neither are runs that finish while their bucket isn’t selected.
- Each data point arrives several minutes after its run finishes, which is later than many rollup
rules accept, so a rollup rule on these metrics can miss data points. When XCOR answers
an aggregate query from such a rule, a query like
sum(sum_over_time(RunscopeTestRun_test_failed[30m]))can come back lower than the stored data. - Labels set in a configuration’s
metric_labelsfield through the Cortex XCOR API aren’t added to these metrics. - XCOR reads only the public API at
https://api.runscope.com. A private or regional API host isn’t supported.