Skip to main content
The BlazeMeter API Monitoring integration connects Cortex XCOR with BlazeMeter API Monitoring, the product formerly called Runscope. BlazeMeter API Monitoring runs your API tests on a schedule from its cloud locations or from your own Radar agents. XCOR collects every passing and failing test run in the buckets your access token can see. The collected results include pass or fail, assertion, script, and variable results, response times, status codes, and response sizes for each request step.

Supported telemetry types

The BlazeMeter API Monitoring integration supports these telemetry types:

Compatibility

The BlazeMeter API Monitoring integration works with:
  • BlazeMeter API Monitoring tests, through the API Monitoring API at https://api.runscope.com.
  • Every bucket the access token’s user can see, unless you list the buckets to collect or to skip. Buckets added later are picked up automatically, unless you list the buckets to collect.
  • Runs from cloud locations and from Radar agents.
  • Every test in the buckets one configuration collects.
XCOR reads each collected test’s results once a minute and lists your buckets, tests, and test steps every five minutes, using your access token.

Prerequisites

The BlazeMeter API Monitoring integration requires:
  • A BlazeMeter API Monitoring access token. Creating an API application in BlazeMeter API Monitoring issues an access token for your own account. For details, see Authentication in the BlazeMeter API Monitoring API documentation.
  • The api:read, test:read, and team:read scopes on the token. BlazeMeter doesn’t document which scopes the access token it issues for your own account carries.
The access token provides read access to every bucket its user can see. If the token carries write scopes, it can also make changes, although the integration only reads. Create the API application as a dedicated BlazeMeter API Monitoring user that belongs to only the teams whose buckets you want collected.

Configure the integration

  1. In BlazeMeter API Monitoring, sign in as the dedicated user, create an API application, and copy the access token that BlazeMeter issues for your own account.
  2. Decide which buckets to collect. Leave both bucket lists empty to collect every bucket the token can see. To collect only some buckets, list their keys in Buckets to collect. To collect all but some, list their keys in Buckets to skip. Use one list or the other, not both. A bucket key is the identifier in the bucket’s API URLs, not its name.
  3. In XCOR, open the Integrations Hub and select BlazeMeter API Monitoring.
  4. On the Configurations tab, add a configuration, enter a name, the access token, and your bucket list, and save.
A configuration collects every test in the buckets you select. XCOR paces the requests it makes with each access token, and reads every detail of at least about 375 tests per configuration. When a larger selection outruns that pace, the runs’ results still come first: some step details are skipped and some results can arrive late. Every BlazeMeter API Monitoring configuration in your account shares the same collection capacity. Past about 520 tests in one configuration, or when two configurations each hold more than about 375 tests, XCOR can fall behind. Then some runs aren’t collected, for every configuration and not only the large ones. Splitting a selection across configurations or access tokens doesn’t add that capacity. The larger the selection, the longer each listing of your tests and steps takes, so new tests and steps can take longer to be picked up. To collect every detail of a large selection, narrow it. Two configurations that share a token spend its request rate twice as fast. Give each access token one configuration, and don’t let two configurations collect the same bucket. Two configurations that can both see a bucket, through the same token or through users who share a team, read its runs twice. This doubles the number of BlazeMeter API Monitoring requests and can occasionally count a run twice. You can’t save a second configuration that uses the same connection as an existing one, but XCOR can’t detect when you enter the same token in two configurations.

Save validation

When you save, XCOR checks the token’s form and verifies the token by listing one bucket in BlazeMeter API Monitoring. A failure returns one of these messages:
  • Invalid value for field `access_token`: must not contain whitespace or control characters: the pasted token includes a space or a line break. XCOR refuses it before calling BlazeMeter API Monitoring.
  • BlazeMeter API Monitoring access_token is not valid: BlazeMeter API Monitoring rejected the token. Check that you copied the whole token and that it hasn’t been revoked.
  • BlazeMeter API Monitoring refused the credential check (HTTP <code>): BlazeMeter API Monitoring refused the request for a reason other than the token.
  • BlazeMeter API Monitoring credential verification temporarily unavailable (HTTP <code>): BlazeMeter API Monitoring is throttling or unavailable. Try again later.
  • BlazeMeter API Monitoring credential verification got an unexpected answer (HTTP <code>), or BlazeMeter API Monitoring credential verification failed: <reason>: XCOR couldn’t complete the check. Try again later, and contact Cortex XCOR Support if it persists.
You also can’t save a configuration that lists buckets to collect and buckets to skip together. Save doesn’t validate the token’s scopes or which buckets the token can see. A configuration with a token whose user can’t see any buckets saves, but collects nothing until the user can see a bucket.

Validate metrics ingestion

  1. Allow up to about 20 minutes after saving for the first metrics to appear. XCOR starts collecting within about 10 minutes of saving, and runs that finish before then might not be collected. After that, make sure a test in a collected bucket finishes a run, and look for it about 7 minutes later.
  2. On the Configurations tab, check the configuration’s status. OK means the token works and can see every bucket in Buckets to collect. Error comes with a message, which Troubleshooting explains.
  3. In Metrics Explorer, count the runs collected in the last 30 minutes. Each run writes one data point, stamped at the run’s finish time, so use a range rather than the current instant:
    A result means data is flowing. The newest data point is about 5 minutes old or older, so for tests that run less often than every 20 minutes, widen the range. To filter for one bucket, add its key, for example RunscopeTestRun_test_failed{bucket_key="<your bucket key>"}.

Dashboards

The integration includes an Overview dashboard that shows failing tests, failed runs and pass rate by bucket, passed and failed runs by test, response times by test and request step, failed assertions, the share of request steps collected without step detail, and collection volume. Open it from the Assets tab on the BlazeMeter API Monitoring integration page in the Integrations Hub. For more about bundled dashboards, see Use bundled assets.

Metric semantics

The 13 RunscopeTestRun_* metrics hold one value for each run, and the 12 RunscopeTestRequest_* metrics hold one value for each request step of a run. They’re the metric names Runscope sends to Datadog, so queries ported from Datadog keep their metric names and label names, except agent_expired and the requested URL tag on request metrics, which the integration doesn’t carry. Label values are normalized the way Datadog normalizes Runscope’s tags, so a ported filter on bucket, test_name, or environment matches the same value. One value differs by design: run metrics’ status reads pass or fail where Runscope sends passed or failed, so change a ported filter such as status:passed to status="pass". The names contain no dots, so plain selectors such as RunscopeTestRun_test_failed{bucket="payments"} work. Every series carries bucket, test_name, bucket_key, and test_uuid. It can also carry team, environment, and environment_uuid. A run from a cloud location carries region and region_name, the location’s name, such as us_virginia, and a run from a Radar agent carries agent. Run metrics add status, which reads pass or fail. Request metrics add request_method, note, and step_index, which records the step’s position in the run as a value from 1 to 100. Runscope’s Datadog metrics have no step_index, so aggregate it away. The Metrics tab lists every label. XCOR normalizes provider label values the way Datadog normalizes tag values, including bucket_key and test_uuid, so filter on the lowercased form. Each value is lowercased, every character other than a letter, digit, -, :, ., or / becomes _, repeated underscores collapse to one, and a trailing underscore is removed. A leading underscore stays, as it does in Datadog. For example, a bucket named Payments API (v2) reads payments_api_v2, and a step note that starts [Orders] Create reads _orders_create. Every series also carries {"xcor.integration.name"="blazemeterapimonitoring"}. Because a series has a data point only when a run finishes:
  • Wrap every query in a range function, such as sum_over_time, avg_over_time, or max_over_time, over a range at least as long as the time between runs plus about 10 minutes. A plain selector rarely shows data.
  • Count failed runs with sum(sum_over_time(RunscopeTestRun_test_failed[30m])). Each run writes RunscopeTestRun_test_failed once, as 1 for a failed run and 0 for a passed run, so the failure ratio is sum(sum_over_time(RunscopeTestRun_test_failed[1h])) / sum(count_over_time(RunscopeTestRun_test_failed[1h])).
  • The average of RunscopeTestRun_test_failed across series isn’t the failure ratio. Each value of status is its own series, so the average weighs series, not runs. Datadog shows the same average for the same query.
  • Each data point arrives about 5 to 7 minutes after its run finishes. A query or monitor over the last 5 minutes can come back empty even when a run failed in that time, so use a range of at least 10 minutes.
  • After long-term downsampling, about five days after ingest, XCOR keeps the last data point of each series in each five-minute interval. Over older ranges, run and failure counts undercount a test that finishes more than one run with the same result, location, and environment within five minutes, and failure ratios can read higher or lower than they were, usually higher for a test that mostly passes.
For the full metric catalog, open the Metrics tab on the BlazeMeter API Monitoring integration page in the Integrations Hub.

Troubleshooting

The following table lists symptoms and how to resolve them:

Limitations

  • Only runs that pass or fail are collected, once they finish. Runs that are still running or queued, canceled runs, and runs that take longer than 15 minutes from start to finish aren’t collected.
  • Only a test’s own HTTP request steps are collected, up to entry 100 in a run. Requests that a subtest step makes aren’t collected. If a collected request step occurs after entry 100, RunscopeTestRun_total_response_time is absent.
  • Response times might not match the values Runscope sent to Datadog for the same run. XCOR takes a step’s response time from the step’s detail when BlazeMeter API Monitoring returns it, and otherwise adds up the step’s timings, so the source can vary from run to run. Without the step’s detail, a step that received no response can’t be told apart, so its response time is the time it waited.
  • Runscope’s per-run Datadog events, its request URL tag, and its data iteration metrics aren’t collected. The integration collects no logs, traces, or events, and uses no webhooks.
  • Run history from before XCOR starts collecting isn’t collected. After an interruption on the XCOR side, runs that finished more than about an hour before collecting resumed aren’t collected. Runs that finish while BlazeMeter API Monitoring rejects the token, or throttles it past the waits XCOR allows, aren’t collected later. Neither are runs that finish while their bucket isn’t selected.
  • Each data point arrives several minutes after its run finishes, which is later than many rollup rules accept, so a rollup rule on these metrics can miss data points. When XCOR answers an aggregate query from such a rule, a query like sum(sum_over_time(RunscopeTestRun_test_failed[30m])) can come back lower than the stored data.
  • Labels set in a configuration’s metric_labels field through the Cortex XCOR API aren’t added to these metrics.
  • XCOR reads only the public API at https://api.runscope.com. A private or regional API host isn’t supported.