SonarCloud no longer returns (many) test-related measures from Web API

[znerd (Ernst de Haan)]

Since a couple of weeks (at least) it appears that the SonarCloud “GET measures” API no longer returns several test-related measures, even though the metrics are still supported according to the “GET metrics” API.

Steps to reproduce:

1. Call the GET api/metrics/search API with ps=500 to get all of them in one go: https://sonarcloud.io/api/metrics/search?ps=500
2. Search for keys that start with “test” – e.g. by searching for "key": "test; note these down.
3. Pick a SonarCloud project that has been scanned, and where you have passed test (coverage) information (e.g. using a JaCoCo data file); note down the project key.
4. Call the GET api/measures/component API and pass the component=KEY (replace KEY with your actual project key) and metricKeys=XXX (replace XXX with the keys you noted down in step 1 plus ncloc, comma-separated) – e.g. in the case of project key acme-utils: https://sonarcloud.io/api/measures/component?component=acme-utils&metricKeys=test_execution_time,test_errors,test_failures,test_success_density,tests,ncloc

Expected Results:

1. Call in step 1 succeeds with a 200 OK and valid JSON.
2. The JSON from step 1 contains at least dozens of metrics.
3. Step 2 returns with a 200 OK, and valid JSON.
4. In step 2, in the JSON, you find these metric keys: test_execution_time, test_errors, test_failures, test_success_density, tests and ncloc (order is irrelevant).

Actual results:

1. Call in step 1 succeeds with a 200 OK and valid JSON.
2. The JSON from step 1 contains 112 metrics.
3. In step 2, we indeed get these metric keys: test_execution_time, test_errors, test_failures, test_success_density, tests (order is irrelevant).
4. In the JSON, we only get ncloc, the other metrics are not returned.

Actual JSON returned in step 2 (only replaced both the project key and name by acme-utils):

{
  "component": {
    "id": "AYvTOZjQfHtM5VPZMJY4",
    "key": "acme-utils",
    "name": "acme-utils",
    "qualifier": "TRK",
    "measures": [ {
      "metric": "ncloc",
      "value": "31701"
    } ]
  }
}

[Colin]

Hey there.

@znerd

• In step 2, we indeed get these metric keys: test_execution_time, test_errors, test_failures, test_success_density, tests (order is irrelevant).

These measures will rely on test execution parameters being passed (which is different than coverage, and neither requires the other).

[znerd (Ernst de Haan)]

@Colin : OK, that’s something I didn’t know! Learned something today.

There were probably some changes over the past months, such as a Maven Surefire Plugin update (or perhaps even an update to the configuration).

Thanks. Will investigate and then update this post.

[znerd (Ernst de Haan)]

@Colin : That was indeed the problem. I wasn’t passing the test metrics; stupid me.

This happened:

• I created a Bitbucket Pipeline.
• During one step, I’m running something like mvn clean install, which generates Surefire XML reports.
• Then I package things up in using tar cf, and mark the .tar file as an artifact, so it gets transferred to all the subsequent steps (in which I would have to unpack them with tar xf).
• However, if Bitbucket Pipelines finds any Surefire XML reports on any of the steps, it reports those tests on that step. Therefore I dediced to exclude those files from the .tar file.
• What I didn’t realize is, that the Sonar Scanner then also doesn’t get the Surefire XML reports, and consequently it’s unable to pass those metrics to SonarCloud.

Solution I put in place:

• Don’t exclude the Surefire XML files from the .tar file.
• Instead, remove the Surefire XML files from every step that expanded the .tar file (right at the end of the step, before Bitbucket Pipelines looks for them).

All good now, thanks a lot !