Code Coverage Metrics not matching Azure DevOps

SonarQube Cloud
, , ,
1

  • CI system used
    Azure Dev Ops

  • Scanner command used
    SonarScanner.MSBuild.exe begin /k:"[redacted]" /d:sonar.login="[redacted]" /o:"[redacted]" /d:sonar.host.url="https://sonarcloud.io" /d:sonar.vbnet.vscoveragexml.reportsPaths="$(Agent.TempDirectory)\**\*.coveragexml" /d:sonar.vbnet.vstest.reportsPaths="$(Agent.TempDirectory)\**\*.trx"

  • In the above scanner command I’ve also tried adding /d:sonar.cs.vscoveragexml.reportsPaths commands and /d:sonar.cs.vstest.reportsPaths - However I get the same coverage metric, whether or not I provide both vbnet and cs or either cs or vbnet. No difference, which is fine.

  • I then run the build and tests using msbuild and vstest

  • Followed by
    SonarScanner.MSBuild.exe end /d:sonar.login="[redacted]"

  • Languages of the repository
    Mostly VB.NET & C #.NET (.NET FRAMEWORK 4.8)

Observations

  • Azure DevOps provides me a coverage metric of 73.84%, yet SonarCloud shows 30.6%.

  • I’ve run comparisons between the .coveragexml file that is sent to SonarCloud, against the .coverage file that is published in Azure Dev Ops, and the only common statistics are around Blocks of code. There are minute differences, but the numbers more or less line up.

  • The processed .coveragexml file sent to Sonar Cloud contains additional stats around lines of code

I’ve seen this post which probably explains why I see this difference:- C#/VB.NET: SonarQube and SonarCloud support Branch/Condition Coverage data, but I can’t be sure.

Questions:- Have I configured the scanner command correctly? Is there any more data I can mine out of my build pipeline? The task for the team is to ensure the metric in Sonar Cloud matches what we see in Azure Dev Ops

1 Like
2

Hi,

There are possibly two mechanisms at work here. The first is coverage exclusions. If you’ve configured them in-line in your files, your native coverage engines are honoring those exclusions, but SonarCloud is not.

And that brings us to the second mechanism: executable lines. This old blog post introduces the concept. Basically, if your coverage report omits a file for whatever reason, we helpfully count the lines in it that could (should?) have been tested and include them in the calculations.

If either of these things is at work, manually (re)setting your coverage exclusions - ideally via the UI - will start to bring the numbers back into line.

 
HTH,
Ann

4

Thanks Ann,

I’ll have a look at the coverage exclusions today, and then report back with a meaningful update.

1 Like
5

This topic was automatically closed after 5 days. New replies are no longer allowed.