Flaky code coverage with .NET Core test task in Azure DevOps

SonarQube Server / Community Build
1

Hi,

We are experiencing an issue on our SonarQube integration for Azure DevOps. Every run reports different coverage (around 50% of our projects are missing code coverage data). However every run excludes different projects every time.

We are using MSBuild integration, and the DotNetCli test task.

The task is set to publishTestResults=true. The .coverage data in Azure DevOps reports correctly, i.e. we believe there is an issue in the conversion from .coverage to .coveragexml as explained in this thread.

We have 21 unit tests projects. If we run the tests in parallel it only tries to read in ~14 coveragexml files. As you can see in the output these coveragexml files are named with suffixes due to name clashes of them completing simultaneously.
parallel_run.txt (5.2 KB)

However, if we run the test in sequence all the coverage is loaded and corresponds to the .coverage data.
sequential_run.txt (7.2 KB)

Is this a known issue? Are there any other ways to circumvent it than to run the tests in sequence?

SonarQube task version info:

Task         : Run Code Analysis
Description  : Run scanner and upload the results to the SonarQube server.
Version      : 4.9.0
1 Like
3

An update. The issue seems to be that the task falsely identifies two separate coverage files as duplicates, due to them having the same name (in different directories), hence filtering out some of the .coverage files. I ran the task with System.Debug=true and this is the output.

parallel_run_debug.txt (9.8 KB)

sequential_run_debug.txt (11.1 KB)

Please note that in the parallel run the Unique coverage files are much lower (12), while in the sequential run they are the expected number (21).

4

Hi,

So yes, indeed, we are only relying on the file name, not the full path, that’s the reason why if counts for duplicates, even if they aren’t located on the same folder.

I created a ticket to improve this :https://github.com/SonarSource/sonar-scanner-msbuild/issues/848

In the mean time, i suggest that you stay on the sequential run, or, you can create a script after the tests and before the Run Code Analysis task that will take care of renaming some files to avoid duplicates.

Mickaël

5

I created a PR in your repo addressing the issue.

6

Thank you very much. We’ll review it and get back to you.

1 Like