Sonarscanner converts codecoverage to xml then appears to not use it

We’re analyzing code coverage on Azure Pipelines, with the VSTest task create a .coverage file. Looking at the logs, Sonarscanner then uses the codecoverage.exe to convert the file to xml which reports no errors. See below:

2020-01-15T08:03:39.0953044Z 08:03:39.077 The following code coverage attachments were found from the trx files: d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage
2020-01-15T08:03:39.0953376Z 08:03:39.077 Not using the fallback mechanism to detect binary coverage files.
2020-01-15T08:03:39.1075579Z 08:03:39.093 Executing file C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe
2020-01-15T08:03:39.1076197Z Args: analyze /output:d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coveragexml d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage
2020-01-15T08:03:39.1077531Z Working directory: d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28
2020-01-15T08:03:39.1077758Z Timeout (ms):60000
2020-01-15T08:03:39.1077962Z Process id: 4224
2020-01-15T08:03:41.1948132Z 08:03:41.191 Process returned exit code 0
2020-01-15T08:03:41.2270324Z Generating SonarQube project properties file to d:\a\1.sonarqube\out\sonar-project.properties
2020-01-15T08:03:41.2621049Z Setting analysis property: sonar.visualstudio.enable=false

However, after the job is almost complete, the scanner tries to parse the .coverage file rather than the .coveragexml file and then fails. Resulting in no code coverage being uploaded:

2020-01-15T08:04:41.9188498Z 08:04:41.193 INFO: Parsing the Visual Studio coverage XML report d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage
2020-01-15T08:04:41.9189509Z 08:04:41.208 WARN: Unable to get next XML event while parsing file ‘d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage’
2020-01-15T08:04:41.9196373Z 08:04:41.208 WARN: Could not import coverage report ‘d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage’ because ‘Error while parsing the XML file: d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage’
2020-01-15T08:04:41.9201017Z 08:04:41.208 DEBUG: The current user dir is ‘d:\a\1’.
2020-01-15T08:04:41.9201957Z 08:04:41.208 INFO: Parsing the Visual Studio coverage XML report d:\a\1\s\TestResults\dbef655a-6461-4635-8bd5-5bd0ebd33a5a\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage
2020-01-15T08:04:41.9202423Z 08:04:41.208 WARN: Unable to get next XML event while parsing file ‘d:\a\1\s\TestResults\dbef655a-6461-4635-8bd5-5bd0ebd33a5a\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage’
2020-01-15T08:04:41.9202907Z 08:04:41.208 WARN: Could not import coverage report ‘d:\a\1\s\TestResults\dbef655a-6461-4635-8bd5-5bd0ebd33a5a\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage’ because ‘Error while parsing the XML file: d:\a\1\s\TestResults\dbef655a-6461-4635-8bd5-5bd0ebd33a5a\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage’
2020-01-15T08:04:41.9203341Z 08:04:41.208 DEBUG: Analyzing coverage after aggregate found ‘0’ coverage files.
2020-01-15T08:04:41.9203697Z 08:04:41.208 DEBUG: The total number of file count statistics is ‘0’.
2020-01-15T08:04:41.9204071Z 08:04:41.208 INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=15ms
2020-01-15T08:04:41.9204440Z 08:04:41.208 INFO: Sensor C# Unit Test Results Import [csharp]

Using properties:
sonar.cs.nunit.reportsPaths=(System.DefaultWorkingDirectory)\TestResults\**\*.trx, sonar.cs.vscoveragexml.reportsPaths=(System.DefaultWorkingDirectory)\TestResults***.coverage

Once the conversion has taken place, should the scanner not be referencing the xml files rather than trying to read the .codecoverage binary file?

1 Like

Greetings,

Do you pass sonar.cs.vscoveragexml.reportsPath yourself? If so – you should set it to the location of the eventually produced XML files (I’d be a little surprised if you have to set anything at all)

Yes, additional properties as the comment:
sonar.cs.vscoveragexml.reportsPaths=(System.DefaultWorkingDirectory)\TestResults***.coverage

The converted xml file is in the same folder as the coverage binary. As sonar performs the conversion, would this not need to stay at .coverage?

I’ve just found that using .coverage* it may work, but you still get errors where sonar is trying to parse xml from the .coverage file (rather than the converted file).

Hi, is it a known bug? Where it finds the .coverage from the path provided, converts it to xml, but then proceeds to try and parse the original .coverage file?

sonar.cs.vscoveragexml.reportsPaths should be set to the XML report that is generated (.xml, not the .coverage report). The Scanner for MSBuild handles this before analysis actually starts, if I remember correctly.

Sure, but it’s Sonars own task that is converting the .coverage binary file in the path to coveragexml using codecoverage.exe. So, until the sonar task runs, there is no xml file. The path points to the binary coverage file that sonar finds, and then converts as below, but then goes back to the binary file when processing.

If I change the path to the xml straight away, sonar won’t convert the coverage binary, and therefore there is no xml file.

Executing file C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe
2020-01-15T08:03:39.1076197Z Args: analyze /output:d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coveragexml d:\a\1\s\TestResults\VssAdministrator_fv-az28_2020-01-15_08_03_06\In\fv-az28\VssAdministrator_fv-az28 2020-01-15 08_02_43.coverage

Hi, is there a more formal way to raise this as a bug?

Hi.

This is the proper channel to raise issues – but remember that contributors on this forum are doing it out of goodwill, with no SLA. If you need shorter response times, I reccomend looking into commercial support.

You seem to know a lot about how this works :slight_smile: And, I think you have come to a community to ask experts, so you might trust our advice!

That is not how it works – the SonarScanner for MSBuild checks the .trx (test results) file for an associated .coverage attachment, and then falls back to searching the temp directory. The value of sonar.cs.vscoveragexml.reportPaths is of no consequence here.

For example (this already appears to be happening in the logs you’ve shared)

2019-01-09T14:07:33.9623117Z 14:07:33.663  Fetching code coverage report information from TFS...
2019-01-09T14:07:33.9623193Z 14:07:33.665  Attempting to locate a test results (.trx) file...
2019-01-09T14:07:33.9623430Z 14:07:33.671  Looking for TRX files in: D:\a\1\TestResults, D:\a\1\s\TestResults
2019-01-09T14:07:33.9623520Z 14:07:33.671  The following test results files were found: D:\a\1\s\TestResults\VssAdministrator_fv-az24 2019-01-09 14_07_26.trx
2019-01-09T14:07:33.9623609Z 14:07:33.683  The following code coverage attachments were found from the trx files: D:\a\1\s\TestResults\VssAdministrator_fv-az24 2019-01-09 14_07_26\In\fv-az24\VssAdministrator_fv-az24 2019-01-09 14_06_54.coverage 

The file will be converted, and that file would then need to be provided to sonar.cs.vscoveragexml.reportsPaths (and should happen by default most of the time. No need to supply any value). Also, just to correct myself from earlier, it is specifically a .coveragexml file, not an .xml file expected.

I’d really encourage you to stop setting sonar.cs.vscoveragexml.reportsPaths at all and see what happens.

Colin

1 Like

Thank you. Just specifying the trx did the trick where it found the coverage, converted and then used the coveragexml.

And once I realised that the VSTest tasks in Azure Devops delete trx files once the next one runs, we seem to be in business!

Thanks for you help, very much appreciated.

1 Like

Hi,

I have the same problem what did you do to solve?

Hii , you said that like

sonar.cs.vscoveragexml.reportsPaths=C:\Sonar\DemoProject\TestResults*.coverage**

Only using this:
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)\Sonar***.trx

and not specifying the coverage or setting location which sonar seems to find itself.

1 Like

Hi AndrewB
In my case does not generated .trx what I will do
when i converted .coverage file to xml at that time i used codecoverage.exe analyze /output:“location of coverage file” but it’s not generated

i am using visual studio

I’m using Azure Devops, so the sonar extension will convert the coverage file itself (using the codecoverage.exe), but this way it’s then aware of the location of the file.

with or without sonar parameters, I still can’t see the analysis report on SonarQube.
No problem, sonarscanner finds the .trx and the .xml

12:01:18.958 The following code coverage attachments were found from the trx files: D:\agent_work\1\s\TestResults… 2020-07-01 11_52_01.coverage
12:01:18.958 Not using the fallback mechanism to detect binary coverage files.
12:01:18.989 Executing file C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe
Args: analyze /output:D:\agent_work\1\s\TestResults… 2020-07-01 11_52_01.coveragexml D:\agent_work\1\s\TestResults… 2020-07-01 11_52_01.coverage
Working directory: D:\agent_work\1\s\TestResults…
Timeout (ms):60000
Process id: 9576

12:03:07.510 INFO: Parsing the Visual Studio coverage XML report D:\agent_work\1\s\TestResults… 2020-07-01 11_52_01.coveragexml

12:03:07.604 INFO: Adding this code coverage report to the cache for later reuse: D:\agent_work\1\s\TestResults… 2020-07-01 11_52_01.coveragexml
12:03:07.604 DEBUG: Analyzing coverage after aggregate found ‘0’ coverage files.
12:03:07.604 DEBUG: The total number of file count statistics is ‘0’.
12:03:07.604 INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=94ms
12:03:07.604 INFO: Sensor C# Unit Test Results Import [csharp]
12:03:07.604 DEBUG: Pattern matcher extracted prefix/absolute path ‘D:\agent_work\1\s\TestResults…_2020-07-01_11_53_04.trx’.
12:03:07.604 DEBUG: Pattern matcher returns a single file: ‘D:\agent_work\1\s\TestResults…2020-07-01_11_53_04.trx’.
12:03:07.604 DEBUG: The current user dir is ‘D:\agent_work\1’.
12:03:07.604 INFO: Parsing the Visual Studio Test Results file ‘D:\agent_work\1\s\TestResults…_2020-07-01_11_53_04.trx’.
12:03:07.619 DEBUG: Parsed Visual Studio Test Times - duration: 553138.
12:03:07.619 DEBUG: Parsed Visual Studio Test Counters - total: 214, failed: 0, errors: 0, timeout: 0, aborted: 0, executed: 214.
12:03:07.619 INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=15ms
12:03:07.619 INFO: ------------- Run sensors on project
12:03:07.619 DEBUG: ‘Generic Coverage Report’ skipped because one of the required properties is missing

So, coverage file is found, but it seems to be ignored.
As a result I have 0% in SonarQube for my project

Whereas I can see that the test result is OK in Azure DevOps

Somebody knows why the coverage is not reported in SonarQube ?

IMHO this is a very important tip that should be more clear on the docs. I spent some hours trying to make it work 'til I get here and finally got it.

Only with deep knowledge about how the SonarScanner works someone can be able to achieve an complete analysis on a dotnet framework application. Please, pay attention on the checked answer for more details.

The one line answare for your answare is:

Just declare the extra property sonar.cs.vstest.reportsPaths

The sonar.cs.vscovaragexml.reportsPath property acts like an alternative, but with a different use case.

Hi,

Thank you for your tip but unfortunately it doesn’t work !

I add the porperties in the “prepare the sonarqube analysis task”
sonar.cs.vstest.reportsPaths=$(Common.TestResultsDirectory)*.trx

Then, the search of trx files is well skipped because it detects that the path has been provided, as you said

10:39:36.279 Attempting to locate the CodeCoverage.exe tool using setup configuration…
10:39:36.295 Code coverage command line tool: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe
10:39:36.295 Property ‘sonar.cs.vstest.reportsPaths’ provided, skipping the search for TRX files in default folders…
10:39:36.513 Absolute path to coverage file: D:\agent_work\1\s\TestResults\XXXXXXXX.coverage
10:39:36.513 The following code coverage attachments were found from the trx files: D:\agent_work\1\s\TestResults\XXXXXXXX.coverage
10:39:36.513 Not using the fallback mechanism to detect binary coverage files.

But it doesn’t change anything and the analysis doesn’t embedded the coverage in SonarQube.

Other information later in the logs

sonar.cs.vstest.reportsPaths=D:\agent\_work\1\TestResults\.trx*
sonar.visualstudio.enable=false

10:41:23.169 DEBUG: Pattern matcher extracted prefix/absolute path ‘D:\agent_work\1\TestResults’ from the given pattern ‘D:\agent_work\1\TestResults*.trx’.
10:41:23.169 DEBUG: Gathering files for wildcardPattern '.trx’.*
10:41:23.169 DEBUG: Pattern matcher returns ‘0’ files.