Code Coverage Not working for Visual Studio Test results

We’re using Sonar Cloud to analyse our Visual Studio code. We have discovered for some reason the code coverage aspect of Sonar Cloud has never worked for us. In our Azure DevOps builds we correctly configure the prepare analysis step and we can see that the files below are created;

16 | 16:55:58.694  The following code coverage attachments were found from the trx files: F:\Azagent\******[Obfiscated by me]\GBIBC-TFS17BUILD_GBIBC-ITD-22-V 2020-10-02 16_53_17.coverage

The run code analysis step then identifies the files but the results are never displayed in the Sonar Cloud web interface. 

905 | INFO: Sensor C# [csharp] (done) | time=15502ms

906 | INFO: Sensor C# Tests Coverage Report Import [csharp]

907 | INFO: Parsing the Visual Studio coverage XML report F:\Azagent\A1\_work\160\.\a\TestResults\GBIBC-TFS17BUILD_GBIBC-ITD-22-V 2020-10-02 16_53_05\In\GBIBC-ITD-22-V\GBIBC-TFS17BUILD_GBIBC-ITD-22-V 2020-10-02 16_53_17.coveragexml

908 | INFO: Adding this code coverage report to the cache for later reuse: F:\Azagent\A1\_work\160\.\a\TestResults\GBIBC-TFS17BUILD_GBIBC-ITD-22-V 2020-10-02 16_53_05\In\GBIBC-ITD-22-V\GBIBC-TFS17BUILD_GBIBC-ITD-22-V 2020-10-02 16_53_17.coveragexml

909 | INFO: Coverage Report Statistics: 223 files, 4 main files, 4 main files with coverage, 219 test files, 0 project excluded files, 0 other language files.

910 | INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=3844ms

911 | INFO: Sensor C# Unit Test Results Import [csharp]

912 | INFO: Parsing the Visual Studio Test Results file 'F:\[***Obfiscated by me]\GBIBC-TFS17BUILD_GBIBC-ITD-22-V 2020-10-02 16_53_05.trx'.

913 | INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=4266ms

914 | INFO: Sensor Zero Coverage Sensor

915 | INFO: Sensor Zero Coverage Sensor (done) | time=1328ms

We’re using Visual Studio 2015 Professional on our build server to create the code coverage report.

Any ideas why SonarCloud is not interpreting the coveragexml file? We have managed to view it in a version of Visual Studio Enterprise and it appears to be a valid coverage report.

2020-10-07T08:33:00Z We have subsequently installed a DevOps build agent on a local machine where Visual Studio Enterprise is installed and ran the build and code coverage again. A .coveragexml file was created but not imported by Azure Dev Ops or Sonar Cloud. When we try to open them in Visual Studio we get an error reading that the coveragexml file is not a valid coverage xml file or not the expected version.
Any further suggestions about what the problem may be would be greatly appreciated.

1 Like

hi @Roadkingscot

Could you please give us the verbose output of the commands (please add the sonar.verbose parameter set to true in the prepare step, and please attach the output of analysis step)? The DEBUG statements in the logs may tell us more.

I don’t understand here - you are saying that the coverage file can be opened, and then that it cannot…

Can you send us the coverage file to test it locally?

Hi,

I’m from the same team as Scot and have been looking into the issue also. Thanks for getting back to us and apologies for the delay in responding.

Please find attached the .coverage file, .coveragexml file, and the verbose output of the prepare and run code analysis stages.
CodeCoverageAndVerboseOutput_slewis96-12_10_2020.zip (1.2 MB)
The .coverage file opens successfully in Visual Studio Enterprise but the .coveragexml file returns us an error stating it’s “not valid or not the expected version”

hi @slewis96 @Roadkingscot

Please accept my apologies for the delay.

I took a look in the logs. Important line:

2020-10-12T12:19:10.6616013Z 13:19:10.646 INFO: Coverage Report Statistics: 223 files, 4 main files, 4 main files with coverage, 219 test files, 0 project excluded files, 0 other language files.

You can open the coveragexml in notepad++ and check its contents.

Is the coverage file that you uploaded the one that is successfully parsed on sonarcloud (with only 4 main sources), or a different one? More specifically: are the logs in the zip archive the ones for the coveragexml?

Because when I debug the parser with [svc_acc]_[build_server]-10-12 13_10_01.coveragexml, I see 293 files being found in the report, not 223 like in the logs you’ve sent.

It would be nice to better understand this.

Thanks!
Andrei

Hi,

That’s no problem, we seem to have everything working now.

Previously we changed the datacollector line in our testsettings file to:
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
This didn’t however fix our issue.

Instead what we then tried was swapping our testsettings file for a runsettings file (attached) for all our projects along with ensuring that the test results weren’t going into a temp directory and instead were being outputted into the Build.ArtifactStagingDirectory\TestResults this successfully solved our issue.
From what we can see there is no difference between the coverage files created but may have something to do with when using a testsettings file Azure DevOps creates a merged testsettings file and an additional runsettings file at the time of unit test execution.
RunsettingsAndOuput.zip (6.1 KB)

We aren’t 100% on why this is now fixed, but it’s now working well.

Thanks for the help,
Seb

1 Like