Scanning VB project in DevOps 2019 no code coverage registered

SonarQube Server / Community Build
, ,
1

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    SonarQube 7.9, Azure Devops Server 2019, .NET 4.7.2

  • what are you trying to achieve
    I have a project written in VBnet, but tests written in C#. Im trying to use the Azure Devops Server sonar tasks to analyze and report code coverage to a sonar server.

  • what have you tried so far to achieve this
    I have been at this for several days now.

I run tasks in the following order

  • SonarQubePrepare
  • VSBuild
  • VSTest
  • SonarQubeAnalyze
  • SonarQubePublish

I have checked the checkbox “code coverage” in the VSTest-task

When reviewing the logs i find the following.

During the VSTest phase two files are created a trx file and seperate coverage file.

Snippet from the logs

Results File: D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20.trx

Attachments:
  D:\BuildAgents\Agent1\_work\8\s\TestResults\3884bd45-6d21-4008-be94-0dcd7906ada9\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coverage

But during the SonarQubeAnalyzis task, i can see that it locates the trx file, and from that file it derives another coverage file and not the i posted above.

This file is located in a in folder (some sort of temp folder).

17:33:29.4  Attempting to locate a test results (.trx) file...
17:33:29.51  Looking for TRX files in: D:\BuildAgents\Agent1\_work\8\TestResults, D:\BuildAgents\Agent1\_work\8\s\TestResults
17:33:29.51  The following test results files were found: D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20.trx
17:33:29.619  Absolute path to coverage file: D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20\In\E75LWNWS1286V\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coverage
17:33:29.619  The following code coverage attachments were found from the trx files: D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20\In\E75LWNWS1286V\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coverage
17:33:29.619  Not using the fallback mechanism to detect binary coverage files.
17:33:29.635  Executing file D:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe
  Args: analyze /output:D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20\In\E75LWNWS1286V\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coveragexml D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20\In\E75LWNWS1286V\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coverage 
  Working directory: D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20\In\E75LWNWS1286V
  Timeout (ms):60000
  Process id: 9604
17:33:29.681  Process returned exit code 0

TLDR

// Generated in test step
D:\BuildAgents\Agent1\_work\8\s\TestResults\3884bd45-6d21-4008-be94-0dcd7906ada9\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coverage

// Derived from the trx file
D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20\In\E75LWNWS1286V\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coverag

The settings that SonarQubeAnalyzer is running are the following

sonar.cs.vstest.reportsPaths=D:\\BuildAgents\\Agent1\\_work\\8\\s\\TestResults\\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20.trx
sonar.cs.vscoveragexml.reportsPaths=D:\\BuildAgents\\Agent1\\_work\\8\\s\\TestResults\\E75LWNWS1286V$_E75LWNWS1286V_2020-04-14_17_33_20\\In\\E75LWNWS1286V\\SYSTEM_E75LWNWS1286V 2020-04-14 17_33_14.coveragexml
sonar.host.url=********
sonar.projectKey=********
sonar.projectName=*********
sonar.projectVersion=1.0
sonar.pullrequest.key=**********
sonar.pullrequest.base=master
sonar.pullrequest.branch=feature/code-coverage
sonar.pullrequest.provider=vsts
sonar.pullrequest.vsts.instanceUrl=***********
sonar.pullrequest.vsts.project=***********
sonar.pullrequest.vsts.repository=**********
sonar.scanner.metadataFilePath=D:\\BuildAgents\\Agent1\\_work\\_temp\\sonar\\20200414.21\\90676aad-b831-becc-3fd6-d858b0a949c2\\report-task.txt
sonar.visualstudio.enable=false

I have not explicitly configured anything except for sonar.logging=verbose

But according to the settings, sonar thinks this is a C# project, and not a VB project.

The analysis finishes and i have 0% coverage

So my questions are the following:

Why the trxfile?
Why do i have two coverage files?
Why is SonarQube only using the one derived from the trx file.
Why does it think it is only a C# project when 90% of the project code is VBnet
Why no coverage?

2

so no one has any input?

3

still looking for any feedback in accordance to this issue

4

hi @Tandolf

could you please share with us the commands you are running in your CI script? are you running all commands from the same folder?

Also, can you please share the verbose logs? (please run SonarScanner.MSBuild.exe begin /k:"MyProject" /d:sonar.verbose=true as the begin step, and please attach the output of END step)

5

Here @Andrei_Epure here is the azure pipline setup including a report convertion step.

#prepare the analysis
- task: SonarQubePrepare@4
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'MSBuild'
    projectKey: 'foobar'
    projectName: 'foobar'
    extraProperties: |
      sonar.verbose=true

# Build is done here and test step, not included for abbriviation

# Convert the report
- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      $filePath = gci -Recurse ${Build.SourcesDirectory}TestResults\ -Filter *.coverage | ?{ $_.fullname -notmatch "\\In\\?" } | select -expand FullName
      Write-Host $filePath
      $filePath -replace " ", "` "
      $command = "CodeCoverage.exe analyze /output:$(Build.SourcesDirectory)\testresults\result.coveragexml " + "`"$filepath`""
      Write-Host $command
      iex $command


#Analyse and publish
- task: SonarQubeAnalyze@4

- task: SonarQubePublish@4
  inputs:
    pollingTimeoutSec: '300'
7

Thanks for the detailed response and the logs!

The TRX file is for test results (we don’t push the results for SQ, you can read here details)

14:39:28.575 INFO: Sensor C# Unit Test Results Import [csharp]
14:39:28.575 INFO: Parsing the Visual Studio Test Results file D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-22_14_39_09.trx
14:39:28.591 INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=16ms

The coverage XML is for code coverage - what you’re interested in.

14:39:28.512 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-22_14_39_09\In\E75LWNWS1286V\SYSTEM_E75LWNWS1286V 2020-04-22 14_39_03.coveragexml
14:39:28.575 INFO: Adding this code coverage report to the cache for later reuse: D:\BuildAgents\Agent1\_work\8\s\TestResults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-22_14_39_09\In\E75LWNWS1286V\SYSTEM_E75LWNWS1286V 2020-04-22 14_39_03.coveragexml
14:39:28.575 INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=63ms

If you want VB.NET tests to appear, you should use the VBNET property:

sonar.vbnet.vscoveragexml.reportsPaths
9

Thank you @Andrei_Epure for your quick response. I updated the azure-pipelines.yml with adding the sonar.vbnet.vscoveragexml.reportsPaths property shown here below. But code coverage still shows 0%

- task: SonarQubePrepare@4
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'MSBuild'
    projectKey: 'foobar'
    projectName: 'foobar'
    extraProperties: |
      sonar.verbose=true
      sonar.vbnet.vscoveragexml.reportsPaths="$(Build.SourcesDirectory)\testresults\*"

I can read out the following in the logs

11:55:53.492 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\Agent1\_work\8\s\testresults\result.coveragexml
11:55:53.492 INFO: Adding this code coverage report to the cache for later reuse: D:\BuildAgents\Agent1\_work\8\s\testresults\result.coveragexml

So it looks like that it did find the reports, i still don’t understand why im getting 0%, quite frustrating

here are the logs for the latest run
sonarlogs.txt (251.5 KB)

10

and i have another question

i can see this in the logs

11:55:53.476 INFO: Sensor VB.NET Tests Coverage Report Import [vbnet]
11:55:53.492 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\Agent1\_work\8\s\testresults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-24_11_55_32
11:55:53.492 WARN: Could not import coverage report 'D:\BuildAgents\Agent1\_work\8\s\testresults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-24_11_55_32' because 'java.io.FileNotFoundException: D:\BuildAgents\Agent1\_work\8\s\testresults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-24_11_55_32 (Access is denied)'
11:55:53.492 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\Agent1\_work\8\s\testresults\f65fe4e2-4fed-48cb-99a8-40a412de7d7e
11:55:53.492 WARN: Could not import coverage report 'D:\BuildAgents\Agent1\_work\8\s\testresults\f65fe4e2-4fed-48cb-99a8-40a412de7d7e' because 'java.io.FileNotFoundException: D:\BuildAgents\Agent1\_work\8\s\testresults\f65fe4e2-4fed-48cb-99a8-40a412de7d7e (Access is denied)'
11:55:53.492 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\Agent1\_work\8\s\testresults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-24_11_55_32.trx
11:55:53.492 WARN: Could not import coverage report 'D:\BuildAgents\Agent1\_work\8\s\testresults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-24_11_55_32.trx' because 'Missing root element <results> in D:\BuildAgents\Agent1\_work\8\s\testresults\E75LWNWS1286V$_E75LWNWS1286V_2020-04-24_11_55_32.trx at line 2'
11:55:53.492 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\Agent1\_work\8\s\testresults\result.coveragexml
11:55:53.492 INFO: Adding this code coverage report to the cache for later reuse: D:\BuildAgents\Agent1\_work\8\s\testresults\result.coveragexml
11:55:53.492 INFO: Sensor VB.NET Tests Coverage Report Import [vbnet] (done) | time=16ms

Im converting the rapports using CodeCoverage.exe manually as a build step, is this step really needed? I’t looks like in the logs that SonarAnalyzer is trying to find reports that it has converted?

11

That’s because you don’t provide a path to a folder with only coverage reports, and the scanner will open all files in $(Build.SourcesDirectory)\testresults\* and see if they’re valid coverage reports or not.

11:55:53.492 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\Agent1\_work\8\s\testresults\result.coveragexml
11:55:53.492 INFO: Adding this code coverage report to the cache for later reuse: D:\BuildAgents\Agent1\_work\8\s\testresults\result.coveragexml
11:55:53.492 INFO: Sensor VB.NET Tests Coverage Report Import [vbnet] (done) | time=16ms

I just realized now that you are using SonarQube 7.9, and since then we have greatly improved the debug logging of coverage importing. Can you please update the SonarCSharp and SonarVBNet plugins to the latest version (from the SQ Marketplace on your instance) or at least 8.1 where we improved the verbose logging.

And then redo the verbose scan and add the logs.

Also - can you verify the coveragexml file is not empty?