TSQL Unit Test and Code Coverage Results

steftt · July 7, 2020, 3:59pm

I am trying to get coverage stats for our databases within SQ. Scans for issues etc. work as expected however I cannot get stats for tests and Code Coverage.

Unit Tests are created in Visual Studio and we are running ADO Builds. Typical build steps are:

1: Prepare Analysis on SQ - tried with and without the following advanced parameter - /d:sonar.cs.vscoveragexml.reportsPaths=$(Common.TestResultsDirectory)/**/*.coveragexml
2: Build Solution - Visual Studio Build
3: Test Assemblies - Visual Studio Test - Code Coverage Enabled
4: Run Code Analysis
5: Publish Quality Gate Result

Run Code Analysis seems to locate the files as expected:

The following test results files were found: D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-07_16_46_43.trx

The following code coverage attachments were found from the trx files: D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-07_16_46_43\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-07 16_46_16.coverage

Args: analyze /output:D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-07_16_46_43\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-07 16_46_16.coveragexml D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-07_16_46_43\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-07 16_46_16.coverage

However I do not see any further analysis/processing on these files.

Any help would be appreciated

Thanks

Joshua_Martin · July 7, 2020, 6:48pm

Howdy!

There a few things that can cause this, but to know for sure, I would recommend running the analysis with debug logging enabled. You should be able to do this by setting -Dsonar.verbose=true in the command that runs the SonarScanner. With that set, you’ll be able to see more detailed logs when coverage is collected, particularly, whether or not the coverage sensor is skipping coverage data, and why. (mistmatched filenames is a common culprit in that case).

Anywho, let me know if you find something in the Debug logs.

steftt · July 8, 2020, 8:31am

Thanks Joshua, I added debug logging and can see the following:

Finds the following files:

D:\BuildAgents\pwtfs04_db.2_work\102\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-08_09_10_20\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-08 09_09_54.coveragexml

D:\BuildAgents\pwtfs04_db.2_work\102\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-08_09_10_20\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-08 09_09_54.coverage

D:\BuildAgents\pwtfs04_db.2_work\102\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-08_09_10_20.trx

Logs the following configuration:

sonar.cs.vstest.reportsPaths=D:\BuildAgents\pwtfs04_db.2\_work\102\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-08_09_10_20.trx

sonar.cs.vscoveragexml.reportsPaths=D:\BuildAgents\pwtfs04_db.2\_work\102\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-08_09_10_20\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-08 09_09_54.coveragexml

‘VisualStudioCoverage’ skipped because there is no related file in current project
‘C# Unit Test Results Import’ skipped because there is no related file in current project

Thanks

Joshua_Martin · July 9, 2020, 2:33pm

That’s interesting. On its face, the error below means that the coverage sensor is looking in the directory provided, but isn’t finding a coverage xml file. Can you check the directory reported in the logs and make sure the coverage xml file is there?

D:\BuildAgents\pwtfs04_db.2\_work\102\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-08_09_10_20\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-08 09_09_54.coveragexml

I’ve also noticed coverage issues depending on what scanner version is being used. Can you give me a little more detail on your build environment?

Visual studio version.
msbuild version.
Scanner version.

Let me know when you can.

steftt · July 13, 2020, 3:42pm

Hi Joshua, I have checked the file and it exists, extract below, I have replaced object names with “XXX”:

>    <function id="8660" token="0x6000006" name="XXX()" namespace="XXXDatabaseTests" type_name="XXX" block_coverage="100.00" line_coverage="100.00" blocks_covered="3" blocks_not_covered="0" lines_covered="4" lines_partially_covered="0" lines_not_covered="0">
>           <ranges>
>             <range source_id="2" covered="yes" start_line="16" start_column="9" end_line="16" end_column="48" />
>             <range source_id="2" covered="yes" start_line="17" start_column="9" end_line="17" end_column="10" />
>             <range source_id="2" covered="yes" start_line="18" start_column="13" end_line="18" end_column="35" />
>             <range source_id="2" covered="yes" start_line="19" start_column="9" end_line="19" end_column="10" />
>           </ranges>
>         </function>

Here is the additional info as requested:

Visual studio version. - 15.9.28307.1093
msbuild version. - 4.7.1
Scanner version. 4.8.0

Thanks for your help so far.

steftt · July 15, 2020, 10:33am

So a bit more playing around, I added a c# file into the database solution and the sonar version changed from 4.8.0 to 4.11.0, and we now have the number of tests executed and skipped etc. in Sonarqube however we still do not have coverage.

It now looks like its processing the coveragexml file:

2020-07-15T10:14:42.3405053Z 11:14:42.333 INFO: Sensor C# Tests Coverage Report Import [csharp]

2020-07-15T10:14:42.3458150Z 11:14:42.333 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-15_11_13_25\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-15 11_13_13.coveragexml

2020-07-15T10:14:42.3736599Z 11:14:42.348 INFO: Adding this code coverage report to the cache for later reuse: D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-15_11_13_25\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-15 11_13_13.coveragexml

2020-07-15T10:14:42.3737068Z 11:14:42.348 INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=15ms

I have checked an application where we are getting coverage and test coverage and the scanner version is 4.8.0

Joshua_Martin · July 16, 2020, 2:30pm

Hey there!

The scanner version and msbuild version look good. I forgot to check on this, but can you check what version of the C# plugin you have on your SonarQube server? That’s the last part where version can affect coverage and overall analysis.

Also, to drill down on the coverage, can we set debug logging for the analysis? You can get these by setting -Dsonar.verbose=true in SONAR_SCANNER_OPTS. Then we can check the messages for the C# coverage sensor. The debug logs will show us why the sensor is ignoring coverage information.

steftt · July 17, 2020, 8:01am

Hi Joshua, the version in the log is: Processing plugin: csharp version 7.15.0.8572

Verbose has already been set to true in the prepare analysis step in ADO, I can see the following in the log file:

2020-07-15T10:14:40.5074400Z 11:14:40.505 DEBUG: ‘VisualStudioCoverage’ skipped because there is no related file in current project

2020-07-15T10:14:40.5171255Z 11:14:40.505 INFO: Sensor C# Properties [csharp]
2020-07-15T10:14:40.5212277Z 11:14:40.520 DEBUG: Analyzer working directory ‘D:\BuildAgents\pwtfs04_db.1_work\139.sonarqube\out\0\output-cs’ contains 5 .pb file(s)
2020-07-15T10:14:40.5215017Z 11:14:40.520 DEBUG: Analyzer working directory ‘D:\BuildAgents\pwtfs04_db.1_work\139.sonarqube\out\2\output-cs’ contains 5 .pb file(s)
2020-07-15T10:14:40.5217364Z 11:14:40.520 DEBUG: Analyzer working directory ‘D:\BuildAgents\pwtfs04_db.1_work\139.sonarqube\out\4\output-cs’ contains 5 .pb file(s)
2020-07-15T10:14:40.5219305Z 11:14:40.520 DEBUG: Found Roslyn issues report
2020-07-15T10:14:40.5226512Z 11:14:40.520 INFO: Sensor C# Properties [csharp] (done) | time=15ms

2020-07-15T10:14:42.1156654Z 11:14:42.098 DEBUG: ‘VisualStudioCoverage’ skipped because there is no related file in current project

2020-07-15T10:14:42.1164249Z 11:14:42.098 DEBUG: ‘T-SQL Sensor’ skipped because there is no related file in current project

2020-07-15T10:14:42.3404562Z 11:14:42.333 INFO: Sensor C# [csharp] (done) | time=125ms
2020-07-15T10:14:42.3405053Z 11:14:42.333 INFO: Sensor C# Tests Coverage Report Import [csharp]
2020-07-15T10:14:42.3458150Z 11:14:42.333 INFO: Parsing the Visual Studio coverage XML report D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-15_11_13_25\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-15 11_13_13.coveragexml
2020-07-15T10:14:42.3736599Z 11:14:42.348 INFO: Adding this code coverage report to the cache for later reuse: D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-15_11_13_25\In\PWTFS00004\TFSSERVICE_PWTFS00004 2020-07-15 11_13_13.coveragexml
2020-07-15T10:14:42.3737068Z 11:14:42.348 INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=15ms
2020-07-15T10:14:42.3737193Z 11:14:42.348 INFO: Sensor C# Unit Test Results Import [csharp]
2020-07-15T10:14:42.3737273Z 11:14:42.348 INFO: Parsing the Visual Studio Test Results file D:\BuildAgents\pwtfs04_db.1_work\139\s\TestResults\TFSSERVICE_PWTFS00004_2020-07-15_11_13_25.trx
2020-07-15T10:14:42.3737393Z 11:14:42.364 INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=16ms

Thanks

Joshua_Martin · July 17, 2020, 2:30pm

Thanks for Sharing that info. Can you confirm what version of SonarQube you’re using? Based on that, we may need to upgrade the C# plugin.

steftt · July 17, 2020, 3:11pm

Version is: 7.9.1.27448

Thanks