Published Coverage Differs from VSTest Coverage Report

SonarQube Cloud
, ,
1

Quality Gate extension failing with lower than expected coverage:

image
image1735×1114 58.1 KB

Note: Unchanged lines are being reported as new lines.

VS Test coverage significantly higher:

image
image864×399 20.5 KB

  • Using the latest version, i.e. v4, of the Sonar build pipeline tasks.
  • ALM: Azure DevOps)
  • CI system: Azure DevOps
  • Scanner command used when applicable (private details masked)
  • Languages of the repository: C#
  • No workaround identified.

Have reviewed the pre-requisites here:

Git shallow fetch is disabled. Seems to have no impact either way.

Sonar Prepare build task log:

Task : Prepare Analysis Configuration

Description : Prepare SonarQube Cloud analysis configuration

Version : 4.2.5

Author : sonarsource

Help : [More Information]( Azure DevOps Extension | SonarQube Cloud | Sonar Documentation )

==============================================================================

[INFO] SonarQube Cloud: Server version: 8.0.0.94738

[INFO] SonarQube Cloud: Default branch of this repository is ‘refs/heads/main’

C:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\4.2.5\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe begin /k: /o:

SonarScanner for .NET 11.2.1

Using the .NET Framework version of the Scanner for .NET

Pre-processing started.

Preparing working directories…

12:11:48.711 Updating build integration targets…

12:11:49.555 Using SonarCloud.

12:11:51.954 The JRE provisioning is a time consuming operation.

JRE provisioned: OpenJDK21U-jre_x64_windows_hotspot_21.0.9_10.zip.

If you already have a compatible Java version installed, please add either the parameter “/d:sonar.scanner.skipJreProvisioning=true” or “/d:sonar.scanner.javaExePath=”.

12:11:52.972 Fetching analysis configuration settings…

12:11:54.199 Provisioning analyzer assemblies for cs…

12:11:54.199 Installing required Roslyn analyzers…

12:11:54.199 Processing plugin: architecturecsharpfrontend version 2.32.0.10055

12:11:54.308 Processing plugin: csharpenterprise version 10.27.0.140913

12:11:54.511 Processing plugin: securitycsharpfrontend version 11.36.0.47442

12:11:55.152 Provisioning analyzer assemblies for vbnet…

12:11:55.152 Installing required Roslyn analyzers…

12:11:55.152 Processing plugin: securityvbnetfrontend version 11.36.0.47442

12:11:55.215 Processing plugin: vbnetenterprise version 10.27.0.140913

12:11:55.293 Incremental PR analysis: Base branch parameter was not provided.

12:11:55.293 Cache data is empty. A full analysis will be performed.

12:11:55.340 Pre-processing succeeded.

Sonar Analysis build task log:

Task : Run Code Analysis

Description : Run scanner and upload the results to SonarQube Cloud.

Version : 4.2.5

Author : sonarsource

12:32:53.098 INFO: Miss the cache for 0 out of 0

12:32:53.098 INFO: Sensor JavaScript/TypeScript/CSS analysis [javascript] (done) | time=10ms

12:32:53.099 INFO: Sensor IaC Project Sensor [iac]

12:32:53.100 INFO: Sensor IaC Project Sensor [iac] (done) | time=2ms

12:32:53.101 INFO: Sensor CSharpArchitectureSensor [architecture]

12:32:53.113 INFO: Found 1 potential Udg file location(s) for “cs” in “C:\a\1\.sonarqube\out”

12:32:53.113 INFO: - C:\a\1\.sonarqube\out\architecture\cs

12:32:53.131 INFO: Architecture analysis is enabled with the following features: legacy, discovery, flaws, smells, deviations, directives

12:32:53.138 INFO: * Protobuf reading starting | memory total=264 | free=111 | used=152 (MB)

12:32:53.138 INFO: * Reading SonarArchitecture UDG data from directory “C:\a\1\.sonarqube\out\architecture\cs”

12:32:53.281 INFO: * Files successfully loaded: “1” out of “1”

12:32:53.281 INFO: * Purging externals (components not scanned) from graphs

12:32:53.283 INFO: * Purging excluded nodes from graphs

12:32:53.283 INFO: * Protobuf reading complete | memory total=264 | free=98 | used=165 (MB)

12:32:53.322 INFO: * Build architecture.graph.cs.namespace_graph.default_perspective hierarchy graph complete (filtered=false) | memory total=264 | free=94 | used=169 (MB)

12:32:53.348 INFO: * No intended architecture defined or detected, using an empty model

12:32:53.363 INFO: No directives configured

12:32:53.509 INFO: Sensor CSharpArchitectureSensor [architecture] (done) | time=408ms

12:32:53.510 INFO: Sensor Zero Coverage Sensor

12:32:53.511 INFO: Sensor Zero Coverage Sensor (done) | time=1ms

12:32:53.511 INFO: Sensor Architecture Telemetry [architecture]

12:32:53.512 INFO: Sensor Architecture Telemetry [architecture] (done) | time=1ms

12:32:53.529 INFO: ------------- Gather SCA dependencies on project

12:32:53.530 INFO: Inferred api base url ‘https://api.sonarcloud.io’ from host url ‘https://sonarcloud.io’.

12:32:53.699 INFO: Dependency analysis skipped

12:32:53.702 INFO: CPD Executor 8 files had no CPD blocks

12:32:53.702 INFO: CPD Executor Calculating CPD for 12 files

12:32:53.716 INFO: CPD Executor CPD calculation finished (done) | time=14ms

12:32:53.718 INFO: SCM Publisher SCM provider for this project is: git

12:32:53.718 INFO: SCM Publisher 8 source files to be analyzed

12:32:54.162 INFO: SCM Publisher 8/8 source files have been analyzed (done) | time=444ms

12:32:54.194 INFO: SCM revision ID ‘’

12:32:54.275 INFO: SCM writing changed lines

12:32:54.282 INFO: Merge base sha1:

12:32:54.374 INFO: SCM writing changed lines (done) | time=99ms

12:32:55.324 INFO: Analysis report generated in 1156ms, dir size=419 KB

12:32:55.366 INFO: Analysis report compressed in 42ms, zip size=104 KB

12:32:55.621 INFO: Analysis report uploaded in 255ms

12:32:55.621 INFO: ANALYSIS SUCCESSFUL, you can find the results at: SonarQube Cloud

12:32:55.621 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report

12:32:55.621 INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=

12:32:55.621 INFO: Inferred api base url ‘https://api.sonarcloud.io’ from host url ‘https://sonarcloud.io’.

12:32:56.012 INFO: Inferred api base url ‘https://api.sonarcloud.io’ from host url ‘https://sonarcloud.io’.

12:32:56.741 INFO: Successfully sent architecture data

12:32:56.741 INFO: ------------- Upload SCA dependency files

12:32:57.029 INFO: Sensor cache published successfully

12:32:57.192 INFO: Analysis total time: 20.956 s

12:32:57.194 INFO: SonarScanner Engine completed successfully

12:32:57.612 The scanner engine has finished successfully

12:32:57.612 Post-processing succeeded.

Sonar Publish build task log:

Task : Publish Quality Gate Result

Description : Publish SonarQube Cloud’s Quality Gate result on the Azure Pipelines build result. To be used after the actual analysis.

Version : 4.2.5

Author : sonarsource

Help : [More Information]( Azure DevOps Extension | SonarQube Cloud | Sonar Documentation )

==============================================================================

[INFO] SonarQube Cloud: Server version: 8.0.0.94738

[INFO] SonarQube Cloud: Task completed

[INFO] SonarQube Cloud: Overall Quality Gate status: failed

2

Hi @BDRichardson,

A couple of things going on here:

  1. You’re comparing two different scopes. The 50% figure is SonarQube’s Coverage on New Code, it only looks at the 2 lines flagged as new/changed in this analysis (1 of 2
    covered = 50%). The 64.48% from VSTest is coverage across the entire assembly (256/397 lines). These numbers aren’t meant to match, see SonarQube and code coverage for why comparing a tool’s raw report to SonarQube’s published metric is often misleading, and the New Code docs for how the New Code scope is defined.

  2. The “only 2 new lines” number seems a bit weird here.
    You mentioned unchanged lines are being reported as new, and the logs show that:
    Incremental PR analysis: Base branch parameter was not provided, Cache data is empty. A full analysis will be performed., and a blank merge-base SHA1. That points to the SCM diff not resolving the target/base branch correctly, which would explain why the new-code line count looks so small and may be wrong.

To narrow this down:

  • Is this pipeline run configured as a PR build (with PR.TargetBranch etc. set), or is it
    analyzing the branch standalone? The “base branch parameter was not provided” line
    suggests SonarQube didn’t get PR context.
  • Can you confirm the checkout step does a full git fetch (not shallow, and with enough
    history to reach the merge base)? A shallow/partial clone is a common cause of “unchanged lines counted as new.”
  • What is your project’s New Code definition (previous version / number of days /
    reference branch)?

If you can share the full analysis log around the “SCM Publisher” section, we can confirm
whether it’s actually finding the correct changed-lines diff.

Cheers,

Stevan