Inaccurate Code Coverage Reporting in Pull Requests

Template for a good new topic, formatted with Markdown:

  • ALM used (Bitbucket Cloud, Azure DevOps)
  • CI system used (Azure DevOps)
  • Scanner command used when applicable (Uses Azure DevOps Sonar Cloud plugins)
  • Languages of the repository (C#, .Net Core, React / Angular, Python)
  • Error observed (“Code Coverage on New Code not reflecting accurately. Analysis results show metrics for the entire codebase rather than focusing on newly added code.”)
  • Steps to reproduce (“We operate multiple branches and numerous Pull Requests on Bitbucket, utilizing Sonar Cloud for Static Code Analysis. Our primary concern is that when running code analysis, the tool does not provide code coverage data specifically for newly added code lines. Instead, it aggregates data from the entire codebase.”)
  • Potential workaround (we kindly request a telephonic or virtual meeting to walk us through the expected behavior of Sonar Cloud in our scenario. Your insights and guidance will be instrumental in helping us configure Sonar Cloud to meet our specific requirements.)

Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!

Hey there.

We won’t be able to hop on a call. This is a community forum.

How about you start by sharing a screenshot demonstrating the issue?

Here is screenshot, for one of the analysis after the PR is created.

Code Coverage on New Code not reflecting accurately. Analysis results show metrics for the entire codebase rather than focusing on newly added code

In the upper right it says the last analysis had a warning. What is the warning?

Here is screenshot for warning

Well, there we go. SonarCloud has failed to detect SCM data, which is required to correctly detect new code. That’s the problem that needs to get fixed.

What tool are you using to build your code? Bitbucket pipelines? If so, can you share your bitbucket-pipelines.yml configuration?

We are using Azure DevOps classic pipelines

steps:

  • task: SonarCloudPrepare@1
    displayName: ‘Prepare analysis on SonarCloud’
    inputs:
    SonarCloud: ‘xxxxxx’
    organization: ‘xxxx’
    projectKey: ‘xxxxxx’
    projectName: ‘xxxxxx’
    extraProperties: |
    extraProperties: |
    sonar.exclusions=/obj/,/*.dll,/opencover.xml
    sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)//coverage.opencover.xml
    sonar.cobertura.reportsPath=$(Build.SourcesDirectory)/
    /coverage.cobertura.xml
    sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/
    .trx

There’s not much to glean from there. How about your full Azure DevOps Pipeline YML?

Attaching the full YAML with masked details

pool:
  name: xxxxxx
  demands: java

steps:
- task: UseDotNet@2
  displayName: 'Use .NET Core sdk 5.x'
  inputs:
    version: 5.x

- task: NuGetToolInstaller@1
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: DotNetCoreCLI@2
  displayName: 'dotnet restore'
  inputs:
    command: restore
    projects: 'xxxxx/*.csproj'

- task: DotNetCoreCLI@2
  displayName: 'restore test'
  inputs:
    command: restore
    projects: |
     project1
     project2
     project3
     project4
     project5

- task: SonarCloudPrepare@1
  displayName: 'Prepare analysis on SonarCloud'
  inputs:
    SonarCloud: 'xxxxx'
    organization: 'xxxxx'
    projectKey: 'xxxxx'
    projectName: 'xxxxx'
    extraProperties: |
     extraProperties: |
     sonar.exclusions=**/obj/**,**/*.dll,**/*opencover.xml
     sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
     sonar.cobertura.reportsPath=$(Build.SourcesDirectory)/**/coverage.cobertura.xml
     sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
     

- task: DotNetCoreCLI@2
  displayName: 'dotnet build'
  inputs:
    projects: 'xxxxx/*.csproj'

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: test
    projects: |
     project1
     project2
     project3
     project4
     project5
    arguments: '--configuration $(BuildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --logger trx'

- bash: |
   dotnet tool install dotnet-reportgenerator-globaltool --tool-path .
   ./reportgenerator "-reports:$(Build.SourcesDirectory)/**/coverage.opencover.xml" "-targetdir:coverage/Cobertura" "-reporttypes:Cobertura;HTMLInline;HTMLChart"
  displayName: 'Bash Script'

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: VSTest
    testResultsFiles: '**/*.trx'
    searchFolder: '$(Agent.TempDirectory)'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/Cobertura.xml'
    reportDirectory: Reports

- task: SonarCloudAnalyze@1
  displayName: 'Run Code Analysis'
  inputs:
    jdkversion: 'JAVA_HOME_11_X64'

- task: SonarCloudPublish@1
  displayName: 'Publish Quality Gate Result'

- task: DotNetCoreCLI@2
  displayName: 'dotnet publish'
  inputs:
    command: publish
    publishWebProjects: false
    projects: 'xxxxxxx/*.csproj'
    arguments: '--output $(Build.ArtifactStagingDirectory)/$(Build.BuildId)'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId)'
    ArtifactName: 'xxxxxx_$(Build.BuildId)'

Thanks.

Just to state the obvious – is this project using .git? Is there a .git folder in your repository?

This project uses Git and we have our code stored in Bitbucket and used Azure DevOps for CI/CD
.git folder is present at repository folder level.

Can you try running an ls -la (for linux) or dir /a:hd (for windows) immediately before the SonarCloudAnalyze task and report the results?

Here is the log
ls-la(linux).logs.txt (3.7 KB)

Hi @Colin

Can we get an update?

Hi @Colin, We are waiting for your update

Hi @Prasad_Dodla

Thank you for your patience. Colin flagged the post for expert attention, and sadly this thread got a bit lost in the process.

Since the .git directory is present on the logs you sent, another frequent cause of this issue is the clone depth.
If the source code is cloned from Git as a Shallow clone (clone depth 0, meaning there is no history) then our engine is not able to properly detect the New Code. It considers the whole codebase as new, leading to what you observe.

Could you check the cloning configuration on your Azure pipeline, and make sure the whole history is cloned?
I’m not an Azure expert at all, I suspect this documentation could be useful: Options for Git repositories - Azure Pipelines | Microsoft Learn
We are conscious it can slow down the clone operation if the repository is huge. To reduce the impact, instead of cloning the whole history and depending on what your development workflow looks like, you can limit it to the latest N commits. N should be big enough to make sure the history contains the commit from which the pull request branch was created on the base branch.

Hope that helps,
Claire

Hi @Claire_Villard,

We are not using shallow clone option in our pipelines. Please refer to the below screenshot
Additionally, our repos are not large and those are medium in size

image