Inaccurate Code Coverage Reporting in Pull Requests

Prasad_Dodla · January 11, 2024, 1:27pm

Template for a good new topic, formatted with Markdown:

ALM used (Bitbucket Cloud, Azure DevOps)
CI system used (Azure DevOps)
Scanner command used when applicable (Uses Azure DevOps Sonar Cloud plugins)
Languages of the repository (C#, .Net Core, React / Angular, Python)
Error observed (“Code Coverage on New Code not reflecting accurately. Analysis results show metrics for the entire codebase rather than focusing on newly added code.”)
Steps to reproduce (“We operate multiple branches and numerous Pull Requests on Bitbucket, utilizing Sonar Cloud for Static Code Analysis. Our primary concern is that when running code analysis, the tool does not provide code coverage data specifically for newly added code lines. Instead, it aggregates data from the entire codebase.”)
Potential workaround (we kindly request a telephonic or virtual meeting to walk us through the expected behavior of Sonar Cloud in our scenario. Your insights and guidance will be instrumental in helping us configure Sonar Cloud to meet our specific requirements.)

Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!

Colin · January 16, 2024, 10:16am

Hey there.

We won’t be able to hop on a call. This is a community forum.

How about you start by sharing a screenshot demonstrating the issue?

Prasad_Dodla · January 17, 2024, 5:16am

Here is screenshot, for one of the analysis after the PR is created.

Code Coverage on New Code not reflecting accurately. Analysis results show metrics for the entire codebase rather than focusing on newly added code

Colin · January 17, 2024, 3:35pm

In the upper right it says the last analysis had a warning. What is the warning?

Prasad_Dodla · January 18, 2024, 5:26am

Here is screenshot for warning

Colin · January 18, 2024, 8:02am

Well, there we go. SonarCloud has failed to detect SCM data, which is required to correctly detect new code. That’s the problem that needs to get fixed.

What tool are you using to build your code? Bitbucket pipelines? If so, can you share your bitbucket-pipelines.yml configuration?

Prasad_Dodla · January 19, 2024, 5:14am

We are using Azure DevOps classic pipelines

steps:

task: SonarCloudPrepare@1
displayName: ‘Prepare analysis on SonarCloud’
inputs:
SonarCloud: ‘xxxxxx’
organization: ‘xxxx’
projectKey: ‘xxxxxx’
projectName: ‘xxxxxx’
extraProperties: |
extraProperties: |
sonar.exclusions=/obj/,/*.dll,/opencover.xml
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)//coverage.opencover.xml
sonar.cobertura.reportsPath=$(Build.SourcesDirectory)//coverage.cobertura.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/.trx

Colin · January 19, 2024, 1:35pm

There’s not much to glean from there. How about your full Azure DevOps Pipeline YML?

Prasad_Dodla · January 22, 2024, 9:29am

Attaching the full YAML with masked details

pool:
  name: xxxxxx
  demands: java

steps:
- task: UseDotNet@2
  displayName: 'Use .NET Core sdk 5.x'
  inputs:
    version: 5.x

- task: NuGetToolInstaller@1
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: DotNetCoreCLI@2
  displayName: 'dotnet restore'
  inputs:
    command: restore
    projects: 'xxxxx/*.csproj'

- task: DotNetCoreCLI@2
  displayName: 'restore test'
  inputs:
    command: restore
    projects: |
     project1
     project2
     project3
     project4
     project5

- task: SonarCloudPrepare@1
  displayName: 'Prepare analysis on SonarCloud'
  inputs:
    SonarCloud: 'xxxxx'
    organization: 'xxxxx'
    projectKey: 'xxxxx'
    projectName: 'xxxxx'
    extraProperties: |
     extraProperties: |
     sonar.exclusions=**/obj/**,**/*.dll,**/*opencover.xml
     sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
     sonar.cobertura.reportsPath=$(Build.SourcesDirectory)/**/coverage.cobertura.xml
     sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
     

- task: DotNetCoreCLI@2
  displayName: 'dotnet build'
  inputs:
    projects: 'xxxxx/*.csproj'

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: test
    projects: |
     project1
     project2
     project3
     project4
     project5
    arguments: '--configuration $(BuildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --logger trx'

- bash: |
   dotnet tool install dotnet-reportgenerator-globaltool --tool-path .
   ./reportgenerator "-reports:$(Build.SourcesDirectory)/**/coverage.opencover.xml" "-targetdir:coverage/Cobertura" "-reporttypes:Cobertura;HTMLInline;HTMLChart"
  displayName: 'Bash Script'

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: VSTest
    testResultsFiles: '**/*.trx'
    searchFolder: '$(Agent.TempDirectory)'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/Cobertura.xml'
    reportDirectory: Reports

- task: SonarCloudAnalyze@1
  displayName: 'Run Code Analysis'
  inputs:
    jdkversion: 'JAVA_HOME_11_X64'

- task: SonarCloudPublish@1
  displayName: 'Publish Quality Gate Result'

- task: DotNetCoreCLI@2
  displayName: 'dotnet publish'
  inputs:
    command: publish
    publishWebProjects: false
    projects: 'xxxxxxx/*.csproj'
    arguments: '--output $(Build.ArtifactStagingDirectory)/$(Build.BuildId)'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId)'
    ArtifactName: 'xxxxxx_$(Build.BuildId)'

Colin · January 24, 2024, 10:42am

Thanks.

Just to state the obvious – is this project using .git? Is there a .git folder in your repository?

Prasad_Dodla · January 29, 2024, 5:16am

This project uses Git and we have our code stored in Bitbucket and used Azure DevOps for CI/CD
.git folder is present at repository folder level.

Colin · January 31, 2024, 1:06pm

Can you try running an ls -la (for linux) or dir /a:hd (for windows) immediately before the SonarCloudAnalyze task and report the results?

Prasad_Dodla · February 1, 2024, 9:14am

Here is the log
ls-la(linux).logs.txt (3.7 KB)

Prasad_Dodla · February 6, 2024, 12:58pm

Hi @Colin

Can we get an update?

Prasad_Dodla · February 12, 2024, 5:46am

Hi @Colin, We are waiting for your update

Claire_Villard · February 12, 2024, 3:22pm

Hi @Prasad_Dodla

Thank you for your patience. Colin flagged the post for expert attention, and sadly this thread got a bit lost in the process.

Since the .git directory is present on the logs you sent, another frequent cause of this issue is the clone depth.
If the source code is cloned from Git as a Shallow clone (clone depth 0, meaning there is no history) then our engine is not able to properly detect the New Code. It considers the whole codebase as new, leading to what you observe.

Could you check the cloning configuration on your Azure pipeline, and make sure the whole history is cloned?
I’m not an Azure expert at all, I suspect this documentation could be useful: Options for Git repositories - Azure Pipelines | Microsoft Learn
We are conscious it can slow down the clone operation if the repository is huge. To reduce the impact, instead of cloning the whole history and depending on what your development workflow looks like, you can limit it to the latest N commits. N should be big enough to make sure the history contains the commit from which the pull request branch was created on the base branch.

Hope that helps,
Claire

Prasad_Dodla · February 20, 2024, 10:43am

Hi @Claire_Villard,

We are not using shallow clone option in our pipelines. Please refer to the below screenshot
Additionally, our repos are not large and those are medium in size

Claire_Villard · February 26, 2024, 11:34am

Hi @Prasad_Dodla ,

Thanks for getting back to me.
It seems we can’t use the Git data to get the history.

Could you enable the Debug logging on the SonarCloudAnalyze task, and post the complete logs please?

Could you also run a git --version command immediately before the SonarCloudAnalyze task and report the results?

Thanks,
Claire

Prasad_Dodla · March 6, 2024, 5:24am

Hi @Claire_Villard
I have already shared the logs in the earlier comments, could you please refer those

Prasad_Dodla · March 12, 2024, 4:34am

Hello @Claire_Villard
Can we get an update on this?