Situation
We have got a .NET Core 3.0 solution with 4 (business logic) projects and 5 unit testing projects. These test projects each cover one specific part of the code. We use Azure DevOps pipelines to deploy and test the solution and report the code coverage the Sonar Source. For reporting the code coverage we use the OpenCover format. After the recent update on the code coverage (which is much better than just line coverage) our coverage dropped from 90% to about 60% which seems like a lot.
When we took a look at the reported code coverage in Azure DevOps we noticed that it was completely different from the coverage reported in SonarSource. SonarSource seems to report way too many conditions. For instance, a simple if(condition) would result in 10 possible conditions which clearly is incorrect, where Azure DevOps just reports 2 possible conditions.
The problem appears to be that for every test project an OpenCover file is created which shows the coverage for that single test projects. Every OpenCover file contains all statements and their conditions. SonarSource properly merges line coverage, but it appears to sum the amount of conditions. So where the most simple if statement should have 2 conditions, SonarSource actually reported 2 x 5 = 10 conditions (we have got 5 test projects) with only 2 conditions being covered.
Pipeline
We use the settings below to generate the OpenCover files.
SonarCloud prepare task
- task: SonarCloudPrepare@1
inputs:
SonarCloud: 'MySonarCloud'
organization: 'MyOrg'
scannerMode: 'MSBuild'
projectKey: 'SomeProjectKey'
projectName: 'MyProject'
extraProperties: |
sonar.cs.vstest.reportsPaths=$(TestOutputDirectory)/*.trx
sonar.cs.opencover.reportsPaths=$(TestOutputDirectory)/*/coverage.opencover.xml
sonar.coverage.exclusions=**/Migrations/*.cs,**/*Tests*/**/*
sonar.verbose=true
Dotnet Test task
- task: DotNetCoreCLI@2
displayName: 'dotnet test'
inputs:
command: test
publishTestResults: false
projects: '$(project)'
arguments: '--no-restore --no-build --configuration $(BuildConfiguration) --logger trx --collect:"XPlat Code Coverage" --results-directory $(TestOutputDirectory) --settings $(Build.SourcesDirectory)/coverlet.runsettings'
Where the $(project) points to our solution file. We also tried to test each project individually but the result was exactly the same.
We use ReportGenerator to report the code coverage in Azure DevOps.
- task: reportgenerator@4
inputs:
reports: '$(TestOutputDirectory)/*/coverage.opencover.xml'
targetdir: '$(TestOutputDirectory)/mergedcoveragereport'
reporttypes: 'HtmlInline_AzurePipelines;Cobertura'
assemblyfilters: '-*Tests*'
filefilters: '-*/Migrations/*.cs'
Workaround
To fix the problem we change the SonarCloudPrepare task to use the coverageReportPaths instead of OpenCover and we added SonarQube to the reporttypes created by the reportgenerator tasks. Now SonarSource is using the SonarQube.xml file to report the code coverage and we have got it back to about 90%.
\