Code Coverage is not shown in the SonarCloud.io

any update on this, @luru?

Hi Andrei,

Thanks for looking into this.

I’m going to reply for @luru.

As a side note, we have our test running on a docker container, however the path reference seems to be correct and the scan can find the files. After that, it will start ignoring all the files

The exclusion rules are the following, there are no additional exclusions in the Sonarcloud side:

extraProperties: |
sonar.exclusions=/test
sonar.cs.opencover.reportsPaths=
/results/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=**/results/coverage.opencover.xml
sonar.coverage.exclusions="*Tests.cs"

The working dir is this one, it seems to be pointing to but can find files:

020-01-27T17:52:41.6872221Z 17:52:41.686 INFO: Importing 2 Roslyn reports
2020-01-27T17:52:41.8228277Z 17:52:41.822 INFO: Sensor C# [csharp] (done) | time=674ms
2020-01-27T17:52:41.8231835Z 17:52:41.823 INFO: Sensor C# Tests Coverage Report Import [csharp]
2020-01-27T17:52:41.8262059Z 17:52:41.823 DEBUG: Analyzing coverage with wildcardPatternFileProvider with base dir ‘/home/vsts/work/1/.’ and file separator ‘/’.
2020-01-27T17:52:41.8271918Z 17:52:41.826 DEBUG: Pattern matcher extracted prefix/absolute path ‘/home/vsts/work/1/.’ from the given pattern ‘/results/coverage.opencover.xml’.
2020-01-27T17:52:41.8277597Z 17:52:41.827 DEBUG: Gathering files for wildcardPattern '
/results/coverage.opencover.xml’.
2020-01-27T17:52:41.8916016Z 17:52:41.891 DEBUG: Pattern matcher returns ‘1’ files.
2020-01-27T17:52:41.8920706Z 17:52:41.891 DEBUG: The current user dir is ‘/home/vsts/work/1’.
2020-01-27T17:52:41.8925099Z 17:52:41.892 INFO: Parsing the OpenCover report /home/vsts/work/1/./s/{Redacted}/results/coverage.opencover.xml

Then it ignores them:

DEBUG: Skipping the fileId ‘2’, line ‘31’, vc ‘1’ because file ‘/home/vsts/work/1/{Redacted}/System/AnalyticsService.cs’ is not indexed or does not have the supported language.
DEBUG: Skipping the fileId ‘2’, line ‘32’, vc ‘1’ because file ‘/home/vsts/work/1/{Redacted}/System/AnalyticsService.cs’ is not indexed or does not have the supported language.
DEBUG: Skipping the fileId ‘2’, line ‘33’, vc ‘21’ because file ‘/home/vsts/work/1/{Redacted}/System/AnalyticsService.cs’ is not indexed or does not have the supported language.
DEBUG: Skipping the fileId ‘2’, line ‘34’, vc ‘9’ because file ‘/home/vsts/work/1/{Redacted}/System/AnalyticsService.cs’ is not indexed or does not have the supported language.

However, we don’t have any further ignore rules in the project configuration.
Attached the Code Analysis Logs with the verbose command enabled as requested.

TaSonarcloud_Scan_Verbose.txt|attachment (857.4 KB)

Hi,

I was wondering if your able to see the logs attached. Re-uploaded them, just in case.
Anyone to take a look at them?

Sonarcloud_Scan_Verbose.txt (857.4 KB)

Thank you @sconde for the detailed log file, I managed to find the problem using it.

If you search in the log file for .sonar.sources=\ , you will see that the given source files are in the /home/vsts/work/1/s/Redacted/src/Redacted folder. Notice the s folder inside work/1/s/Redacted.

The relevant lines for “indexed” files (the files considered by the scanner):

2020-01-27T17:52:26.7468107Z Using longest common projects root path as project base directory: '/home/vsts/work/1/s/Redacted'.
2020-01-27T17:52:26.7734906Z B2F834B9-7FC5-4650-A5B4-68D92CA44510.sonar.sources=\
2020-01-27T17:52:26.7756371Z "/home/vsts/work/1/s/Redacted/src/Redacted/DataIngestion/PipelineCreation/AzureSearch/PipelineManager.cs",\

However, the code coverage report contains file paths leading to a different directory: /home/vsts/work/1/Redacted/ (notice that the s folder is missing there):

2020-01-27T17:52:42.5619723Z 17:52:42.507 DEBUG: Skipping the fileId '65', line '128', vc '0' because file '/home/vsts/work/1/Redacted/src/Redacted/DataIngestion/PipelineCreation/AzureSearch/PipelineManager.cs' is not indexed or does not have the supported language.

What’s interesting is that the coverage file itself is in the same root folder as the source files /home/vsts/work/1/./s/Redacted/results/coverage.opencover.xml (notice the s folder in the path):

2020-01-27T17:52:41.8925099Z 17:52:41.892 INFO: Parsing the OpenCover report /home/vsts/work/1/./s/Redacted/results/coverage.opencover.xml

Bottom line, when you generate the code coverage, the paths are different.

The following file is known to the scanner:
/home/vsts/work/1/s/Redacted/src/Redacted/<FOO>/PipelineManager.cs

The following file is present in the coverage report, but unknown to the scanner:
/home/vsts/work/1/Redacted/src/Redacted/<FOO>/PipelineManager.cs

I suggest you check how you generate the code coverage report, in what folder the command is being run.

What are the commands you are running?

Hi Andrei,

Thanks for taking a look at this.

That was my initial guess, I seem to believe it comes from here:

 script: |
          mkdir ${{ parameters.workingDir }}/results
          sed '**s|/src/src|**${{ parameters.workingDir }}/src|g' ${{ parameters.workingDir }}/builds/.test-results/coverage.opencover.xml > ${{ parameters.workingDir }}/results/coverage.opencover.xml
          sed '**s|/src/src**|${{ parameters.workingDir }}/src|g' ${{ parameters.workingDir }}/builds/.test-results/coverage.cobertura.xml > ${{ parameters.workingDir }}/results/coverage.cobertura.xml
          dotnet tool install dotnet-reportgenerator-globaltool
          ./reportgenerator "-reports:${{ parameters.workingDir }}/results/coverage.cobertura.xml" "-targetdir:${{ parameters.workingDir }}/results" "-reporttypes:Cobertura;HTMLInline;HTMLChart;HTMLSummary"
          cat ${{ parameters.workingDir }}/results/coverage.opencover.xml
        displayName: 'Coverage'

/home/vsts/work/1/s/Redacted/src/Redacted//PipelineManager.cs is the build dir

So, it may seem the results are actually not being moved across? However from the logs, it indicates the base dir with the /s/ but I don’t seem to find why it then recurrs to try to find the COverage reports in the other dir without the /s/

INFO:   Base dir: /home/vsts/work/1/s/redacted/src/redacted
DEBUG:   Source paths: Common/APIConstants.cs, Common/CoreParameterFactory.cs, Common/CorePipelinerParameter.cs, Common/ICoreParametersFactory.cs, DataIngestion/AssetNameService.cs, DataIngestion/AssetPathInformation.cs, DataIngestion/AssetProcessInformation.cs, DataIngestion/AssetProcessService.cs, DataIngestion/BaseQueryToEntityConverter.cs, DataIngestion/BlobUploadService.cs, DataIngestion/DataFactoryService.cs, DataIngestion/DataIngestionRetryImportService.cs, DataIngestion/DataIngestionService.cs, DataIngestion/DataTypeInferenceService.cs, DataIngestion/DB2QueryToEntityConverter.cs, DataIngestion/Factories/BlobUploadNameFactory.cs, DataIngestion/Factories/QueryToEntityConverterFactory.cs, DataIngestion/IAssetNameService.cs, DataIngestion/IAssetProcessService.cs, DataIngestion/IBlobUploadService.cs, DataIngestion/IDataFactoryService.cs, DataIngestion/IDataIngestionRetryImportService.cs, DataIngestion/IDataIngestionService.cs, DataIngestion/IDataTypeInferenceService.cs, DataIngestion/IMetadataService.cs, DataIngestion/IPipelineManager.cs, DataIngestion/IPipelineRunnerService.cs, DataIngestion/IQueryToEntityConverter.cs, DataIngestion/IQueryToEntityConverterFactory.cs, DataIngestion/IStorageService.cs, DataIngestion/MetadataService.cs, DataIngestion/Model/AssetFromLanding.cs, DataIngestion/Model/DataTypeInferenceRequest.cs, DataIngestion/Model/DataTypeInferenceResponse.cs, DataIngestion/Model/FileNameSuffixStyle.cs, DataIngestion/Model/SqlDataTypeDto.cs, DataIngestion/PipelineCreation/AzureSearch/DatasetManager.cs, DataIngestion/PipelineCreation/AzureSearch/IAzureSearchHelperExtensions.cs, DataIngestion/PipelineCreation/AzureSearch/IAzureSearchPipelineManager.cs, DataIngestion/PipelineCreation/AzureSearch/PipelineCreator.cs, DataIngestion/PipelineCreation/AzureSearch/PipelineManager.cs, DataIngestion/PipelineCreation/Constants.cs, DataIngestion/PipelineCreation/IDataSetManager.cs, DataIngestion/PipelineCreation/IPipelineComposer.cs, DataIngestion/PipelineCreation/IPipelineCreator.cs, DataIngestion/PipelineCreation/ITemplateResolverService.cs, DataIngestion/PipelineCreation/Model/ActivityOrder.cs, DataIngestion/PipelineCreation/Model/DatasetTemplateJson.cs, DataIngestion/PipelineCreation/PipelineComposer.cs, DataIngestion/PipelineCreation/TemplateResolverService.cs, DataIngestion/PipelineRunnerService.cs, DataIngestion/StorageService.cs, DataIngestion/TSQLQueryToEntityConverter.cs, DataIngestion/UploadedAssetInformation.cs, External/AzureSearchService.cs, External/GraphAPIAuthenticationProvider.cs, External/GraphAPIClient.cs, External/IAzureSearchService.cs, External/IdentityServerService.cs, External/IGraphAPIClient.cs, External/IIdentityServerService.cs, External/IndexerStatusDto.cs, External/KnowledgeStore.cs, External/KnowledgeStoreDto.cs, ModelServing/IModelService.cs, ModelServing/Models/ComputeTypeEnum.cs, ModelServing/Models/ImageCreation.cs, ModelServing/Models/ModelDeployment.cs, ModelServing/Models/ModelExecution.cs, ModelServing/ModelService.cs, Notifications/IPipelineNotificationsService.cs, Notifications/PipelineNotificationsService.cs, Query/DatabricksJobRun.cs, Query/DatabricksService.cs, Query/IDatabricksService.cs, Query/IQueryComposer.cs, Query/IQueryService.cs, Query/Model/ExecuteQueryParameter.cs, Query/Model/ExecuteQueryParameterByEntity.cs, Query/Model/ExecuteQueryParameterByEntityDefault.cs, Query/Model/ExecuteQueryParameterByEntityPrefiltered.cs, Query/Model/ExecuteQueryParameterByTable.cs, Query/Model/GetQueryResultParameter.cs, Query/Model/IQueryResultResponse.cs, Query/Model/QueryDefinition.cs, Query/Model/QueryDefinitionDefault.cs, Query/Model/QueryDefinitionPrefiltered.cs, Query/Model/QueryDefinitionTable.cs, Query/Model/QueryExecutionLifecycleState.cs, Query/Model/QueryExecutionResultState.cs, Query/Model/QueryExecutionStatus.cs, Query/Model/QueryOutputFormat.cs, Query/Model/QueryResultSasResponse.cs, Query/Model/QueryResultStreamResponse.cs, Query/QueryComposer.cs, Query/QueryService.cs, Security/IPermissionService.cs, Security/PermissionInformation.cs, Security/PermissionService.cs, System/AnalyticsService.cs, System/ClusterService.cs, System/DataBricksClusterState.cs, System/DataLakeService.cs, System/HDInsightClusterState.cs, System/IAnalyticsService.cs, System/IClusterService.cs, System/IDataLakeService.cs, System/ILogService.cs, System/IModuleService.cs, System/LogService.cs, System/ModuleService.cs

17:52:36.767 INFO:   Excluded sources: **/*test*
17:52:36.768 INFO:   Excluded sources for coverage: **Tests*.cs
17:52:36.779 DEBUG: 'src/redacted/Common/APIConstants.cs' generated metadata with charset 'UTF-8'
17:52:36.782 DEBUG: 'src/redacted/Common/APIConstants.cs' indexed with language 'cs'
17:52:36.786 DEBUG: 'src/redacted/Common/CoreParameterFactory.cs' generated metadata with charset 'UTF-8'
17:52:36.787 DEBUG: 'src/redacted/Common/CoreParameterFactory.cs' indexed with language 'cs'

And then it seems to find one file (I guess is the empty coverage report?) but should find the others from the wildcard config.

DEBUG: Gathering files for wildcardPattern ‘**/results/coverage.opencover.xml’.
Pattern matcher returns ‘1’ files.
The current user dir is ‘/home/vsts/work/1’.
Parsing the Visual Studio Test Results file ‘/home/vsts/work/1/./s/redacted/results/coverage.opencover.xml’.

Also, it seems to be having issues importing the unit test reports.
WARN: Could not import unit test report ‘./s/Redacted/results/coverage.opencover.xml’

The code coverage results are being found and parsed. For example:

DEBUG: Skipping the fileId '65', line '128', vc '0' because file '/home/vsts/work/1/Redacted/src/Redacted/DataIngestion/PipelineCreation/AzureSearch/PipelineManager.cs' is not indexed or does not have the supported language.

FileId 65, line 128 and the file path is information from the coverage.opencover.xml which gets parsed and processed by the SonarC# plugin.

However, the coverage.opencover.xml contains an invalid path: /home/vsts/work/1/Redacted/src/Redacted/DataIngestion/PipelineCreation/AzureSearch/PipelineManager.cs. And this invalid path gets created by the code coverage tool. For the code coverage import to work, you need to run the code coverage tool in a way to generate the proper path to the file on disk.

Are you having the project sources in both /home/vsts/work/1/Redacted and /home/vsts/work/1/s/Redacted folders?

As I said, the coverage file is not empty.

That’s because you are passing the coverage file instead of the unit test report. For unit test results, we support VSTest, NUnit and xUnit. See our guide [Coverage & Test Data] Generate Reports for C#, VB.net.

Hi Andrei,

Thank you for your answer. Even when changing all the references to move the reports to /home/vsts/work/1/ to have everything in the same directory, it’s still not indexing the report.

DEBUG: Analyzing coverage with wildcardPatternFileProvider with base dir ‘/home/vsts/work/1/.’ and file separator ‘/’.
DEBUG: Pattern matcher extracted prefix/absolute path ‘/home/vsts/work/1/results/coverage.opencover.xml’ from the given pattern ‘/home/vsts/work/1/results/coverage.opencover.xml’.
DEBUG: Pattern matcher returns a single file: ‘/home/vsts/work/1/results/coverage.opencover.xml’.
DEBUG: The current user dir is ‘/home/vsts/work/1’.
INFO: Parsing the OpenCover report /home/vsts/work/1/results/coverage.opencover.xml
DEBUG: Skipping the fileId ‘1’, line ‘18’, vc ‘0’ because file ‘/home/vsts/work/1/redacted/src/redacted.API/DataIngestion/PipelineCreation/AzureSearch/IAzureSearchHelperExtensions.cs’ is not indexed or does not have the supported language.

Find attached the verbose logs. Could you please take a look?

I’ve tried changing the ProjectBaseDir as well, but it errors and the rest seem to be working as well.

SonarAnalyze_verboselogs20200309.txt (868.6 KB)

hi @sconde


Files indexed by the sonar scanner

I see

2020-03-09T13:51:36.7619387Z sonar.projectBaseDir=/home/vsts/work/1/s/redacted

and

2020-03-09T13:51:36.7666720Z B2F834B9-7FC5-4650-A5B4-68D92CA44510.sonar.sources=\
2020-03-09T13:51:36.7667228Z "/home/vsts/work/1/s/redacted/src/redacted.API/Common/APIConstants.cs",\

So the indexed files are in /home/vsts/work/1/s/redacted/src/.

2020-03-09T13:52:08.7542085Z 13:52:08.753 DEBUG: 'src/redacted.API/DataIngestion/PipelineCreation/AzureSearch/IAzureSearchHelperExtensions.cs' indexed with language 'cs'

The above src folder is inside /home/vsts/work/1/s/redacted .


File paths inside the report generated by your code coverage tool

Inside the OpenCover report there are different paths:

2020-03-09T13:52:12.2126721Z 13:52:12.212 DEBUG: Skipping the fileId '1', line '18', vc '0' because file '/home/vsts/work/1/redacted/src/redacted.API/DataIngestion/PipelineCreation/AzureSearch/IAzureSearchHelperExtensions.cs' is not indexed or does not have the supported language.

As you see, the opencover report contains paths inside the folder /home/vsts/work/1/redacted/src/ , without the s inside (as oposed to the sources).

As far as I can see, it’s exactly the same situation as before . Given the indexed paths are inside /home/vsts/work/1/s/redacted/src/ and the coverage report file contains paths in '/home/vsts/work/1/redacted/src/ - these are two different folders, so the files from the coverage report are not accepted.


Let’s try and find the problem

  • Can you collect the coverage report before you do the sed operations?
  • Does the coverage report before you do the sed operations contain the path with s inside?
  • Can you run the operations on your local machine and debug locally the problem?

The 2 possible problems I see:

  • EITHER with your scripts, you move files around during the build process. You run the coverage tool in one folder, then copy the results to the build folder, and because of that the paths are incorrect. You should do all the operations in the same folder, without moving files around.
  • OR you run the coverage tool in the same folder where you do the build, but for some reason you modify the paths inside the opencover resulted report and make them incompatible. You should make sure the paths in the opencover report file are consistent with the paths on disk where you do the sonarscanner analysis.

Hi Andrei,

I agree, however I’m struggling to find the path definition for all of them to merge them into the same folder. If we go through the configuration (find attached) CI.Templates.txt (4.7 KB)

To give you context, in Azure DevOps the variables equal to:
I changed it to $(Agent.BuildDirectory) which is /home/vsts/work/1/ to see that fixed things with the path in the reports. Main path seems to be /home/vsts/work/1/s/ for everything.
${{ parameters.workingDir }} is Redacted/src

The process that follows is, first the Sonarcloud prepare 1.Sonarcloud_Prepare.txt (28.4 KB)

Then, it runs the tests in a Docker container, which are where in calculates the coverage result 2.Tests.txt (34.0 KB) :

Calculating coverage result…
Generating report ‘/var/temp/coverage.opencover.xml’
Generating report ‘/var/temp/coverage.cobertura.xml’

In the container configuration the reports are generated like this:

-p:ParallelizeTestCollections=false --logger trx --results-directory /var/temp -p:CollectCoverage=true -p:CoverletOutputFormat="opencover,cobertura" -p:CoverletOutput=/var/temp/ -p:Exclude="[xunit.]"

Then the build, which doesn’t show much 3. Dotnet build.txt (84.4 KB)

After that, it gets moved from /var/temp to what is associated to ${{ parameters.workingDir }}.
Then it gets moved to $(Agent.BuildDirectory)/results and the Report Generator gets run from there. 4. Report Generator.txt (33.7 KB)

So, in answer to your questions:

  • Can you collect the coverage report before you do the sed operations?

Can’t from the agent. I’ll try to see if the containers leave any log or can be configure to remain active after the tests.

  • Does the coverage report before you do the sed operations contain the path with s inside?

Will need to confirm from the container, but for what I can gather in the logs, they do indeed. However, I can’t crack from where is getting the path without the ‘s’ inside during the report.

  • Can you run the operations on your local machine and debug locally the problem?

I’ll see if I can set up a lab, but with the container configuration can prove rather difficult. I’ll post an update.

Thanks again for dedicating time into this.

1 Like

Indeed, I see

2020-03-09T13:47:56.1902052Z ##[debug]workingDirectory=/home/vsts/work/1/s/...

So I imagine the paths in the report created in the container should be the correct ones.

Can you just output the full file in the logs when inside the container, and then output it after it gets transferred and processed (to narrow down the problem)?

Hi Andrei,

I hope you and the team are doing well with all this madness.

The verbose logs inside the container are in the file called “2.Tests.txt”, I’ll attach them again for you.
2.Tests.txt (34.0 KB)

I’ve managed to pull the report before the sed operation and seem very generic using the working directory as source, find attached.
open.coverage.pre-sedoperation.sample.txt (1.2 MB)

From this point on I’ll be concentrating in a new set of builds using Azure Containers that seem to be more forgiving.

Kindest regards,
Sara

hi @sconde

I am well, thank you. Staying home, like most of us, I guess. I hope you and your team are well, too.

I had another look at the sed operation you mentioned some time ago. I now understand that, somehow, the container where you execute the code has some sort of “chroot jail”, so you need the sed operation to make the paths the same as on the main environment.

It seems that from the sed operation, you are missing the s directory:

sed '**s|/src/src|**${{ parameters.workingDir }}/src|g'

And I guess that parameters.workingDir is different than the buildDir - it does not contain the s folder.

Hi @Andrei_Epure

Yep, we’re all good. Stay safe!

That’s correct, the containers isolate the tests as they perform quite a few operations

So ${{ parameters.workingDir }} = Redacted
Comparing both logs, the only thing that the sed operation seems to be doing is modifying the subdirectory where the solution is; adding an extra /src after Redacted, replacing /src/src/Redacted.API to Redacted/src/Redacted.API/src.

The other paths are added by the Azure agent, so it will look like buildDir + ${{ parameters.workingDir }}, in this case /home/vsts/work/1/ + Redacted/src/Redacted.API/src/Logservice.cs for example.
given that:
buildDir=/home/vsts/work/1/ (I changed to this one recently)
workingDirectory=/home/vsts/work/1/s (originally configured like that)

Neither seem to work.

What I haven’t tried yet is to sed all the paths to make sure are the same.
I’ll let you know how that turns out!

Kindest regards,
Sara

1 Like

Hi Andrei,

I hope everything is fine over there, and you’re still happy and safe.

We shifted the approach to use Azure Containers in our builds instead of Docker. The change was decided for many reasons, not just because of this particular issue.

The great news is that it doesn’t print any foreign paths into the test reports, so we managed to make it work and we can finally see Coverage Reports in Sonarcloud (yay!).

Came across that coverage is down to a bit over 60%, I’m unsure if this figure is realistic or is under-reporting somewhere. Is there a way to figure this out other than comparing the coverage from VS Tools and the resultant one, which rarely matches?

In your experience, can you think of any case where Sonar could be under-reporting for coverage? Like missing exclusions and the sort. Not assuming this is the case, but want to discard everything.

We have the following switches in ignore:

sonar.exclusions=**/test
sonar.coverage.exclusions=“*Tests.cs”

And in DotNet Test

“/p:Exclude="[xunit.]"”

Thanks loads for all your help in advance.

Cheers,
Sara

hi Sara

We’re good and I hope you are good as well!

Great to hear you finally got this working.

Yes, we recently added support for branch coverage and there is an open bug (Fix Code coverage import from multiple test projects · Issue #3296 · SonarSource/sonar-dotnet · GitHub) for the case where there are multiple test projects. So if you have multiple tests projects, this bug manifests itself and you’ll see lower coverage than expected. We plan to fix it and do a new release in the following weeks.

Another possible problem may come from assembly shadowing , see this comment.

And if you want to further discuss the code coverage discrepancy problems, please open a new topic (as this one about “code coverage is now shown” has now a solution) :slight_smile:

Thanks!
Andrei

That’s fantastic. Thanks for your support!

1 Like

hey Sara,

I hope you are well!

just as an FYI, we’re released sonar-csharp 8.6.1 , which reverts the changes we’ve made for VS coverage. It will get deployed to :sonarcloud: in the following days. You should expect the code coverage numbers to increase after that gets deployed (https://sonarcloud.io/api/plugins/installed lists the version of deployed plugins).

A post was split to a new topic: Code coverage broken on SonarCloud for .NET projects