MSBuild Scanner missing unit tests details for NUnit 3

hobovsky · March 23, 2020, 2:27pm

Hello everybody,

My CI environment is running SonarQube 7.9.1 with sonar-scanner-msbuild-4.6.2.2108-net46 .

I am trying to include code coverage and unit test execution results in my analysis. After a bit of struggle I managed to get reports into it and I get correct numbers:

Please don’t mind the actual numbers. They are… uhmm… subject to change But unit tests details still remain empty. After googling and reading I found hints about sonar.tests et al. not being set, and I tried to set these, but I am probably doing something wrong because I get analysis errors. I run scanner with verbose logging and here are some excerpts from scanner log:

[exec] Starting 'c:\jenkins\workspace\MyPortal_v30_Sonar\tools\sonar-scanner-msbuild-4.6.2.2108-net46\SonarScanner.MSBuild.exe (begin /k:MyPortal_v30 "/n:MyPortal v30" /v:20200314_1951 /d:sonar.sources=c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/MyExchange,c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/MyExchange.Installer,c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/CommonNet,c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/MyAdmin,c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/DesignSpace,c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/Integrations,c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/NMyClient,c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30/Tools /d:sonar.tests=c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30\MyExchangeTest\UnitTests /d:sonar.exclusions=c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30\MyExchangeTest\UnitTests/**/* /d:sonar.tests.inclusions=c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30\MyExchangeTest\UnitTests/**/* /d:sonar.cs.nunit.reportsPaths=c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30\MyExchangeTest\TestResult_nunit3.xml /d:sonar.cs.opencover.reportsPaths=c:\jenkins\workspace\MyPortal_v30_Sonar\coverage_report.xml /d:sonar.verbose=true)' in 'c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30'
[...snip...]
[exec] sonar.sources=c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/MyExchange,c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/MyExchange.Installer,c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/CommonNet,c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/CpAdmin,c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/DesignSpace,c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/Integrations,c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/NMyClient,c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30/Tools
[exec] sonar.tests=c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30\\MyExchangeTest\\UnitTests
[exec] sonar.exclusions=c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30\\MyExchangeTest\\UnitTests/**/*
[exec] sonar.tests.inclusions=c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30\\MyExchangeTest\\UnitTests/**/*
[exec] sonar.cs.nunit.reportsPaths=c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\sw\\pceapp\\my\\MyPortal\\rel\\v30\\MyExchangeTest\\TestResult_nunit3.xml
[exec] sonar.cs.opencover.reportsPaths=c:\\jenkins\\workspace\\MyPortal_v30_Sonar\\coverage_report.xml
[...snip...]
[exec] 20:03:26.993 INFO: Indexing files of module 'UnitTests.Base'
[exec] 20:03:26.993 INFO:   Base dir: C:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30\MyExchangeTest\UnitTests\UnitTests.Base
[exec] 20:03:26.994 DEBUG:   Source paths: Asserts/AuthorizationAssert.cs, [...snip...]
[exec] 20:03:26.994 DEBUG:   Test paths: c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30\MyExchangeTest\UnitTests
[exec] 20:03:26.994 INFO:   Excluded sources: c:\jenkins\workspace\MyPortal_v30_Sonar\sw\pceapp\my\MyPortal\rel\v30\MyExchangeTest\UnitTests/**/*
[exec] 20:03:26.996 DEBUG: Evaluate issue exclusions for 'MyExchangeTest/UnitTests/UnitTests.Base/Asserts/AuthorizationAssert.cs'
[exec] 20:03:26.997 DEBUG: 'MyExchangeTest/UnitTests/UnitTests.Base/Asserts/AuthorizationAssert.cs' generated metadata with charset 'UTF-8'
[exec] 20:03:26.997 DEBUG: 'MyExchangeTest\UnitTests\UnitTests.Base\Asserts\AuthorizationAssert.cs' indexed with language 'cs'
...snip...
[exec] 20:03:27.139 DEBUG: Evaluate issue exclusions for 'MyExchangeTest/UnitTests/UnitTests.Base/Asserts/AuthorizationAssert.cs'
[exec] 20:03:27.139 DEBUG: 'MyExchangeTest/UnitTests/UnitTests.Base/Asserts/AuthorizationAssert.cs' generated metadata as test  with charset 'UTF-8'
[exec] 20:03:27.147 INFO: ------------------------------------------------------------------------
[exec] 20:03:27.147 INFO: EXECUTION FAILURE
[exec] 20:03:27.147 INFO: ------------------------------------------------------------------------
[...snip...]
[exec] 20:03:27.235 ERROR: Error during SonarQube Scanner execution
[exec] 20:03:27.235 ERROR: File MyExchangeTest/UnitTests/UnitTests.Base/Asserts/AuthorizationAssert.cs can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files

Maybe all I need is just another pair of eyes because I cannot see why sources and conditions would not be disjoint, setting source exclusions and test inclusions to the exactly same values. But still, AuthorizationAssert.cs (which a very first source file of the very first test project being analyzed) is indexed twice, both as source and as test.

What am I missing here? What do I need to do to get unit tests result into my analysis details?

Thanks for any help, and best regards!

hobovsky · March 21, 2020, 11:14am

A bit of follow up: when analysis is run with default settings for sources and tests (i.e. none specified, no inclusions, no exclusions), two (and only two) of my tests project seem to be mis-classified and analyzed:

I do not know why yet, and I think they do not stand out from the remaining ones in any way, but going to investigate on it. I think they somehow are “forced” as source project even if I specify inclusions/exclusions with appropriate command line params. And if anyone had any idea why these two are analyzed while others are not, or any idea regarding the original issue, I’d really appreciate.

Thanks!

hobovsky · March 22, 2020, 2:32am

So in the meantime I upgraded MSBuild Scanner to 4.7.1 to see if this would help, but unfortunately it did not. I got categorization of projects sorted out, and as it turns out, one of the categorization features MSBuild scanner uses is presence of “Test Explorer” service tag, i.e. if you ran unit tests via VS in your project, it will be recognized by the scanner as a test project. This behavior is not documented in official documentation, but you can find its description here: https://github.com/SonarSource/sonar-scanner-msbuild/wiki/Analysis-of-product-projects-vs.-test-projects After I set sonar.msbuild.testProjectPattern to appropriate value, I got all my tests projects categorized correctly.

But still, problem with duplicate indexing persists, and for some reason scanner seems to recognize the root, working folder, as a module and drills down its directory structure, effectively attempting to reindex files it had already covered while indexing (sub)modules. I have no idea how to circumvent this behavior and work around it. And I do not even know whether I will get unit test reports after I resolve duplicate indexing issue, or something new will pop out. I have to admit, the amount of yak shaving required to add unit tests execution report to the analysis slowly gets overwhelming for me I might be dense, but the way how sonar.sources, sonar.tests, inclusion and exclusion patterns, sonar.msbuild.testProjectPattern (why is separate mechanism even needed? Would not sonar.tests be enough?) work together is really confusing.

For the time being, I give up and I will probably use some separate dashboard with unit test reports. Thanks everyone for rubber-ducking me through the process, it really helped even if I did not get what I needed

Regards, and good luck!

mickaelcaro · March 23, 2020, 7:58am

Hi @hobovsky and welcome to the community !

Thank you for your insights, that will certainly help us when doing some improvements on the UX of that Scanner.

Just to be super clear but you might have discover that on your own : sonar.sources and sonar.test, in the context of the Scanner for MSBuild, are meant to be filled by it, with automatic detection, so you don’t have to provide values at all for that.

That being said, and as per the wiki page you discover (this one is pretty new BTW, it will be included in the docs very soon), make sure that, by providing corresponding tags and, if needed, projects Guid (are you targeting a .NET core project BTW ? Please note that those one are not yet fully supported, so you have, in some case, to manually put ProjectGuids either in the SLN file if you have one, or in the csproj directly) to make sure that the detection will work properly.

For tests coverage files, you will have to provide manually the nunit property, as the default behavior is to detect and fill only vstest coverage and binary files.

HTH,
Mickaël

hobovsky · March 23, 2020, 4:38pm

Ooookay, so while writing this post it dawned on me: maybe the behavior I get is expected one, the results observed by me are as expected and supported currently, and I am chasing something what is not even there (although I would expect it to be)? If so, dammit, my RTFM-fu is weak…

Hello @mickaelcaro, and thank you very much for your response.

Okay i get it now, and I admit this part exactly was not clear. In posts above I complained about mutual impact of sonar.tests and sonar.msbuild.testProjectPattern and I probably stuck to this because missing sonar.tests is very often attributed as the cause of empty test report and pops up very often in google searches. This lead me to adding sonar.tests to command line, and this lead to problems.

So I removed sources and tests path from my command line, left only report paths and test project pattern, and now my analysis log looks like this:

 [exec] Starting 'c:/CDP/sonar-scanner-msbuild-4.7.1.2311-net46/SonarScanner.MSBuild.exe (begin /k:MyPortal_v23 "/n:MyPortal v23" /v:20200323_1158 /d:sonar.verbose=true /d:sonar.msbuild.testProjectPattern=.*UnitTests\..*\.csproj$ /d:sonar.cs.nunit.reportsPaths=c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23\MyExchangeTest\TestResult.xml /d:sonar.cs.opencover.reportsPaths=c:\jenkins\workspace\MyPortal_v23_Sonar\coverage_report.xml)' in 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23'
 ...snip...
 [exec] sonar.verbose=true was specified - setting the log verbosity to 'Debug'
 [exec] sonar.projectKey=MyPortal_v23
 [exec] sonar.projectName=MyPortal v23
 [exec] sonar.projectVersion=20200323_1158
 [exec] sonar.working.directory=c:\\jenkins\\workspace\\MyPortal_v23_Sonar\\sw\\pceapp\\cpd\\MyPortal\\rel\\v23\\.sonarqube\\out\\.sonar
 [exec] sonar.projectBaseDir=c:\\jenkins\\workspace\\MyPortal_v23_Sonar\\sw\\pceapp\\cpd\\MyPortal\\rel\\v23
 [exec] sonar.sources=\ ...snip...
 [exec] sonar.msbuild.testProjectPattern=.*UnitTests\\..*\\.csproj$
 [exec] sonar.cs.nunit.reportsPaths=c:\\jenkins\\workspace\\MyPortal_v23_Sonar\\sw\\pceapp\\cpd\\MyPortal\\rel\\v23\\MyExchangeTest\\TestResult.xml
 [exec] sonar.cs.opencover.reportsPaths=c:\\jenkins\\workspace\\MyPortal_v23_Sonar\\coverage_report.xml
 [exec] sonar.host.url=http://sonar.mycompany.intra:9000
 [exec] sonar.scm.disabled=true
 [exec] sonar.issue.ignore.allfile=generated,auto
 [exec] sonar.issue.ignore.allfile.generated.fileRegexp=\\/\\/\\s+This file has been generated by
 [exec] sonar.issue.ignore.allfile.auto.fileRegexp=\\/\\/.+auto-generated
 [exec] sonar.issue.enforce.multicriteria=cmdline
 [exec] sonar.issue.enforce.multicriteria.cmdline.ruleKey=csharpsquid:S2228
 [exec] sonar.issue.enforce.multicriteria.cmdline.resourceKey=**/Tools.ArchiveLauncher/**/*.cs
 [exec] sonar.issuesReport.html.enable=true
 [exec] sonar.issuesReport.console.enable=true
 [exec] sonar.visualstudio.enable=false
 [exec] sonar.modules=...bunch of modules...
 ...snip...
 [exec] 12:10:55.543 INFO: Sensor C# [csharp] (done) | time=19763ms
 [exec] 12:10:55.544 INFO: Sensor C# Tests Coverage Report Import [csharp]
 [exec] 12:10:55.545 DEBUG: Analyzing coverage with wildcardPatternFileProvider with base dir 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23\.' and file separator '\'.
 [exec] 12:10:55.546 DEBUG: Pattern matcher extracted prefix/absolute path 'c:\jenkins\workspace\MyPortal_v23_Sonar\coverage_report.xml' from the given pattern 'c:\jenkins\workspace\MyPortal_v23_Sonar\coverage_report.xml'.
 [exec] 12:10:55.546 DEBUG: Pattern matcher returns a single file: 'c:\jenkins\workspace\MyPortal_v23_Sonar\coverage_report.xml'.
 [exec] 12:10:55.547 DEBUG: The current user dir is 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23'.
 [exec] 12:10:55.547 INFO: Parsing the OpenCover report c:\jenkins\workspace\MyPortal_v23_Sonar\coverage_report.xml
 [exec] 12:10:55.645 DEBUG: Skipping the fileId 'xxxxx'... (a bunch of files from outside of analyzed module which happened to end up in coveraage report, nothing suspected here, no product files seem to be reported as skipped)
 ...snip...
 [exec] 12:12:21.187 INFO: Adding this code coverage report to the cache for later reuse: c:\jenkins\workspace\MyPortal_v23_Sonar\coverage_report.xml
 [exec] 12:12:21.195 DEBUG: Analyzing coverage after aggregate found '408' coverage files.
 [exec] 12:12:21.725 DEBUG: The total number of file count statistics is '408'.
 [exec] 12:12:21.726 INFO: Coverage Report Statistics: 408 files, 408 main files, 408 main files with coverage, 0 test files, 0 project excluded files, 0 other language files.
 [exec] 12:12:21.726 INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=86182ms
 [exec] 12:12:21.726 INFO: Sensor C# Unit Test Results Import [csharp]
 [exec] 12:12:21.727 DEBUG: Pattern matcher extracted prefix/absolute path 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23\MyExchangeTest\TestResult.xml' from the given pattern 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23\MyExchangeTest\TestResult.xml'.
 [exec] 12:12:21.727 DEBUG: Pattern matcher returns a single file: 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23\MyExchangeTest\TestResult.xml'.
 [exec] 12:12:21.727 DEBUG: The current user dir is 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23'.
 [exec] 12:12:21.727 INFO: Parsing the NUnit Test Results file 'c:\jenkins\workspace\MyPortal_v23_Sonar\sw\pceapp\cpd\MyPortal\rel\v23\MyExchangeTest\TestResult.xml'.
 [exec] 12:12:21.973 DEBUG: Parsed NUnit test run - total: 5989, totalSkipped: ...snip...
 [exec] 12:12:21.983 INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=257ms

What I get form the log is that reports are discovered, parsed and analyzed successfully, numbers are shown in analysis correctly, but for some reason, report is still empty (see secreenshot in post #1). Also, when report is parsed, no files are reported as missing/ignored, and I checked generated XML for paths and they seem to be OK (dlls and pdbs reside in paths listed in report XML). So I presume I am still missing something. By “nunit property”, did you mean sonar.cs.nunit.reportsPaths?

I don’t think my problem lies with how coverage report is processed, because I think coverage results are OK, as I get all numbers, margin colors, etc.:

My problem is that test results are not presented with breakdown into successful test cases, failed, skipped, which, and why. I get correct numbers in the filter on the left, but all other views are either empty, or seem to miss actual values:

no results:

Long column of unlabelled 0s, no information about test case failure:

Please, tell me I’m doing something wrong and it’s not how it’s currently supported…

Thank you, and best regards

ADDED: I forgot about two things I wanted to mention additionally:

My projects are “regular” .NET Framework projects (not .NET Core), if it’s meaningful in any way
You mentioned that official documentation is going to be updated. If so, you might want to remember updating also docs for SonarQube 7.9 as the latest LTS version, if new information is applicable for it.

Andrei_Epure · March 24, 2020, 8:42am

Hi @hobovsky

For the sonar-dotnet plugins (C# and VB .NET), we don’t push unit test results data to SonarQube. Historically, it has been a missing feature of the plugin and we’ve had issue #886 to track it.

However, recently we decided to close the issue as Won't fix - you can read the motivation in this comment.

Having said that, you can import test data using Generic Test Data and it will work (see DotNet core code coverage).