Problems with test execution and coverage upload

SonarQube Server / Community Build
,
1

Greetings

We have recently started using the sonar scanner to get some info on our project codebase. The particular project is a .NET Core 3.1 web app using React.js as an SPA frontend.

We run unit tests with NUnit on the .net side and Jest for the react/typescript side.

I have been fighting with the configuration now for some time to get our test execution and coverage data on the server but haven’t had luck yet.

For out specific sonar server, we have a developer edition that is Version 8.1 (build 31237). I have followed the following user guide:
https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/

the project structure is as follows:

root
|-client-app
   |-src  //client side react code
   |-tests // client unit tests via Jest
|-src // backend code in C#
|-tests // nunit test project with unit tests

I have used the command dotnet tool install --global dotnet-sonarscanner to make the scanner available to me. Next I have used the following commands in the root directory to generate an analysis:

dotnet sonarscanner begin `
		/k:project-key `
		/d:sonar.branch.name=tests `
		/d:sonar.modules=frontend,backend `
		/d:frontend.sonar.sources=client-app/src `
		/d:frontend.sonar.testExecutionReportPaths=test-results/jest-report.xml `
		/d:frontend.sonar.javascript.lcov.reportPaths=test-results/jest-coverage/lcov.info `
		/d:frontend.sonar.tests=client-app/tests `
		/d:frontend.sonar.test.inclusions=*.test.tsx `
		/d:frontend.sonar.inclusions=*.tsx `
		/d:frontend.sonar.sourceEncoding=utf-8 `
		/d:backend.sonar.cs.nunit.reportsPaths=test-results/dotnet-report.xml `
		/d:backend.sonar.cs.opencover.reportsPaths=test-results/dotnet-coverage/coverage.xml `
		/d:backend.sonar.tests=tests `
		/d:backend.sonar.test.inclusions=*Tests.cs `
		/d:backend.sonar.sources=src `
		/d:backend.sonar.inclusions=*.cs `
		/d:backend.sonar.sourceEncoding=utf-8

	npm test --prefix ./client-app/

	dotnet test `
		/p:AltCover=true `
		/p:AltCoverXmlReport=../test-results/dotnet-coverage/coverage.xml `
		--test-adapter-path:. `
		--logger:"nunit;LogFilePath=../test-results/dotnet-report.xml"
	dotnet sonarscanner end

The test results get generated by several tools:

  • for the frontend, Jest generates an execution report with the jest-sonar-reporter npm package and generates a jest-report.xml as an execution report, and an lcov report on coverage
  • for the backend, nunit uses the NunitXml.TestLogger library and generates coverage.xml and dotnet-report.xml

the resulting structure of reports is this:

root
|-test-results
| |-dotnet-coverage
| | |-coverage.xml
| |-jest-coverage
|   |-lcov-report/
|   |-lcov.info
|-dotnet-report.xml
|-jest-report.xml

Looking at the configuration I have setup, all of the information should be uploaded correctly but unfortunately isn’t. I was hoping someone would point me to a mistake I have made in configuration. If not, I am able to provide outputs generated by the commands executed in order to help detect what the problem is.

3

hi @dkralj , welcome to our community!

  • are you running all commands from the same folder?
  • what is the version of the sonarscanner you are using?
  • could you please give us the verbose output of the commands (please add /d:sonar.verbose=true to the begin step, and please attach the output of END step
  • what are the paths on disk of the coverage files?

You may be interested in this related topic: Javascript Code Coverage not working when using MSBUILD Scaner

4

Greetings @Andrei_Epure!

I have a couple of answers to your questions and also some additional info about my progress on this issue.

For the questions you posted:

  • All of the commands are run from the same directory (the sequence of dotnet sonarscanner begin, npm test, dotnet test and dotnet sonarscanner end)
  • Version => INFO: SonarQube Scanner 4.2.0.1873
  • Here is the verbose output => Verbose Output (341.8 KB)
  • the test-results\ directory that has the execution and coverage files is in the same root as the project

In terms of the new information I have gathered:

I have been playing around with the parameters and looking at the information generated in the .sonarqube\out\sonar-project.properties file. I got to upload the execution and coverage in very specific scenarios, and not all of the data but rather either the js test executions and coverage or the .net execution and coverage. Getting these results together is still a problem.

If I for example have these parameters in my start command:

dotnet sonarscanner begin `
	/k:project-name `
	/d:sonar.branch.name=ui-tests `
	/d:sonar.cs.nunit.reportsPaths=test-results/dotnet-report.xml `
	/d:sonar.cs.opencover.reportsPaths=test-results/dotnet-coverage/coverage.xml

my coverage and execution info for the nunit tests shows up in the scan report.

If I however run the following command:

dotnet sonarscanner begin `
	/k:project-name `
	/d:sonar.branch.name=ui-tests `
	/d:sonar.testExecutionReportPaths=test-results/jest-report.xml `
	/d:sonar.javascript.lcov.reportPaths=test-results/jest-coverage/lcov.info `
	/d:sonar.tests=client-app/tests `

I at first get this error:

... //removed for brevity
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 11.532s
INFO: Final Memory: 23M/396M
ERROR: Error during SonarQube Scanner execution
ERROR: File client-app/tests/components/loader/Loader.test.tsx can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files
ERROR:
INFO: ------------------------------------------------------------------------
The SonarQube Scanner did not complete successfully
13:30:10.563  Post-processing failed. Exit code: 1

I can however fix this by editing the file .sonarqube\out\0\FilesToAnalyze.txt and removing all of the files listed in the sources list that was auto-generated but is actually in the client-app\tests\ directory.

after this edit, i can run dotnet sonarscanner end again in the work directory and it will this time upload the execution and coverage of the frontend tests.

5

Hello again. Wanted to give an update on how this is going.

I have tried to fiddle with the files generated by the dotnet sonarscanner tool. I have invoked the start command as such:

dotnet sonarscanner begin `
	/k:project-name `
	/d:sonar.branch.name=ui-tests `
	/d:sonar.cs.nunit.reportsPaths=test-results/dotnet-report.xml `
	/d:sonar.cs.opencover.reportsPaths=test-results/dotnet-coverage/coverage.xml `
	/d:sonar.testExecutionReportPaths=test-results/jest-report.xml `
	/d:sonar.javascript.lcov.reportPaths=test-results/jest-coverage/lcov.info `
	/d:sonar.tests=client-app/tests,tests

next I ran the dotnet test and npm test commands that generate the execution and coverage reports.

After that, I navigated to the .sonarqube\out\0\FilesToAnalyze.txt file and removed all the test files from the client frontend from the list (solves the error that I cannot index files multiple times, since they are listed as both sources and test files).

Lastly I ran the dotnet sonarscanner end command. An interesting part of the output follows:

... redacted
INFO: Analysing [C:\git\project-name\test-results\jest-coverage\lcov.info]
... redacted
INFO: Sensor TypeScript analysis [javascript] (done) | time=14870ms
INFO: Sensor Generic Test Executions Report
INFO: Parsing C:\git\project-name\test-results\jest-report.xml
INFO: Imported test execution data for 2 files
INFO: Sensor Generic Test Executions Report (done) | time=16ms
INFO: Sensor C# [csharp]
INFO: Importing results from 5 proto files in 'C:\git\project-name\.sonarqube\out\0\output-cs'
INFO: Importing 1 Roslyn report
INFO: Sensor C# [csharp] (done) | time=593ms
INFO: Sensor C# Tests Coverage Report Import [csharp]
INFO: Parsing the OpenCover report C:\git\project-name\.\test-results\dotnet-coverage\coverage.xml
INFO: Adding this code coverage report to the cache for later reuse: C:\git\project-name\.\test-results\dotnet-coverage\coverage.xml
INFO: Coverage Report Statistics: 38 files, 34 main files, 34 main files with coverage, 4 test files, 0 project excluded files, 0 other language files.
INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=242ms
INFO: Sensor C# Unit Test Results Import [csharp]
INFO: Parsing the NUnit Test Results file 'C:\git\project-name\.\test-results\dotnet-report.xml'.
INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=8ms

So the scanner reports it has processed both the frontend and backend execution and coverage data. Problem is, when I look at the finished analysis on the portal, it only lists frontend test results.

All of the paths are correct, all of the files are present. when replicating the 2 methods mentioned in my previous post, I can get test results in isolation for backend and frontend, but I was under the impression that I can import both results within one project.

Right now it seems my only course of action is to do separate scans and have separate projects on the sonar server. Is this the intended way of use or am I missing something? The post linked previously suggests that this might be my only recourse.

7

hi @dkralj

I believe what you are hitting here is the lack of support for “mono (multiple technologies) repos”. We have it on the SonarQube Roadmap for the next LTS.

Yes. This is what we do for our sonar-dotnet project, we have 2 projects