Unable to upload the Generic Test Execution XML file to SonarQube server

Description

We have a solution that is written in C++ language and uses Catch2 as the testing framework. Using the functionality mentioned in this link: Catch2/ci-and-misc.md at devel · catchorg/Catch2 · GitHub, we were able to create an XML file that follows the Generic Test Execution format. Then using the sonar.testExecutionReportPaths argument in the SonarScanner.MSBuild.exe begin command, I tried to upload the generated XML to our SonarQube server but it doesn’t show anything. There is no error or success message in the logs.

Repro steps

  1. Generate the generic test execution XML file using the following command provided by the catch2 framework, in the Post Build event of the solution.:
    NativeTests.exe -r sonarqube -o NativeTests.xml
  2. Pass the location of the file in SonarScanner.MSBuild.exe begin command as follows:
    C:\SonarQube\bin\SonarScanner.MSBuild.exe begin /k:NativeTests.sln /n:NativeTests.sln /v:20.1.0 /d:sonar.language=c++ /d:sonar.sources=aa /d:sonar.testExecutionReportPaths=C:\Temp\NativeTests.xml /d:sonar.verbose=true
  3. Perform the build and run the SonarScanner.MSBuild.exe end command as follows:
    C:\SonarQube\bin\SonarScanner.MSBuild.exe end

Expected behavior

I expected that the test cases present in the XML get imported to our local SonarQube Server.

Actual behavior

SonarScanner.MSBuild.exe fails after it starts to parse the XML file. There is no other error shown in the logs. Following is shown in the log file:

09:13:19.527 INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
09:13:19.527 INFO: Sensor Generic Test Executions Report
09:13:19.528 INFO: Parsing C:\Temp\NativeTests.xml
09:13:21.599 INFO: ------------------------------------------------------------------------
09:13:21.599 INFO: EXECUTION FAILURE
09:13:21.599 INFO: ------------------------------------------------------------------------
09:13:21.599 INFO: Total time: 7:00.190s
09:13:21.872 INFO: Final Memory: 30M/1762M
09:13:21.872 INFO: ------------------------------------------------------------------------
Process returned exit code 1

I also tried to navigate the code present in sonarqube/GenericTestExecutionSensor.java at master · SonarSource/sonarqube · GitHub and sonarqube/GenericTestExecutionReportParser.java at 88131a33e07fe611f32883079614d1e768f60076 · SonarSource/sonarqube · GitHub but couldn’t find anything that would tell me if there is any issue with the XML that I am providing in SonarScanner begin command.

Known workarounds

None at the moment

Related information

  • SonarScanner for MSBuild version: 4.9.0.17385
  • SonarQube server version: Community Edition - Version 7.9.1 (build 27448)
  • Languages in your solution: C++

Unfortunately, due to company policies, I won’t be able to attach the solution. I have attached the XML file as a ZIP file. You will be able to get the XML file after unzipping the ZIP.
NativeTests.zip (483 Bytes)

Things that I tried but still didn’t work

  1. Removing the XML comment
  2. Changing the encoding of the file
  3. Reducing the number of test cases to 1
  4. Changing the slashes in the file path
  5. Using a relative path than an absolute path in the file path.

Thanks in advance.

I had previously posted this on SonarSource/sonar-scanner-msbuild GitHub: Unable to upload the Generic Test Execution XML file to SonarQube server · Issue #877 · SonarSource/sonar-scanner-msbuild · GitHub

Hi,

Welcome to the community!

Your version is past EOL. You should upgrade to either the latest version or the current LTS at your earliest convenience. Your upgrade path is:

7.9.1 → 8.9.8 → 9.4 (last step optional)

You may find the Upgrade Guide and the LTS-to-LTS Upgrade Notes helpful. If you have questions about upgrading, feel free to open a new thread for that here.

Regarding your issue, I have a few factoids that might shed some light:

  • Community Edition doesn’t support C++
  • The Generic Test import doesn’t support “unknown” languages (which C++ is in a Community Edition context)

 
HTH,
Ann

Hi G Ann Campbell,
We recently got the enterprise edition and the execution run was successful. But I am unable to upload the unit test cases data to SonarQube sever.

I am running the following command:
C:\SonarQube\bin\SonarScanner.MSBuild.exe begin /k:AnalyticsW.sln /n:AnalyticsW.sln /v:22.1.0 /d:sonar.cfamily.threads=8 /d:sonar.cfamily.build-wrapper-output=cfamily-output /d:sonar.testExecutionReportPaths=C:\Temp\Merged.xml,C:\Temp\AnalyticsCoreTests.exe.sonarqube,C:\Temp\AnalyticsNativeTests.exe.sonarqube,C:\Temp\AnalyticsWTests.exe.sonarqube /d:sonar.verbose=true

Following are the entries I see in the logs:

11:24:10.421 INFO: Sensor Generic Test Executions Report
11:24:10.422 INFO: Parsing C:\Temp\Merged.xml
11:24:10.562 INFO: Imported test execution data for 0 files
11:24:10.562 INFO: Test execution data ignored for 54 unknown files, including:
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\BasicBarriersTest_CmBarrierOpts.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\BasicBarriersTest_FxBarrierOpts.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\BuiltInsTest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\CashflowGuideTest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\DateRangeSetUtilitiesTest.cpp
11:24:10.562 INFO: Parsing C:\Temp\AnalyticsCoreTests.exe.sonarqube
11:24:10.598 INFO: Imported test execution data for 0 files
11:24:10.598 INFO: Test execution data ignored for 20 unknown files, including:
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\BasicBarriersTest_CmBarrierOpts.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\BasicBarriersTest_FxBarrierOpts.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\BuiltInsTest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\CashflowGuideTest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsCoreTests\DateRangeSetUtilitiesTest.cpp
11:24:10.599 INFO: Parsing C:\Temp\AnalyticsNativeTests.exe.sonarqube
11:24:10.613 INFO: Imported test execution data for 0 files
11:24:10.613 INFO: Test execution data ignored for 1 unknown files, including:
C:\Product\HRx\Components\RevalAnalytics\AnalyticsNativeTests\BarrierOptionTests.cpp
11:24:10.613 INFO: Parsing C:\Temp\AnalyticsWTests.exe.sonarqube
11:24:10.632 INFO: Imported test execution data for 0 files
11:24:10.632 INFO: Test execution data ignored for 33 unknown files, including:
C:\Product\HRx\Components\RevalAnalytics\AnalyticsWTests\ActActISMATest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsWTests\CalendarTest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsWTests\CoreUtilsTest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsWTests\DayCountTest.cpp
C:\Product\HRx\Components\RevalAnalytics\AnalyticsWTests\FxRateHistCacheTest.cpp
11:24:10.632 INFO: Sensor Generic Test Executions Report (done) | time=211ms

The ‘cpp’ files are ignored after entering the following function: sonarqube/GenericTestExecutionReportParser.java at 5a2393a0cd4503d84e42631abdde086bfbfb3901 · SonarSource/sonarqube · GitHub

Why are these files getting ignored? Am I missing a parameter? Can you please investigate what is the issue here?

I have attached the log file and xml file in the
UnitTestXMLsAndSonarEndLog.zip (328.4 KB)

I forgot to add details related to the software versions we are using:

SonarScanner for MSBuild 5.2.2
SonarQube Enterprise Edition Version 8.9.8 (build 54436)
Languages in my solution: C and C++

Hi,

Do the file paths in the report match the file paths analysis is seeing? A mismatch there is generally the cause of this problem.

 
Ann

Hi G Ann Campbell,
You were correct. There was a mismatch between the paths specified in the files and the actual path. Once I ran the following command again, it threw a new error.

C:\SonarQube\bin\SonarScanner.MSBuild.exe begin /k:AnalyticsW.sln /n:AnalyticsW.sln /v:22.1.0 /d:sonar.cfamily.threads=8 /d:sonar.cfamily.build-wrapper-output=cfamily-output /d:sonar.testExecutionReportPaths=C:\
Temp\AnalyticsCoreTests.exe.xml,C:\Temp\AnalyticsNativeTests.exe.xml,C:\Temp\AnalyticsWTests.exe.xml

Error:

ERROR: Error during SonarScanner execution
	ERROR: Error during parsing of generic test execution report 'C:\Temp\AnalyticsCoreTests.exe.xml'. Look at the SonarQube documentation to know the expected XML format.
	ERROR: Caused by: Line 3 of report refers to a file which is not configured as a test file: C:\Product\HedgeRx\Dev\Components\RevalAnalytics\AnalyticsCoreTests\BasicBarriersTest_CmBarrierOpts.cpp
	ERROR: 
	The SonarScanner did not complete successfully
	05:59:00.126  Post-processing failed. Exit code: 1

So I searched again and found this issue: Error "file is not configured as a test file".

The folder structure of my Analytics Solution is like this:

The highlighted folders consist of test files. So after referring to the above-mentioned post, I updated my Sonar Begin command(added sonar.tests.inclusions property) and executed it:

C:\SonarQube\bin\SonarScanner.MSBuild.exe begin /k:AnalyticsW.sln /n:AnalyticsW.sln /v:22.1.0 /d:sonar.cfamily.threads=8 /d:sonar.cfamily.build-wrapper-output=cfamily-output /d:sonar.testExecutionReportPaths=C:\
Temp\AnalyticsCoreTests.exe.xml,C:\Temp\AnalyticsNativeTests.exe.xml,C:\Temp\AnalyticsWTests.exe.xml /d:sonar.tests.inclusions=AnalyticsCoreTests/**/*,AnalyticsNativeTests/**/*,AnalyticsWTests/**/*

but this still throws the Line 3 of report refers to a file which is not configured as a test file: C:\Product\HedgeRx\Dev\Components\RevalAnalytics\AnalyticsCoreTests\BasicBarriersTest_CmBarrierOpts.cpp error

So I tried the following variation(added /d:sonar.exclusions property):

C:\SonarQube\bin\SonarScanner.MSBuild.exe begin /k:AnalyticsW.sln /n:AnalyticsW.sln /v:22.1.0 /d:sonar.cfamily.threads=8 /d:sonar.cfamily.build-wrapper-output=cfamily-output /d:sonar.testExecutionReportPaths=C:\
Temp\AnalyticsCoreTests.exe.xml,C:\Temp\AnalyticsNativeTests.exe.xml,C:\Temp\AnalyticsWTests.exe.xml /d:sonar.tests .inclusions=AnalyticsCoreTests/**/*,AnalyticsNativeTests/**/*,AnalyticsWTests/**/*  /d:sonar.exclusions=AnalyticsCoreTests/**/*,AnalyticsNativeTests/**/*,AnalyticsWTests/**/*

But this just ended ignoring all the files like before.

Can you please guide what should be the correct SonarQube Begin command?

Hi,

Test inclusions and exclusions only work on the set of files already understood by analysis to be test files. You cannot add a file to the test set with an inclusion. Nor will excluding it from the source file set turn it into a test file.

Please take a look at the docs that explain how test files are identified.

 
Ann