GenericTestExecution does not work with maven multi-modules

Hi

I’ll let Duarte decide about the logs, but it makes sense to go ahead with a reproducer.

 
Thx,
Ann

I confirm that this problem happens with version 7.9.1. The sonar.testExecutionReportPaths property is ignored when specified in the pom of a submodule. The log miss the any reference to “Sensor Generic Test Executions Report”.

The property is taken in consideration when included in the main module’s pom, an the logs contains this part:

11:15:51  [INFO] ------------- Run sensors on module ***<main module name>***
11:15:51  [INFO] Sensor JaCoCo XML Report Importer [jacoco]
11:15:51  [INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
11:15:51  [INFO] Sensor HTML [web]
11:15:51  [INFO] Sensor HTML [web] (done) | time=0ms
11:15:51  [INFO] Sensor XML Sensor [xml]
11:15:51  [INFO] 1 source files to be analyzed
11:15:51  [INFO] Sensor XML Sensor [xml] (done) | time=13ms
11:15:51  [INFO] Sensor Generic Test Executions Report
11:15:51  [INFO] Parsing /opt/jenkins/workspace/***main module name****/***submodule name****/target/test/javascript/reports/karma/ut_report.xml
11:15:51  [INFO] 1/1 source files have been analyzed
11:15:51  [INFO] Imported test execution data for 0 files
11:15:51  [INFO] Test execution data ignored for 1 unknown files, including:
11:15:51  src/app/app.component.spec.ts
11:15:51  [INFO] Sensor Generic Test Executions Report (done) | time=6ms

As expected the content of the report is ignored because the main module doesn’t cotains the tested files.

Thanks for your confirmation. It’s really disappointing that this feature does not work at all.
Any estimation when this will be fixed?
Is there already an issue ticket created which can be linked here?

Looks like this is broken since 7.6
https://jira.sonarsource.com/browse/SONAR-11530

Hi,

Thanks @reitzmichnicht for the detailed analysis of the issue.
I have the exact same issue where I have a multi-module application made of React modules.
I can generate an individual Generic Test Data report per module using jest-sonar-reporter in each module.
But how can we make SonarQube 7.6+ take these into account ? Any hint here would be welcome.

Thanks,
Nico

@ganncamp any updates on this? Looks like this is a regression introduced in 7.6 which could be easily fixed.

As Andrea said, the sonar.testExecutionReportPaths is simply ignored when specified in the pom of a module.

The only way to trigger the execution of the Generic Test Execution Sensor is to specify the property in the root pom. However, although the sensor executes, it will complain about not being able to find the source files referenced in the reports. Reports are generated by child modules and often reference source files relative to their own base - such relative reference cannot be resolved from the root pom - hence the reason why the Sensor complains (executed from the root).

Although there is a way to trigger the execution of the Sensor - this is not a viable workaround since it will ultimately refuse to consider the content of the read reports :frowning:

The solution is even more confusing that we can specify properties for other sensors in child poms without any problem. Please, have a look at this issue…

BTW: I confirm the issue is still there with SonarQube 8.2.

@reitzmichnicht Did you find any viable workaround for this multi-module setup ?
@ganncamp What’s the status about this issue? Any chance you’ll be looking at it in the near future?

1 Like

BTW, the specs for the Generic Test Execution Report format (https://docs.sonarqube.org/latest/analysis/generic-test/) says:

Insert a file element for each test file. Its path attribute can be either absolute or relative to the root of the module.

As we have seen here, this is definitely not correct… paths must either be:

  • absolute;
  • or relative to the root of the project (i.e. in case of maven multi-module, the path to the project pom)

But certainly not relative to the root of the module!

Sorry, no updates on my side.

 
Ann

Sadly I did not find any workaround yet for this problem.

@ganncamp A year already this issue is opened and still no updates :frowning:

@reitzmichnicht We decided to make our own Sonar Sensor plugin to upload Generic Test Execution reports…

I was able to work around this in my setup but I don’t like it as it is ugly as hell…

The basic steps are :

  • Produce the Generic Test Execution Reports with absolute file names
  • In the root pom.xml, the one used to launch the SonarQube analysis, add a Maven property called sonar.testExecutionReportPaths
  • In the sonar.testExecutionReportPaths, enter the paths to each individual Generic Test Execution Reports files, separated by a comma.These paths can be relative to the root pom.xml. For example, <sonar.testExecutionReportPaths> moduleA/target/generic-test-report.xml,moduleB/target/generic-test-report.xml</sonar.testExecutionReportPaths>

As I was saying, it is ugly as hell because :

  • You have to change the way the Generic Test Execution Reports are generated to use absolute file names, and this is not always possible if you rely on a 3rd party plugin for example
  • You have to manually list all your modules in the property instead of relying on Maven information. This is error prone in case a new module is added for example, it’s easy to add it in Maven <modules> section and forget to update the property.
  • You have to introduce this property in the root pom.xml file. My personal style is to keep this file as clean as possible and use a dedicated pom.xml for plugins configuration
2 Likes

@ganncamp I’m not really satisfied about the product roadmap here.

We are a paying sonarqube customer and this bug is still open on 8.5 we use. That limits the way how we are able to use sonar.
I even do not see a jira ticket created for this issue!

Connecting this to sonar.testExecutionReportPaths is not working at module level where this bug was discussed as well.

And what’s worse, the scanner errors out, rather than just giving a warning, if (one of) the path(s) defined for sonar.testExecutionReportPaths does not exist. And, contrary to all(?) other path properties it does not support wildcards. See Sonar-scanner doesn't support wildcards in sonar.testExecutionReportPaths.

If you force us to define sonar.testExecutionReportPaths at root level it should be possible to enter wildcarded paths that may or may not exist in all Maven (sub) modules.

This bug just consumed a lot of my nerves.

It also gets worse if you’re running on container-based CI infrastructure, because you cannot longer rely on static absolute paths during a pipeline run consisting of multiple jobs.

I’m running a CI setup where multiple tests (jest, cypress) are executed in different jobs, and a “final” sonarqube analysis job gathers all the test execution data. When the final job picks up the the test execution artifacts, the file paths no longer match the file paths in the final job container:

test execution file (produced by first job):

<testExecutions version="1">
  <file path="/builds/umwHayWZ0/myproject/mymodule/tests/unit/mytest.spec.js">
    <!-- .... -->
  </file>
</testExecutions>

log output of final sonarqube analysis job:

[INFO] Parsing /builds/jksdSajkfds0/myproject/mymodule/target/cypress-sonarqube-reporter/cypress-sonarqube-reports.all.xml
[INFO] Imported test execution data for 0 files
[INFO] Test execution data ignored for 21 unknown files, including:
/builds/umwHayWZ0/myproject/mymodule/tests/unit/mytest.spec.js
[....]

So the way I’m currently seeing it, sonar.testExecutionReportPaths is just completely broken.

BTW: Using SonarQube 8.9 LTS

The support for the module-level setting was removed in v7.6: [SONAR-11530] Change Generic coverage/test reports sensors to be global Sensors - SonarSource.

Even with module-level support, you wouldn’t be able to specify absolute paths produced by different jobs. The test execution data needs to be generated using relative paths to the root of the project.

GitHub - SonarSource/sonar-scanner-msbuild: SonarScanner for .NET still uses module-levels.

Is there some way to have a common config for different sort’s of modules? we have a maven build with a common parent pom, but not all modules build java applications. Some build nodejs packages. Those modules do not follow the conventions of the java build and the property sonar.testExecutionReportPaths cannot be the same.

Or is the general idea to configure both maven (junit) and nodes (jest) to write the same report file into the same location?

Hi @wemu,

According to the docs, sonar.testExecutionReportPaths accepts a

Comma-delimited list of paths to execution reports

So it would just be about configuring it with a full / correct list.

However, that parameter has little to do with JUnit and Jest tests.

Perhaps you should create a new thread with your full details.

 
Ann