Jacoco Coverage: switch from deprecated Binary to XML format in a Tycho build shows 0%

beyerj · June 18, 2019, 11:02am

Using SonarQube 7.7 (community), Jacoco Plugin 1.0.1 (build 143).

We have a more or less standard Tycho build with eclipse-plugin and eclipse-test-plugin modules inside a multi-module maven structure. Because of this nature, an eclipse-test-plugin never contains sources of their dependencies.

We are using the common Tycho approach of a global jacoco.exec file with reuseReports. This works.

Now, SonarQube complains about this approach being deprecated, and we should use the Jacoco XML Reports instead. So, knowing how fast SonarQube deprecations are enforced in general, I wanted to act on it before it is too late.

Problem is: I can not get SonarQube to import those XML reports.

If I keep the global jacoco.exec, and run mvn jacoco:report, it does not generate anything useful in the jacoco report (HTML and XML). Running mvn jacoco:report-aggregate, it creates nothing useful in any report. In both cases, I can not expect SonarQube to do something useful.
If I remove the usage of a global jacoco.exec, the separate jacoco.exec files are generated in the test plugins target directories, as expected. Running mvn jacoco:report on this, only coverage information of the test classes is in the report, which is even less than nothing. Running mvn jacoco:report-aggregate, the aggregate contains only coverage information of the dependencies having sources - which is PERFECT (or so I thought).

So the only combination found working is: separate jacoco.exec files for each module, running mvn jacoco:report-aggregate afterwards.

But (and here it comes): SonarQube analysis does seem to pick up the report file (at least it says “[DEBUG] 11:57:37.413 Reading report ‘path\to\eclipse-bundle\target\site\jacoco-aggregate\jacoco.xml’” with mvn sonar:sonar -X). The path is correct, the file is there, and it contains expected content. But the coverage in the project within SonarQube is 0%.

Unfortunately, the SonarQube docs do not reflect this deprecation yet. [1] talks only about the binary format. [2] says the XML format should be used, but not, how it is generated.

Can you please elaborate on how to generate the Jacoco XML report in a multi module maven project with extensive dependencies between the modules, and the test projects only containing sources for tests and not the code to be tested, to have it correctly being picked up by SonarQube?

Thank you very much.

[1] https://docs.sonarqube.org/display/PLUG/Usage+of+JaCoCo+with+SonarJava
[2] https://docs.sonarqube.org/display/PLUG/JaCoCo+Plugin

saberduck · July 16, 2019, 5:21pm

Hello @beyerj,

sorry for the delay. I am not familiar with Tycho, however the fact that you managed to create the report with all coverage data is probably sufficient. My suspicion is that this report is being imported only for a single module, which obviously doesn’t contain all the files.

How do you set sonar.coverage.jacoco.xmlReportPaths property? Can you try to set the same path to the aggregated report in every module? Can you share debug logs from the analysis?

If it’s possible, can you create some minimal project demonstrating the issue and share it here?

beyerj · July 17, 2019, 10:09am

Hi Tibor,

I was able to kind-of solve this.

The problem seems to be that jacoco XML reports seem to have two different formats, depending on how you create them. There’s jacoco:report and jacoco:report-aggregate. First format is understood by Sonar, second format isn’t. From what I have seen (sorry, it’s been a while), it looks like the aggregate format contains the normal reports all listed as XML sub-nodes of another different root node. But I may be wrong there, or simply missing subtle differences.

What I did now was: Create the jacoco.exec the way Tycho expects it (aggregated, with append option set to true, because else the coverage information would be totally wrong), but then run jacoco:report on the tycho multimodule project pointing to the common jacoco.exec for the datafile, which then creates a report separately for every module. Then run sonar:sonar with sonar.coverage.jacoco.xmlReportPaths set to the jacoco XML report for each module. This way, it works.

The not-so-nice problem: I have to run the report manually before sonar:sonar. With using the jacoco.exec file, this step was not necessary. Not a too big problem, but since Tycho is slow starting up a build due to early dependency management with big projects, it slows down the sonar build.

It would have been easier to setup if sonar:sonar would have understood the XML report produced by jacoco:report-aggregate, but this would also not have saved me from running another maven job between the instrumented build+tests and the sonar run.

saberduck · July 22, 2019, 8:46am

hello @beyerj,

I prepared example using aggregate report in our scanning examples repository. The main “trick” is to import the same aggregate report for each module.

I also wrote a guide about this, have a look here [Coverage & Test Data] Importing JaCoCo coverage report in XML format

I didn’t find any issue importing the XML aggregate report, current plugin understands the format produced by JaCoCo.

Feel free to come back to us if you think there are some improvements we can do.

antoine-morvan · July 23, 2019, 9:09am

Hi,

The example and guide you provide do not seem to use Tycho (i.e. build Eclipse plugins).

The main difference in Tycho builds is that dependencies are not explicitly declared in the POM files, but are derived from Manifest files. It seems to me that the report-aggregate goal does not see that the test plugins depend on the implementation plugins, and does not sum the coverage.

I’m also trying to find a solution, but did not succeed yet…

Best.

saberduck · July 23, 2019, 9:24am

@antoine-morvan then this seems to be something to be fixed on JaCoCo project side, to support dependency resolution of the Tycho projects (but I believe same people work on the EclEmma plugin of Eclipse, so they should be familiar with it)

antoine-morvan · July 23, 2019, 9:31am

That is my thought too but still investigating …

antoine-morvan · July 23, 2019, 10:59am

Here is one demo project with tycho, sonar and jacoco. Running mvn clean verify sonar:sonar gives 0% coverage in sonar. Some of the jacoco reports (report-aggregate), however, do report some coverage (in the test modules).

antoine-morvan · July 23, 2019, 11:57am

I managed to get the coverage of both test and implementation coverage generated by Jacoco (see latest commit on example above).

The “report” goal of Jacoco reports the coverage for the current test module under “site/jacoco/jacoco.xml”. The “report-aggregate” reports the coverage for the dependencies (i.e. the implementation modules), in a separate report, under “site/jacoco-aggregate/jacoco.xml”. Both reports contains proper values (i.e. 100% in the example above).

However Sonar only reads/displays the coverage on the test modules, despite setting 2 reportXmlPaths.

Still trying …

saberduck · July 23, 2019, 4:05pm

@antoine-morvan are you setting sonar.coverage.jacoco.xmlReportPaths on every module which should be covered? Can you post debug logs, just to see if the report is found?

beyerj · July 24, 2019, 5:39am

@antoine-morvan I sent a pull request on your Github repo.
This is the configuration which works for me with Tycho and the Jacoco XML report style as coverage source for Sonarqube. The project reports 100% coverage in Sonarqube.
You needed to keep the global aggregated jacoco.exec as is recommended by Jacoco+Tycho even for the old way of using the jacoco.exec in Sonarqube.
(sorry it’s a German screenshot)

I still think it is not nice to have to run jacoco:report manually. But the way Tycho works, and the way Jacoco report works, that probably can’t be changed.

saberduck · July 24, 2019, 9:03am

@antoine-morvan I checked your project and I noticed two properties

    <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
    <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>

they were deprecated and removed long time ago, they have no effect. See https://jira.sonarsource.com/browse/SONARJAVA-475 and https://jira.sonarsource.com/browse/SONAR-5185

@beyerj I am not sure I follow, why do you need to run jacoco:report manually? Can’t you just bind it to verify phase?

beyerj · July 24, 2019, 12:05pm

@saberduck Regarding your question to bind jacoco:report to verify phase:

From what I have seen, this does not work. In Tycho (as in Eclipse projects in general), test sources are not part of the productive source project (there is no /src/main/java and /src/test/java). You can actually see the general structure in the Tycho sample project @antoine-morvan shared. Tests are done in a separate bundle project which has dependencies to the original project. If you run the jacoco:report in verify, you will not generate any coverage information, because at the time jacoco:report runs in the productive bundle, the test has not yet run, therefor there is no coverage information.
The jacoco report has to be created after the tests are run. And the test projects are usually the last projects run (natural dependency order).

antoine-morvan · July 24, 2019, 1:21pm

@saberduck regarding the sonar.coverage.jacoco.xmlReportPaths, I only set the value in the modules running tests as the “implementation” modules will not have any Jacoco report. And I was not aware of the deprecated sonar configs.

Here is the log from the build before @beyerj PR, (mvn -B clean verify sonar:sonar) : log.txt (34.3 KB)

@beyerj Thanks for the PR, I’ll have a look and try to bind the report to some other phase. Also, regarding the long Tycho init time, you can use -Dtycho.mode=maven to skip the P2 dependency resolution (see https://kthoms.wordpress.com/2019/06/12/eclipse-tycho-disable-p2-dependency-resolution-with-tycho-modemaven/ for instance). mvn -Dtycho.mode=maven jacoco:report -Djacoco.dataFile=../../target/jacoco.exec sonar:sonar worked fine.

saberduck · August 13, 2019, 8:34am

@antoine-morvan, did you manage to fix your issue? From the logs I see that report is not found

[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[WARNING] Report doesn't exist: '/home/anmorvan/git/tycho-sonar-jacoco/target/site/jacoco/jacoco.xml'
[WARNING] Report doesn't exist: '/home/anmorvan/git/tycho-sonar-jacoco/target/site/jacoco-aggregate/jacoco.xml'
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
[INFO] Sensor JavaXmlSensor [java]

You should import aggregate report on every module using absolute path.

You can have a look at eclemma eclipse plugin, which is also using a tycho build

antoine-morvan · August 19, 2019, 7:44am

Yes it’s fixed, despite the log showing warnings for missing reports.