Jacoco XML format cannot work as good as the binary format for multi-module maven

SonarQube Server / Community Build
1

SonarQube 8.1, sonar-maven-plugin 3.7.0.1746, jacoco-maven-plugin 0.8.5

At our company we all use the sonar.jacoco.reportPath property and it works great. We let all the maven modules write to a single exec file with the property “append=true”. The maven sonar plugin reports all coverage.

It’s currently impossible to switch to sonar.coverage.jacoco.xmlReportPaths, without doing major changes to all the projects.

As an example, take the setup of 3 maven modules M1, M2, M3. Where M1 is a dependency of M2 and M2 is a dependency of M3.

As we write to 1 exec file for all modules, running sonar:sonar takes all tests into accounts when analysing M1, M2 and M3.

But for the jacoco reports, it’s not possible todo this. The report step of jacoco will already run when M1 is building, and at this stage the tests in M2 and M3 have not. The coverage is incorrect.

Jacoco’s solution is to create a report project and use the report-aggregate. https://github.com/jacoco/jacoco/tree/master/jacoco-maven-plugin.test/it/it-report-aggregate
This is not feasable for all our old projects. We dont want to create a seperate maven module for every tool that is around that we use. This solution is too intrusive on the structure of the project.

Another solution that I tried is to use the report and the report-aggregate goal for all projects. There are in total 6 reports, 2 for each module. And the sum of reports is correct. BUT when running sonar:sonar, sonar analyses M1 and takes only the 2 reports of his module into account, and not the aggregate reports of M2 and M3.

Jacoco could also offer a merge-report step. Or could offer to write to a single location with an “append=true” like the binary format. But they unfortunately do not. Reports are generated seperately and writing to a single location just overwrites the reports. The jacoco-maven-plugin only has a merge goal for exec files, not for report files.

I know the jacoco guys could do something different to allow a full xml report. But they dont.

Sonar tries to push projects to use the xmlReportPaths, but that is just not feasible for multi module maven projects which have tests that cover classes in multiple projects. The needed configuration is very error prone and not ok for big organisations.

The binary jacoco format works out of the box, is dummy proof and easy to configure.

If sonar decides to drop the support of the binary format, we have a problem. I read in forums that this is your plan. Can you please keep supporting it or provide an alternative where the xml reports are as easy to use?

3 Likes
2

Peter,

Thanks for the feedback. Heads-up: support for the JaCoCo binary format is already dropped as of v6.0 of SonarQube’s Java Analyzer.

It was dropped for good reasons (see MMF-1651)

We have two ways to load coverage data generated by JaCoCo for Java projects: thru the .exec or thru the XML file.

We know that the .exec file is not an exchange format. This is an internal format used by JaCoCo developers. It is not at all recommended to use it for integration purpose.

Because of that, we want to get rid of the support of the .exec file format and rely only on the XML one.

It’s a pain for both our Java analyzer developers and for SonarQube users when the binary format gets updated and SonarQube hasn’t had time to catch up. It’s happened before where a range of JaCoCo versions had to be marked as compatible for coverage import, which is just… messy, and inconvenient. The .exec format, while perhaps convenient, is not a stable integration point.

I think your use-case is covered in this post under Multi module builds

If you have any feedback or ideas on how the configuration could be made easier, we’re happy to hear it!

1 Like
3

Hi Colin,

thank you for your answer.

I’m afraid it’s on the one hand more a jacoco problem, but of course it turns into a sonar problem if you only allow xml. I have 2 problems with the suggested approach:

The report-aggregate goal only takes code coverage of direct maven dependencies, not indirect ones. In my example, M2 is a dependency of M3. M1 is an indirect dependency of M3, as its a dependency of M2.

I dont want to make M1 a direct dependency of M3, that’s architecturally not correct. (example: M3 should not call classes of M1, it should use a facade layer in M2, which in his turn calls M1).

A minor issue, but as all modules (m1, m2, m3) have tests, I need to add all xml reports to the sonar property.
But keep in mind that for 3 maven modules, it results in this (report + report aggregate per module): <sonar.coverage.jacoco.xmlReportPaths>…/jacoco-m1/target/site/jacoco-ut/jacoco.xml,…/jacoco-m1/target/site/jacoco-aggregate/jacoco.xml/jacoco.xml,…/jacoco-m2/target/site/jacoco-ut/jacoco.xml,…/jacoco-m2/target/site/jacoco-aggregate/jacoco.xml/jacoco.xml,…/jacoco-m3/target/site/jacoco-ut/jacoco.xml,…/jacoco-m3/target/site/jacoco-aggregate/jacoco.xml</sonar.coverage.jacoco.xmlReportPaths>
For projects with 6 or more maven modules, its getting a bit out of hand :slight_smile:

(If I do this in my test project, it does give me 100% in sonar. But adding M1 as a direct dependency of M3 is I think not feasible in many companies and projects)

So I guess two changes are needed at jacoco side, and possibly an improvement for sonar.

  1. the report aggregate goal should take into account all modules of the project, thus also indirect dependencies. Even easier would be that the report aggregate also takes into account its own module. Currently, the setup is that report-aggregate is for a seperate maven module which only contains integration tests. Therefor, the report and report-aggregate needs to be used.

  2. jacoco should make it possible to write coverage data to the same xml file. (like they do with the binary format, with append=true) This single XML file can be used for sonar. This would make the xml format as easy as the binary.

  3. For sonar, it would be nice to allow wildcards in sonar.coverage.jacoco.xmlReportPaths, as an alternative to 2) above.( “…/**/jacoco.xml”)

Please take into account that the binary format is really easy to configure and straightforward. When writing to a single binary file, you cant make a mistake.

The xml stuff… well thats really tricky. You need to dive deep into how it actually works and what the steps are. And as you know, I still cant get it too work without changes in the actual dependency structure.

4

The binary format still works in sonar 8.1. When will it stop working?

6

Hi Peter,

Short answer
In the next SonarQube version

Long answer
The ticket SONARJAVA-3067 Remove JaCoCo .exec support was part of SonarJava 6.0.
This impacts the following version of SonarQube:

  • SonarQube 8.1 includes by default:
    • SonarJava 5.14 that support binary format with sonar.jacoco.reportPaths (with a deprecation message)
    • JaCoCo plugin 1.0.2 that support xml format with sonar.coverage.jacoco.xmlReportPaths
  • SonarQube 8.2 includes by default:
    • SonarJava 6.1 that does not support anymore JaCoCo binary import (only detect the presence of former property to display a warning).
    • JaCoCo plugin 1.0.2 that support only sonar.coverage.jacoco.xmlReportPaths
7

Thank you for the reply Alban!

We’re currently using the lts version of the enterprise edition. We want to upgrade to 8.2, mainly for the gitlab integration. But I guess that’s not a good idea anymore, as all our projects still use the binary format. (and it’s not that easy to just use the xml)

Is there a way to still enable the binary format in 8.2?

8

Hi Peter,

No, you will not be able to import JaCoCo binary format with SonarQube 8.2.
We need to find a working solution for your configuration using the XML format. As Colin mentions above, the XML format is better because it will not be broken at each new release of JaCoCo.
About your idea to support wildcards in sonar.coverage.jacoco.xmlReportPaths, I created this ticket JACOCO-15, but I don’t see how it could help in your context.
Could you share with us a public simple multi-modules hello world project, where it’s possible to aggregate coverage using binary format but not using XML format? So we could have a concert example to work on and find a solution.

9

Hi Alban,

here is a public github repo with 2 examples: https://github.com/peternelissen/jacoco-sonar

One example uses the binary format and has 100% coverage in sonar.

Another example uses report and report-aggregate goals of jacoco and have 87% coverage. To get it to 87, please also look at the value of sonar.coverage.jacoco.xmlReportPaths, which is obviously not a good idea for big projects.

kind regards,
Peter

2 Likes
10

FYI https://github.com/jacoco/jacoco/issues/974

11

Hello Alban, did you have time to look at the example? Is the problem clear? Let me know if you need more info. Thank you!

12

Anything new about this topic? We also waiting for a solution for this issue.

2 Likes
13

We’re in the same predicament where we’re stuck on 8.1 until a solution for jacoco or support is added back for the binary format.

14

One trick thing is that you need to import only the aggregated jacoco.xml. As we’re generating the normal report for each submodule the aggregated one got overriden and then we could not get overall coverage.

So for us, besides the report-aggregator module which you can find an “official” example here, passing the absolute path of aggregated report to sonar did the trick:

mvn sonar:sonar -Dsonar.coverage.jacoco.xmlReportPaths=absolutePathToReportProject/target/site/jacoco-aggregate/jacoco.xml

But indeed it was quite challenging reaching to this fix, both sonar scanner and jacoco report aggregation must be improved

1 Like
15

Hi Peter,

Thanks for your public github repo, it helped us to reproduce and investigate your problem.
Sorry for the late answer, I did some quick test at the beginning to try to find a solution for your “jacoco-example4” project, but I failed. Then I had to wait to have more time to investigate and some help from a colleague. Unfortunately, even with more time, we also failed to find a solution and believe that using JaCoCo 0.8.5 there’s no solution.

A workaround for the “jacoco-example4” is to add explicitly in “jacoco-example4/jacoco-m3/pom.xml” the “jacoco-m1” dependency.

Otherwise, the only solution that we see is to update the “report-aggregate” of “jacoco-maven-plugin”.
We will discuss about this with the JaCoCo team:

Regards,

Alban

16

Hi @Peter_Nelissen,

One thing I understood from the discussion on JaCoCo issue 974 is that maven lifecycle is not suitable to run something after all modules. A colleague helped me to find the following solution: let’s run the report in a command that comes after the full build:

$ mvn clean install
$ mvn jacoco:report sonar:sonar

So here is my proposal to migrate your jacoco-example1 from native to XML format:

You just need to remove the “sonar.jacoco.reportPath” property and set the “jacoco.dataFile”.

Do not merge it, but can you confirm that it works for you?

17

Hi Alban,

you are right, two seperate commands and it works.

But how many mistakes will be made against this?

How long will it take before someone thinks to optimise the commands and put it together?

Every project also needs his own commands (some will use report aggregate and not report, or whatever). Thats the nice thing about the maven lifeycle, you can attach stuff to it that is project specific and run default maven commands.

Also, how long will it take me to explain why thats needed to all the teams?

The bottom line for me, is that something as basic as code coverage should run simple and be correct.

And it was very simple with the binary format, as we could just write to one single file. The xml reports overwrite each other if you try to write it to same location, it does not have an “append” function.

Sonar plugin also runs at the end of the maven build, which could also be an option for the xml report.

There are good options that make it as easy as ever.

I dont feel comfortable pushing workarounds to the entire company over something that I was hoping is basic in 2020 :slight_smile:

kind regards,
Peter

19

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.