Ignored coverage

I did another test run. This time I configured JaCoCo to only include coverage report from my “coverage” module. Please find the logs of this run bellow:
sonar2.log.zip (2.3 MB)

Once results of this run were uploaded into sonarcloud, final coverage drop from 10% to about 3.9%. That is just because some modules contains custom integration tests and results of this tests were uploaded too (for example module “jfxsoft.easymvc.controls.autocompletion” and “jfxsoft.easymvc.controls.button-bar.parent”).

I also checked logs and I can see that while processing my “coverage” module, coverage report was found:

[INFO] 08:40:25.964 Sensor JaCoCo XML Report Importer [jacoco]
[INFO] 08:40:25.964 Importing 1 report(s). Turn your logs in debug mode in order to see the exhaustive list.
[DEBUG] 08:40:25.964 Reading report '/home/petr/Dokumenty/jfxsoft/jfxsoft.easymvc.parent/jfxsoft.easymvc.coverage/target/site/jacoco-mega-aggregate/jacoco.xml'
[INFO] 08:40:25.990 Sensor JaCoCo XML Report Importer [jacoco] (done) | time=26ms

This report should result in 30% coverage as in my IntelliJ. But for some reason it is not inclued.

Hi,

I’m not shocked that excluding all but one module from the coverage report you pass in to analysis would result in a lower aggregate coverage value.

Of that one module, or overall?

 
Ann

Module “coverage” does not contain any code. It is just used as aggregator of coverage of all modules.

It is overall modules:

Hi,

Yes, I understand that.

So… You’re expecting analysis to show 30% coverage overall by feeding it the coverage of only one module?

Let’s back up. Why are you aggregating everything into one, centralized report that’s stored in a different module? Can you not aggregate per-module?

 
Ann

I see your point. Let me maybe visualize the project structure little bit better:

parent
| - module 1
| |- source code
| |- unit tests
| |- integration tests
| |- JaCoCo report for module1
| - module 2
| |- source code
| |- unit tests
| |- integration tests
| |- JaCoCo report for module2
| - it module
| |- integration tests for module1 and module2 together
| |- JaCoCo report for it module
| - coverage module
| |- JaCoCo report for all modules together: module1 + module2 + it module

The number of “source” modules is much more, but it does not matter.

To your question:

I’m sending aggregated results per-module, but because some parts of code are hard to test in isolated test, I have created a “it-module” where I’m performing this more complex tests.

My expectation is that results from this complex tests will be included in the final coverage. Currently, they are ignored (from my POV), but still successfully uploaded.

Hi,

It seems that each module has both its own integration tests as well as unit tests. Why must the coverage reports be aggregate across the project? Why not put all the results for module 1 into JaCoCo report for module 1?

 
Ann

I did this in the first time. If you check my first attached logs, you will find out that I’m sending results of each module separately. This works like a charm. But result coverage from this way is only 10%. Rest 20% is reached by performing “complex” tests in special “integration” module. And results from this module are ignored.

Hi,

Are those results included in the combined, module-level report?

 
Ann

I would say yes. Based on the logs JaCoCo report is sent to Sonarcloud.

Hi,

You’re generating the normal unit test logs at the module level. You’re generating the integration test logs (at the module level?). Are you performing a merge at the module level?

 
Ann

Yes, I’m performing a merge at the “single” module level. And then on top of this I’m performing a big merge of all modules (in the coverage module) So I have a lot of results sent to sonar.

Hi,

If you’re merging at the module level, then there’s no need for the ‘coverage module’.

Let’s pick one module to focus on. Can you share how the merged coverage report is passed in to analysis?

 
Thx,
Ann

I would like to pick the only one module from which coverage is not “consumed”. The name of this module is “jfxsoft.easymvc.it”. This module contains complementary tests for rest of the modules.

The plugin configuration of JaCoCo is same for all modules and I already posted the configuration in my previous posts. For Sonar plugin, I have defined a bunch of properties, which I can also share and then just performing Maven Sonar plugin to do the magic.

JaCoCo plugin configuration

<!-- org.jacoco.jacoco-maven-plugin -->
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${maven.jacoco.plugin.version}</version>

    <executions>
        <!-- Prepares the property pointing to the JaCoCo runtime agent which
            is passed as VM argument when Maven the Surefire plugin is executed. -->
        <execution>
            <id>pre-unit-test</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
            <configuration>
                <destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
                <!-- Sets the name of the property containing the settings for JaCoCo
                    runtime agent. -->
                <propertyName>jacoco.agent.argLine</propertyName>
                <append>true</append>
            </configuration>
        </execution>
        <!-- Ensures that the code coverage report for unit tests is created
            after unit tests have been run. -->
        <execution>
            <id>post-unit-test</id>
            <phase>test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
                <!-- Sets the output directory for the code coverage report. -->
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
            </configuration>
        </execution>

        <execution>
            <id>pre-integration-test</id>
            <goals>
                <goal>prepare-agent-integration</goal>
            </goals>
            <configuration>
                <destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
                <!-- Sets the name of the property containing the settings for JaCoCo
                    runtime agent. -->
                <propertyName>jacoco.agent.it.argLine</propertyName>
                <append>true</append>
            </configuration>
        </execution>
        <!-- Ensures that the code coverage report for integration tests is created
            after unit tests have been run. -->
        <execution>
            <id>post-integration-test</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>report-integration</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <dataFile>${project.build.directory}/coverage-reports/jacoco-it.exec</dataFile>
                <!-- Sets the output directory for the code coverage report. -->
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
            </configuration>
        </execution>

        <!-- Merge jacoco results -->
        <execution>
            <id>merge-unit-and-integration</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>merge</goal>
            </goals>
            <configuration>
                <fileSets>
                    <fileSet>
                        <directory>${project.build.directory}/coverage-reports/</directory>
                        <includes>
                            <include>jacoco-ut.exec</include>
                            <include>jacoco-it.exec</include>
                        </includes>
                    </fileSet>
                </fileSets>
                <destFile>${project.build.directory}/coverage-reports/jacoco.all.exec</destFile>
            </configuration>
        </execution>

        <!-- Aggregate results -->
        <execution>
            <id>create-merged-report</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/coverage-reports/jacoco.all.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

Basically this will first configure, where to store results of UT and IT:

<outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>

Than using merge goal it will take results from these two directories and create one merged result called jacoco.all.exec.

Once the single “jacoco.all.exec” is created, aggregate result will be generated and stored in directory

<outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>

At this state, every single module in the project contains it’s own jacoco-aggregate result.

The sonar properties are:

<sonar.organization>jfxsoft</sonar.organization>
<sonar.host.url>https://sonarcloud.io</sonar.host.url>
<sonar.junit.reportPaths>${project.build.directory}/surefire-reports,${project.build.directory}/failsafe-reports</sonar.junit.reportPaths>
<sonar.java.pmd.reportPaths>${project.build.directory}/pmd.xml</sonar.java.pmd.reportPaths>
<sonar.java.checkstyle.reportPaths>${project.build.directory}/checkstyle-result.xml</sonar.java.checkstyle.reportPaths>
<sonar.java.spotbugs.reportPaths>${project.build.directory}/spotbugsXml.xml</sonar.java.spotbugs.reportPaths>
<sonar.coverage.jacoco.xmlReportPaths>${project.build.directory}/site/jacoco-aggregate/jacoco.xml</sonar.coverage.jacoco.xmlReportPaths>
<sonar.exclusions>**/generated-sources/**,**/generated-test-sources/**</sonar.exclusions>

The most important property is sonar.coverage.jacoco.xmlReportPaths which correcponds with the path where the jacoco-aggregate result is located.

So far this configuration works as expected which means that when a module contains some tests (either unit or integration), results will be merged, aggregated and sent to sonar. That is a reason of having about 10% coverage. I’m happy with this, but I want more.


As I already tried to explain, I have a module called “jfxsoft.easymvc.it”, which contains complex tests. This means it runs a final application and check behaviour of an application as one big piece. Thanks to this I hope I will be able to cover more lines because sometimes it was not possible for me to make a test in the one “source” module.

Configuration of JaCoCo is same for this module as for the others so in the end I will also get jacoco-aggregate directory with result. BUT this time Sonar is not able to process the results and add them to the rest.

Hi,

Have you tried using the aggregation setup from the docs?

 
Ann

Yes, of course. I have done the aggregation setup in my “coverage” module:

<!-- org.jacoco.jacoco-maven-plugin -->
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <executions>

        <!-- Merge results from all coverage reports into single one -->
        <execution>
            <id>merge-results</id>
            <phase>verify</phase>
            <goals>
                <goal>merge</goal>
            </goals>
            <configuration>
                <fileSets>
                    <fileSet>
                        <directory>${project.build.directory}/../../</directory>
                        <includes>
                            <include>**/target/coverage-reports/jacoco.all.exec</include>
                        </includes>
                    </fileSet>
                </fileSets>
                <destFile>${project.build.directory}/coverage-reports/aggregate.exec</destFile>
            </configuration>
        </execution>

        <!-- Generate merged report from all modules -->
        <execution>
            <id>create-mega-merged-report</id>
            <phase>verify</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <formats>
                    <format>XML</format>
                </formats>
                <dataFile>${project.build.directory}/coverage-reports/aggregate.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-mega-aggregate</outputDirectory>
            </configuration>
        </execution>

        <execution>
            <id>report-aggregate</id>
            <phase>verify</phase>
            <goals>
                <goal>report-aggregate</goal>
            </goals>

            <configuration>
                <formats>
                    <format>XML</format>
                </formats>
                <includeCurrentProject>true</includeCurrentProject>
                <dataFileIncludes>
                    <dataFileInclude>${project.build.directory}/coverage-reports/aggregate.exec</dataFileInclude>
                </dataFileIncludes>
            </configuration>
        </execution>

The last execution of JaCoCo Maven plugin performs “report-aggregate” as described in the docs. Unfortunately this does not solve anything and result is ignored by sonar.

Do you have more ideas of what to try next?

I still do not know, how to solve this issue :⁠-⁠\

Hi,

I’ve flagged this for more expert eyes. Hopefully they’ll be along soon.

 
Ann

1 Like

I wanted to gently remind about this issue. Your insights are essential to my progress, and I’m eager to incorporate them. Could some expert eyes please have a look and maybe provide some solution proposal?

Thank you very much.

Petr

1 Like