Unable to get report from Jacoco Gradle Kotlin project on SonarCloud

[LBeghini (Lavínia Beghini)]

Right, so I have a Kotlin Gradle project with the framework SpringBoot. The project is on GitHub and our CI system is Circle CI.

At the build.gradle.kts file we have the following configs:

sonarqube {
    properties {
        property("sonar.projectKey", "project_key")
        property("sonar.organization", "organization")
        property("sonar.host.url", "https://sonarcloud.io")
        property(
            "sonar.coverage.jacoco.xmlReportPaths",
            "$buildDir/reports/jacoco/jacocoTestReport/jacocoTestReport.xml"
        )
    }
}

fun ignorePackagesInJacocoReport(classDirectories: ConfigurableFileCollection) {
    classDirectories.setFrom(
        files(
            classDirectories.files.map {
                fileTree(it).apply {
                    exclude(
                        "**/company/**/*.java",
                        "**/company/**/*.kts",
                        "**/company/commons/**",
                        "**/company/**/config/**",
                        "**/company**/configuration/**",
                        "**/company/**/requests/**",
                        "**/company/**/responses/**",
                        "**/company/**/dto/**",
                        "**/company/**/infrastructure/scripts/**"
                    )
                }
            }
        )
    )
}


tasks.jacocoTestReport {
    sourceSets(sourceSets.main.get())
    executionData(fileTree(project.rootDir.absolutePath).include("**/build/jacoco/*.exec"))

    reports {
        xml.required.set(true)
        html.required.set(true)
        xml.outputLocation.set(File("$buildDir/reports/jacoco/jacocoTestReport/jacocoTestReport.xml"))
        html.outputLocation.set(layout.buildDirectory.dir("$buildDir/reports/jacoco"))
    }

    ignorePackagesInJacocoReport(classDirectories)
}

Our config.yml used by Circle CI is like the following:

version: 2.1

orbs:
  docker: circleci/docker@0.5.13

jobs:
  unit_test:
   
    steps:
      - run:
          name: Run tests
          command: gradle test -i

      - run:
          name: Save test results
          command: |
            mkdir -p ~/junit/
            find . -type f -regex ".*/build/test-results/.*xml"
            find . -type f -regex ".*/build/test-results/.*xml" -exec cp {} ~/junit/ \;
          when: always

      - persist_to_workspace:
          root: ~/
          paths:
            - project-folder/build/jacoco

  coverage:
    steps:
      - run: ./gradlew dependencies build -x test -x detekt
      - run:
          name: Run coverage report
          command: ./gradlew jacocoTestReport

      - persist_to_workspace:
          root: ~/
          paths:
            - recupera-gateway/build/reports/jacoco

  sonarqube:
    steps:
      - restore_cache:
          keys:
            - app-dependencies-{{ checksum "build.gradle.kts" }}

      - run: ./gradlew dependencies build -x test -x detekt

      - save_cache:
          paths:
            - ~/.gradle
          key: app-dependencies-{{ checksum "build.gradle.kts" }}

      - attach_workspace:
          at: ~/

      - run:
          name: Run sonar analysis
          command: ./gradlew sonarqube


workflows:
  version: 2.1
      - unit_test
      - coverage:
          requires:
            - unit_test
      - sonarqube:
          requires:
            - coverage

Locally, running:

./gradlew jacocoTestReport

returns:

However, on SonarCloud, is the following:

Note that excluded files in Jacoco are present, and the total coverage is completely different.
In SonarCloud General Settings, I am passing JacocoTestReport.xml path to the file:

So, first, I have a question:
Does SonarCloud have the capacity to calculate the coverage by itself? Because it looks like it is taking the coverage results from somewhere else.

And second:
How do I make Jacoco’s Coverage Report match with SonarCloud’s?

Note that I have omitted many information for security and privacy reasons. If any crucial information is missing, please ask and I will see if it can be provided. Also, it might sound like a duplicate, however other topics found does not have a solution for the problem.

[Colin]

Hey there.

In this image, you are specifically seeing the calculation of coverage on new code (code that has changed in the New Code Period).

@LBeghini

image1587×737 101 KB

If you scroll down to “Overall Code” – are the numbers more aligned to your expectations?

[LBeghini (Lavínia Beghini)]

Indeed, well noticed
I will have a second look on this. But actually, our biggest problem is about the files that are not being ignored

[Colin]

Unfortunately, SonarCloud won’t take into account the exclusions you’ve set for JaCoCo – you’ll need to repeat them for sonar.coverage.exclusions (either as an analysis parameter passed to the scanner, or in the SonarCloud UI).

SonarCloud assumes that if a file is entirely absent from the coverage report – it can be covered by tests, but no coverage report has been provided, so it assumes 0% coverage.

[LBeghini (Lavínia Beghini)]

Uau, thank you very much! Just as a curiosity, is there anywhere on the official documentation where I can find this information? I spent an entire week trying to make the ignored files in Jacoco to be reflected on SonarCloud

[Colin]

Thanks for the feedback. Maybe we have been assuming that since the documentation on Analysis Scope doesn’t imply any other way of excluding files from analysis (other than the documented means of restricting the scope of coverage detection)… people won’t think there are other ways. And… I get that’s a lot of assuming.

I’ll ping this over to our docs team to see if we can be more explicit about the behavior somewhere.

[LBeghini (Lavínia Beghini)]

Awesome!!
We are now ignoring the files on SonarCloud as well, but still the coverage from Jacoco is different than SonarCloud’s.

For example, see this file has 75% coverage on SonarCloud:

But on Jacoco, the same file is 64%:

Even presenting the same line as not covered:

And the question remains:
Does SonarCloud have the capacity to calculate the coverage by itself? Because it looks like it is taking the coverage results from somewhere else.

[LBeghini (Lavínia Beghini)]

Should I open another question to address the problem of not matching the coverage between SonarCloud and local Jacoco? Because the topic is partially answered, maybe I can be more specific in another one.

[Colin]

Hey there.

This thread is fine – but a little patience is appreciated, since you only responded yesterday.

To me – the coverage report and SonarCloud match what I would expect. It shows 3/4 of lines covered (75% of lines covered) and no branches (no if or switch statements), and this aligns with how SonarCloud calculates coverage, a mixture of branch and line coverage.

The “64%” you’re seeing is instruction coverage – which isn’t represented in SonarQube. While a good indicator (independent of source code formatting) of how much code is covered, it takes significantly more work to make sense of these numbers and how you should change your code/tests as a result.

So SonarQube isn’t determining coverage on its own (it relies on the report), it just doesn’t consider instruction coverage. You might find this StackOverflow post interesting