Number of unit tests is not measured with nested junit 5 tests

SonarQube Server / Community Build
, , ,
1

Hello everyone

Versions

SonarQube: 6.7.5 lts
Scanner for gradle: 2.6.2
Plugin: SonarJava 5.8

sonarqube gradle configuration:

sonarqube {
   properties {
      property "sonar.projectName", "projectName"
      property "sonar.projectKey", "projectKey"
      property "sonar.host.url", "host"
      property "sonar.scm.disabled", "true"
      property "sonar.java.coveragePlugin", "jacoco"
      property "sonar.junit.reportPaths", ["build/test-results/test"]
      property "sonar.jacoco.reportPaths", "build/reports/jacoco/test.exec"
   }
}

what are you trying to achieve

We are trying to measure the number of executed junit5 tests. Where most of the tests reside in a nested non-static inner class.

@DisplayName("X")
@Tag("x")
class XTest {

  @DisplayName("y")
  @Nested
  class MethodYTest {

    @DisplayName("z")
    @Test
    void shouldZ() {...}
    ...

  }
}

Only one test is found. This test does not use a nested class. Therefore we assume the configuration must be correct.
Also the coverage is gathered successfully.

What is different from the case with nested tests is the naming of the generated report xml-files (and ofc the content).
(The results are taken from different test cases and anonymized in the same manner)

/src/project/sub1/build/test-results/test/TEST-package.XTest.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="package.XTest" tests="1" skipped="0" failures="0" errors="0" timestamp="time" hostname="host" time="ms">
  <properties/>
  <testcase name="shouldZ()" classname="package.XTest" time="0.001"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

vs

/src/project/sub1/build/test-results/test/TEST-package.XTest$YTest.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="package.XTest$YTest" tests="5" skipped="0" failures="0" errors="0" timestamp="time" hostname="host" time="ms">
  <properties/>
  <testcase name="shouldZ()" classname="packageXTest$YTest" time="ms"/>
  <testcase name="shouldZ1()" classname="package.XTest$YTest" time="ms"/>
  <testcase name="shouldZ2()" classname="package.XTest$YTest" time="ms"/>
  <testcase name="shouldZ3()" classname="package.XTest$YTest" time="ms"/>
  <testcase name="shouldZ4()" classname="package.XTest$YTest" time="ms"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

what have you tried so far to achieve this

. played around with the property “sonar.junit.reportPaths” and the deprecated option (reportPaths)

Has anyone a clue what might be the issue here?

I’ll gladly provide more information if needed.

Regards
Florian

2

Hi Florian,

SonarQube analysis just reads your reports, so if the data isn’t want you expect, you should take a look at the data source.

 
Ann

3

The reason why this happens lies here:

Testsuits of nested classes are ignored.
What i don’t understand is why is this the case?

Writing unit tests like this is becoming increasingly popular:

5

Hello @ftan and @kwuerl,

First, let me apologies for the (very long) time it took us to get back to the two of you about this. What you describe here is indeed a limitation of SonarJava, which never properly handled nested test classes. Thanks a lot for the detailed message and reproducer @ftan.

I created the following JIRA ticket to document the issue and track its resolution: SONARJAVA-3222

I’ll also handle @kwuerl’s contribution to SonarJava, which successfully fixes the issue. Thanks a lot for your work, it is really appreciated, even a few months after. Your fix will be part of next released version of SonarJava, which will be version 6.0.

Cheers,
Michael

1 Like
6

Hi @Michael

May I also asks your assistance

I also encountered the same issue with:
sonarqube 7.9 lts
gradle sonarqube plugin 3.0
and default gradle sonarqube properties

I tried to move out the tests from @nested inner class and into the parent class and the correct number of tests is displayed in the sonarqube ui.
Is there something missing in my configuration?

Best regards