Gradle plugin does not tolerate funny Surefire output for JUnit5 Dynamic Tests

Versions:

  • Gradle Plugin 2.6.2
  • SonarQube 6.7.4
  • JDK 1.8.0_66
  • Gradle 4.8.1
  • JUnit 5.2

Error observed:

When running the sonarqube gradle task, the SurefireSensor reports an exception:

gradlew sonarqube --stacktrace --info -Dsonar.host.url=XXXX -Dsonar.login=XXXX -Dsonar.branch=master

java.lang.StringIndexOutOfBoundsException: String index out of range: -1
	at org.sonar.plugins.surefire.data.SurefireStaxHandler.getClassname(SurefireStaxHandler.java:68)
at org.sonar.plugins.surefire.data.SurefireStaxHandler.parseTestCase(SurefireStaxHandler.java:58)
at org.sonar.plugins.surefire.data.SurefireStaxHandler.stream(SurefireStaxHandler.java:50)
at org.sonar.plugins.surefire.StaxParser.parse(StaxParser.java:63)
at org.sonar.plugins.surefire.StaxParser.parse(StaxParser.java:55)
at org.sonar.plugins.surefire.SurefireJavaParser.parseFiles(SurefireJavaParser.java:111)
at org.sonar.plugins.surefire.SurefireJavaParser.parseFiles(SurefireJavaParser.java:102)
at org.sonar.plugins.surefire.SurefireJavaParser.collect(SurefireJavaParser.java:67)
at org.sonar.plugins.surefire.SurefireSensor.collect(SurefireSensor.java:65)
at org.sonar.plugins.surefire.SurefireSensor.execute(SurefireSensor.java:60)
at org.sonar.scanner.sensor.SensorWrapper.analyse(SensorWrapper.java:53)
at org.sonar.scanner.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:88)
at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:82)
at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:68)
at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:88)
at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:177)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:291)
at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:286)
at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:264)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:48)
at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:84)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:121)
at org.sonar.batch.bootstrapper.Batch.doExecuteTask(Batch.java:116)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:111)
at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:63)
at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
at com.sun.proxy.$Proxy170.execute(Unknown Source)
at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:233)
at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:151)
at org.sonarqube.gradle.SonarQubeTask.run(SonarQubeTask.java:99)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:46)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:39)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:26)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:780)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:747)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:121)
at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:110)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:92)
... 29 more

The problem seems to be caused by the XML produced by Surefire, but the SonarQube be tolerant of this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name=")" tests="1" skipped="0" failures="0" errors="0" timestamp="2018-07-12T13:52:50" hostname="NB099" time="0.005">
  <properties/>
  <testcase name="Test1(dynamicTest" classname=")" time="0.005"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

Steps to reproduce
Run SonarQube-plugin against a code base with JUnit5-Dynamic Tests.
I have created a little demo project on GitHub.

Workarounds
Annotating all dynamic test methods with @DisplayName("_") or similar fixes the problem.

PS
There is a similar problem reported on Stackoverflow regarding the Maven plugin and this discussion on the JUnit5 Issue tracker.

Thanks a lot for the reproducer : ticket created to handle the issue https://jira.sonarsource.com/browse/SONARJAVA-2931