Rule java:S2699 (Test should include assertion) should not be activated for disabled test

Clean Code Report False-positive / False-negative...
, ,
1
  • Language: java
  • Rule: java:S2699 (Test should include assertion)
  • Using: SonarCloud & SonarLint (8.0.0.63273 for IntelliJ 2022.3.2)
  • Testing framework: Junit5

We are developing a software that can be used with several databases, provided that the connector for said database (which is developed by us) is compatible with our software.
In order to ensure that, we have at the core of our tests a TCK (technology compatibility kit) that represents the full test suite that a particular implementation must pass.

Unfortunately, sometimes an implementation cannot pass a given test due to some limitation. For use it is easy to spot as the code would have the following pattern:

public class TestsForDatabase1 extends ADatabaseTck {
  @Test
  @Disabled("Not supported by database 1, see ticket ...)
  @Override
  public void testSomeFunctionality() {
    super.testSomeFunctionality()
  }
}

However, this pattern does not play well with Sonar, as it triggers the rule java:S2699 for a disabled test.

In this case (when the test is disabled), in my own opinion, the warning is a false-positive.

4

Hello @NathanEckert,

Thanks for the message. I am not sure I completely understand the problem here.

Afaik, This rule says that your test contains no assertions. Which is correct, as it doesn’t. So I see no false positive here. And the fact that the test is disabled has nothing to do with the content of the test.

As far as I can understand you can’t test unsupported functionality that’s why you disabled the test. But if in the future the functionality will be supported, will you just remove an annotation or rewrite the test? In the first case, you will still have an issue, in the second case if you add any assertion methods, why couldn’t you do it right now in a disabled test?

Also, if method testSomeFunctionality() actually contains some assertions, but the issue is reported, you could add your custom assertion patterns into rule property customAssertionMethods.

Regards,
Margarita

6

Hi,

Thanks for your answer.

That is not totally true that the test does not contain any assertions, as super().someFunctionality() does contain assertions.
I know I could use the property customAssertionMethods of Sonar, but that would be cumbersome, and could be detected systematically knowing that the super() call is the standard pattern for the implementation of disabled tests (it is also the default implementation when overriding a method with IntelliJ shortcuts).

To your question about what would happen in the future if the functionality was supported: i wouldn’t add asserts to the code or remove the @Disabled annotation, I would remove all together the overridden tests, which would then be run each time as it would picked up by Junit5 by collecting the parent class.

I totally agree that the fact that the test does not contain assertions and the @Disabled annotations are not correlated in general, except in this exact case for which you disable a test defined in a parent component (which does contain assertions) because it cannot pass for that specific implementation.

Regards,

7

Thanks, now I got you.

The problem is actually your test does contain assertions in the someFunctionality() method. However, currently, the rule is not smart enough to visit the super method and detect that it actually contains assertions.

Unfortunately, this is currently the known limitation of the analyzer. I noted the need on our side and maybe we can come up with a better solution, for now, you can mark these issues as “FP” or “Won’t fix”.

Regards,
Margarita