Inconsistent results for rules (java:S1161, java:S5803) on simple, valid violations

Hello SonarQube Community,

I am experiencing a strange and inconsistent behavior with some Java rules. The rules only flag a random subset of applicable classes, and I’m having trouble understanding why.

Minimal reproducible example (for java:S1161)

To demonstrate the issue, I have a very simple set of classes. All classes are in the same package.

Base class:

package com.company.app;

public class BaseClass {
    public void doSomething() {
        System.out.println("Doing something in BaseClass");
    }
}

Subclasses:

package com.company.app;

// SubClassOne.java
public class SubClassOne extends BaseClass {
    public void doSomething() { // <-- Should be flagged by S1161
        System.out.println("Doing something in SubClassOne");
    }
}
package com.company.app;

// SubClassTwo.java
public class SubClassTwo extends BaseClass {
    public void doSomething() { // <-- Should be flagged by S1161
        System.out.println("Doing something in SubClassTwo");
    }
}
package com.company.app;

// SubClassThree.java
public class SubClassThree extends BaseClass {
    public void doSomething() { // <-- Should be flagged by S1161
        System.out.println("Doing something in SubClassThree");
    }
}
package com.company.app;

// SubClassFour.java
public class SubClassFour extends BaseClass {
    public void doSomething() { // <-- Should be flagged by S1161
        System.out.println("Doing something in SubClassFour");
    }
}

Observed behavior

I would expect SonarQube to raise a code smell for rule java:S1161 ("@Override" should be used... ) on all four subclasses, since they all override the doSomething() method without using the @Override annotation.

However, the actual behavior is random. On any given scan, SonarQube will report the issue on a different subset of the files.

  • Sometimes it reports on SubClassOne and SubClassThree .
  • Sometimes it only reports on SubClassFour .
  • Sometimes it reports on three of the four classes.

The results are not consistent between scans, even when the code has not changed.

I have also noticed similar inconsistent behavior for other rules. For example, java:S5803 (Class members annotated with "@VisibleForTesting" should not be accessed from production code ). In our codebase, we have several clear violations of this rule, but the scanner only reports the issue in some places, while ignoring other identical violations.

Configuration and environment

  • SonarQube Version: Enterprise Edition v2025.1.3
  • SonarScanner Version: SonarScanner CLI 7.0.2.4839
  • Environment: This issue occurs both when running the scanner on my local machine and in our CI environment.

Crucially, the scanner logs show no warnings or errors. I have run the analysis in debug mode and have confirmed that there are no messages about missing .class files or an incomplete classpath. The scanner appears to be finding all source and binary files correctly.

I have tried everything I can think of to debug this, but the randomness of the issue is what I can’t explain. Has anyone encountered this kind of inconsistent behavior before? Any ideas on what could be causing the Java analyzer to only partially detect these clear violations when the logs appear clean?

Hi @marko_domic,

Sorry for the late reply. Can you answer a couple of questions to help guide the investigation:

  1. Are the classes that are inconsistently reported on by java:S1161 in the same java package?
  2. Have you considered using the Gradle or Maven scanners for your project? They tend to provide findings that are a lot more consistent and stable. If that is not option, we can also look into stabilizing the analysis for consistency

With these questions clarified, we should be able to make some progress on resolving the issue.

Cheers,

Dorian

Hi @Dorian_Burihabwa

  1. Yes, they are. They are all in the same package.
  2. Unfortunately, that is not an option. We are using Bazel as main build tool, for very large monorepo architecture.

Hi again,

Too bad you cannot use these scanners but here are a few things that might help.

I will work under the assumption that:

  1. The code has been compiled and the location of the .class files has been fed into sonar.java.binaries
  2. The java language version has been documented using sonar.java.source

If both of these things are managed then I suggest adding the option sonar.java.fileByFile=true to your analysis.

By default, the java analyzer parses files in batches with the benefit of having most files within the same package resolving shared constants and references at the same time providing better insights to the checks. But to rule out that the batches are not built in an inconsistent way across analyses, let’s try and analyze file by file.

My expectation is that you will have less findings but more consistent results across analyses.

Let us know how that works for you and we will see how we can improve from there.

Cheers,

Dorian

Hi @Dorian_Burihabwa

Thanks for the feedback, and my apologies for the delayed response.

I’ve tried your proposal, but unfortunately, it didn’t resolve the issue. To make it easier to investigate, I’ve created a minimal, reproducible example in a dedicated GitHub repository.

Steps to reproduce

  1. Clone the repository.
  2. Install Bazel (if not already installed).
  3. Build the project by running: bazel build //…
  4. Run a local SonarQube instance and create a new project with the key test-project, using the default “Sonar way” Java quality profile.

Analysis

I’ve discovered that the issue’s visibility depends entirely on the value of the sonar.sources property.

Scenario 1: Correct behaviour (narrow sonar.sources path)

When I run the scanner with a specific path, the analysis works as expected.

Command:

sonar-scanner \
  -Dsonar.projectKey=test-project \
  -Dsonar.sources="src/main/java/com/example/issue" \
  -Dsonar.java.binaries="./bazel-bin" \
  -Dsonar.java.libraries="./bazel-bin/external/**/*.jar" \
  -Dsonar.java.source=17 \
  -Dsonar.host.url="http://localhost:9000" \
  -Dsonar.token=<your-token>

Result: Issues for rule java:S1161 are correctly reported for the following files:

  • src/main/java/com/example/issue/SubClassOne.java
  • src/main/java/com/example/issue/SubClassTwo.java
  • src/main/java/com/example/issue/SubClassThree.java
  • src/main/java/com/example/issue/SubClassFour.java

Scenario 2: Incorrect behaviour (broad sonar.sources path)

However, when I broaden the sonar.sources path to the parent src directory, the issues disappear.

Command:

sonar-scanner \
  -Dsonar.projectKey=test-project \
  -Dsonar.sources="src" \
  -Dsonar.java.binaries="./bazel-bin" \
  -Dsonar.java.libraries="./bazel-bin/external/**/*.jar" \
  -Dsonar.java.source=17 \
  -Dsonar.host.url="http://localhost:9000" \
  -Dsonar.token=<your-token>

Result: The previously mentioned issues are no longer reported. These appear to be false negatives.

The question

Why does changing sonar.sources from a specific subdirectory to its parent directory cause existing issues to be missed?

I have confirmed this behavior across multiple SonarQube instances, which leads me to believe it might be related to how the SonarScanner handles the analysis scope. Could you clarify if this is a configuration error on my part or a potential issue with the scanner?

Thanks for your help.