Pull Request analysis frequently falsely marks issues as resolved or misses issues

When a pull request makes a change to a file, that file is analyzed by the gradle analyzer. However, it does not do a full project analysis due to
-Dsonar.pullrequest.key=${{github.event.pull_request.number}} -Dsonar.pullrequest.branch=${{github.head_ref}} -Dsonar.pullrequest.base=${{github.base_ref}} and so it cannot correctly analyze all issues.

A common false negative is Nullness of parameters should be guaranteed java:S4449. A method in a class not changed by the PR is annotated @NonNull. The changed class passes null to that method. The issue is marked as resolved in the PR. After merging, the issue is reopened.

I think this should be considered a bug. A PR analysis should not give its opinion on issues which it should know it does not have the full resources to analyze. It should either actually load and check the target method to see how its annotated, or just abstain from saying whether or not the issue is fixed.

Sometimes it also misses obvious issues that it should definitely be able to see. For example, a PR which added this line:

.value(item.getValue().map(value -> WebAmountMapper.map(value)))

Should have raised Lambdas should be replaced with method references java:S1612. The correct code is

.value(item.getValue().map(WebAmountMapper::map))

And I can think of no reason why the limited PR analyzer would not be able to see that.

We have a quality gate that enforces no new issues and yet we are constantly adding new issues because they aren’t found until after they are merged.

Hi,

Unfortunately, this is a known issue, and one we’ve stubbed our toes on internally as well. Also unfortunately, this isn’t an easy problem (or it would have been fixed by now).

I don’t have an E.T.A. or any other good news for you. But I’ll raise this (again) internally so that we continue to track the “traction” on the topic. Hopefully there will be enough traction soon.

 
Ann