I read Why are so many rules inactive by default? so I am aware that there is a rationale for not having all rules active by default.
However, I am trying to understand if there was a significant change to the number of active rules in the Sonarway Java QP as of release 8.6.0, as the number of detected issues dropped after upgrading.
For our project, the change in the number of reported issues (by type):
Bugs: down 18%
Vulnerabilities: down 69%
Code Smells: down 65%
The change in vulnerabilities is the most concerning but the comments I make below apply to other types as well. We happened to have in flight changes to remediate rule java:S2384, which was a source of much of the “improvement” since it is now inactive in the QP (so our in progress PR doesn’t actually improve the measured code quality now).
I don’t see how one can justify having ready (non deprecated) vulnerability rules that are inactive. I see that there only seem to be 6 of them, but they have tags that mark them as part of owasp top 10, etc. If they are really not useful for most projects, should they still be tagged like that? Or considered to be ready?
Perhaps these changes were made to lower the number of reported vulnerabilities, but if that is the goal, why not simply have two (or more) QPs. “Lenient Sonarway” and “Strict Sonarway”. For my company, I would rather see more vulnerabilities reported and then we can selectively deactivate ones that we do not feel make sense in our organization than the other way around (our QP extends Sonarway). I imagine others are of the same mindset.
Now, after every upgrade, it seems that I will have to go in and check which rules have been made inactive to the builtin QPs. Maybe the release notes can publish the list of such changes so that it makes it easier to understand the impact of upgrading?