- which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
Sonarqube Scanner CLI: 220.127.116.1147
- what are you trying to achieve
We want our Pull Request quality gates to work based on overall code coverage conditions and not just new code conditions. Its clear from the Sonar documentation and what we see posted on our Bitbucket PRs, that the coverage on new code is only taken into account when passing or failing that gate. It would be great if projects are able to choose if the PR quality gate should work based on new code (or) overall code (or) both conditions. This would give the projects owners the flexibility to choose the conditions/parameters based on their platforms. For our mobile platform, the “new code” conditions does not make sense on many types of PRs. But in all cases, we want to ensure that “overall code” conditions are met and improved upon(Ratcheting coverage) on every PR.
- what have you tried so far to achieve this
Nothing really to try. I have seen this request raised in multiple threads with really no way to achieve this currently with Sonar. Can we please add this as a feature request?
I’d like to understand why. Would you mind explaining?
Sure. Situations when we update code which is not testable(e.g. due to runtime dependencies) or non-code resources like assets, string tables etc. Instead of the PR reviewer needing to make this call, using the overall code coverage metric on the quality gate ensures that its not a subjective call to ignore the quality gate when it fails during these exception scenarios.
Have you considered excluding these files from coverage altogether? IMO this isn’t a PR issue, but a general configuration question.
Do you agree, or do you still see a need for this feature?
Using the exclusions is not a solution for what I am looking for. For us, the overall code coverage metric is more important than just new code coverage numbers. And we want to gate our PRs based on overall code coverage. Hence the ask.
@ganncamp Is this something you could discuss with your product team and give us an update. If we cannot get the overall code coverage metric to be gate on our PR’s, we will have to disable that and add another process to report overall coverage on the PRs. I understand the “focus on new code” philosophy that Sonar recommends. However, as a tool, I think there should be options for the customer teams to cater to their needs, if they want to do something different. Overall code coverage is measured on all PRs as well, is it not? Can we not have the option to have that metric on the gate instead of new code? Thanks.
You shouldn’t count on any changes to this in the short term.