which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
Sonarqube Scanner CLI: 4.7.0.2747
what are you trying to achieve
We want our Pull Request quality gates to work based on overall code coverage conditions and not just new code conditions. Its clear from the Sonar documentation and what we see posted on our Bitbucket PRs, that the coverage on new code is only taken into account when passing or failing that gate. It would be great if projects are able to choose if the PR quality gate should work based on new code (or) overall code (or) both conditions. This would give the projects owners the flexibility to choose the conditions/parameters based on their platforms. For our mobile platform, the “new code” conditions does not make sense on many types of PRs. But in all cases, we want to ensure that “overall code” conditions are met and improved upon(Ratcheting coverage) on every PR.
what have you tried so far to achieve this
Nothing really to try. I have seen this request raised in multiple threads with really no way to achieve this currently with Sonar. Can we please add this as a feature request?
Sure. Situations when we update code which is not testable(e.g. due to runtime dependencies) or non-code resources like assets, string tables etc. Instead of the PR reviewer needing to make this call, using the overall code coverage metric on the quality gate ensures that its not a subjective call to ignore the quality gate when it fails during these exception scenarios.
Using the exclusions is not a solution for what I am looking for. For us, the overall code coverage metric is more important than just new code coverage numbers. And we want to gate our PRs based on overall code coverage. Hence the ask.
@ganncamp Is this something you could discuss with your product team and give us an update. If we cannot get the overall code coverage metric to be gate on our PR’s, we will have to disable that and add another process to report overall coverage on the PRs. I understand the “focus on new code” philosophy that Sonar recommends. However, as a tool, I think there should be options for the customer teams to cater to their needs, if they want to do something different. Overall code coverage is measured on all PRs as well, is it not? Can we not have the option to have that metric on the gate instead of new code? Thanks.
Bumping this thread. We have a need for this feature and I believe one of the feature request boards has this significantly upvoted. I am not having much luck finding it but I remember seeing that recently.
Here is an example of a use case. We’re scanning for CVEs using OWASP dependency check and leveraging the plugin to identify upstream security issues in our dependencies. Because of how the plugin works, the issues are not flagged in PRs. They appear in branch scans post integration which is obviously less than optimal. PR level feedback is typically a very effective way of addressing the issue and preventing a security risk from entering a trunk branch or potentially be released, if the team practices CD. PR scans focus on new code changes and while that works well for may scenarios, it is inadequate for a more sophisticated testing strategy. If we had an ability to apply a project level rule to a PR, we could easily craft a rule that would detect the problem and be a forcing function to address the issue before it is introduced.
I recognize that one could craft processes to work around this limitation, for example, scheduling reviews or reworking workflows towards branch scans, but those approaches are inefficient. It would be great if SQ could extend its feature set and provide more flexibility in this area.
@ganncamp Would it be possible to check if this is something in the cards? If I know that Sonar is never going to ever consider providing the overall coverage as an option for quality gates, we will have to look for a different option. Thanks.
@vicsonvictor I confirm that a portal card exists here that, I feel, corresponds to the need behind your request.
We are actively monitoring the various threads on the topic and seeking to better identify the use cases for looking at the overall code so that we can bring an appropriate answer to these needs. Indeed, we recommend focusing on the code you’re working on for cleaning your code, but are conscious that many users have other needs that may need a dedicated solution.
@Chris@ganncamp Any updates on this? Would love to have the option of setting the quality gate to the overall code threshold and not just new code. Thanks.
We recently had an issue where a focused test slipped through a PR and meant that only that one test ran. What else can happen is someone might mess up test configuration so that a lot of tests that are only on existing code don’t run. We have an estimated merge coverage on the reports, why can’t we just use it?
@QuarkNerd Our PR/CI builds are configured to run all the tests(unit and UI) to get complete coverage numbers, so they can be compared against the overall coverage on the master branch. We do not have a possibility to “mess up” test configuration. Moreover, I do not see a way to even use “estimated merge coverage” as the metric for the Sonar Quality gate.
I am asking for the same thing you are. I was talking about how I can see the estimated code_coverage in my reports for PRs, so it shouldn’t be too cumbersome for them to add the option for us to have a gate.
Would love to have the option of setting the quality gate to the overall code threshold and not just new code. Thanks.