We have recently started working with Sonarcloud, and so far the tool looks great. We have encountered a limitation that we would like to configure, but we found no way of doing so.
We are scanning a Pull Request that has a very few lines changed (13). The code coverage is around 61.5 and our Quality Gate defines that at least 80% is required.
When the PR is scanned, we get the following message:
“Some Quality Gate conditions on New Code were ignored because of the small number of New Lines”
If you hover over the question mark it says: At the start of a new code period, if very few lines have been added or modified, it might be difficult to reach the desired level of code coverage or duplications. To prevent Quality Gate failure when there's little that can be done about it, Quality Gate conditions about duplications in new code and coverage on new code are ignored until the number of new lines is at least 20.
We understand that this can be a pain under some circumstances, but we also believe that we should have the power to enable or disable this behavior. Is there anywhere in the administration options of Sonarcloud a place to configure this? (Either by fully enabling or disabling it, or by tweaking the number of lines required)
I confirm that the number of lines threshold in the changeset (19), and the set of metrics to ignore when below the threshold are currently hardcoded, therefore not possible to customize, yet.
We have to choose our battles. Since this is not a feature in high demand, I’m afraid it’s not likely that we will find time to do this anytime soon.
I also want this. In a few years of using sonarcloud, I dont believe I’ve ever seen this message or anything allowed through the quality gate that was below the coverage level. Is there something that influences when this is evaluated? All our new code is set to 30 days, so its not got a defined start and end date or version or anything.
I came across this rule recently and was also surprised. I believe there are cases where quality gate metrics don’t make sense but these should be treated on a case by case basis rather than applying a blanket rule. This rule means small pull requests have the ability to lower quality overall and we should always be trying to do the opposite.
It can also cause blockers with the way many teams work, where builds from master branch are less frequent. Several smaller pull requests might be merged and the builds will pass individually. But if the accumulated number of lines is above 20 they could block the main branch builds.
I would also love to see this. We endeavour to break down work so that PRs are small, which means that the majority of our PRs aren’t being properly gated.
As Ryan mentioned we also strive to keep pr’s as small as possible, in the end this can really lowers our code coverage.
I also believe that quick/small changes are most the common cause of bugs and this rule really promotes that.
ps it is even stranger that this is not mentioned anywhere in the Quality Gates itself.
This is something that is already planned on our side and will be worked on in the short term. You can subscribe to this card on our roadmap to get any updates on the topic.
