We are using SonarQube 7.9.1. We just went through a major upgrade from version 6.7.6.
In 6.7.6, we were using old-style branches, because new-style branches didn’t support the features we needed. In 7.9.1, we apparently now don’t have a choice, but I’ve been assured that 7.9.1 provides the features we need in the new-style scans.
After starting to use it, we’re discovering that scans of pull request branches don’t appear to be respecting the quality gate set in the base project. The quality gate clearly shows a required 80% test coverage, and I created a pull request that clearly shows the resulting coverage after merge would be well less than 80%, and the little box at the top left, underneath “Quality Gate”, says “Passed”.
Is it possible that we missed something in our configuration?
So you’re telling me that for short-lived branches, the quality gate enforces the code coverage percentage limit against the “Coverage on N New Lines to cover” percentage, instead of on the “Estimated after merge” percentage? There’s no configuration option that lets us apply it to the latter instead? I’m going to guess your answers are “yes” and “no”.
My test case adds an @Ignore annotation to a unit test class, which doesn’t add any new lines of code. This passes the quality gate, and results in a broken build after merge.
I can easily see other test cases that add a small number of new lines that are not covered by tests, which would fail the quality gate, even though the coverage after merge would still pass.
This is not what we need. It is critical for us that the scan prevents developers from breaking the build after merge. After this upgrade, we have now lost that ability.
I see! Adding that one “non-code” line means your coverage reports… no longer reflect what that test covers(?) and so you actively decrease your coverage with no reflection of that in your PR/SLB.
Unfortunately, PRs/SLBs are only going to reflect changes to updated code, so even if you deleted the test altogether they wouldn’t currently show the decrease in coverage because the now-uncovered lines aren’t “new”.
However 8.1, E.T.A. next week, drops the concept of SLB so that you have just “Branches” (which all act like LLBs) and PRs. This means that you get the “full” treatment on them, and this will probably do what you want. I believe we’ve worked hard on the migration to make sure your previously-SLBs still get cleaned up as expected. So… I urge you to test (and adopt!) the new release as soon as it’s available.
Changing tack a bit, can you tell me whether the value of “coverage after merge” from the scan is available through the REST api? I’ve explored and used the REST api for other things. I can see that I can get the details of the quality gate, including what quality gate is set for each project, and I can get some of the details of the results of the last scan, but from a quick look it’s not clear whether I can extract that resulting “coverage after merge” value. Can you detail the path to that value in the api? If I can get this information, I can restore the functionality that we had before this upgrade.
Thanks for the reminder about that. I managed to get the required info before I saw this.
I’ve managed to implement a “post-scan” block that fails the build if the resulting coverage violates the “coverage” threshold. The SonarQube info will still say the QGs passed, but I can’t help that now.
Hopefully this will get back to a more informative view when we eventually upgrade to 8.x, as you mentioned.
I wanted to pop this back up because I’m definitely running into a similar issue.
Running: SonarQube Enterprise 8.6
Problem: Deleting a unittest does not fail the Quality Gate set
Expected Result: The decrease in Coverage for the project should fail the Quality Gate
Actual Result: Deleted lines are not seen as “New” or “Changed”
Steps to reproduce:
Make sure you have a current analysis of the “Main” branch, in my case “develop” that shows the current amount of code coverage
Set the project “New Code” setting to use the “develop” branch as the reference branch. And then you have to set the “develop” branch “New Code” to “Previous Version”
Create a new branch
Delete a unittest
Commit the change, push to origin and create a PR
Once the analysis is done, all I get is an “Estimated coverage after merge” which is WAY below the reference branch coverage and that there are no new lines of code and the Quality Gate passes.
This unfortunately doesn’t work for my use case, as this clearly shows it will not capture events where a test is removed or ignored. Now, I do have an Overall Condition on the Quality Gate, and the branch that gets analyzed does fail that check. So I guess I have to script around this during my build and fail that instead.
While I have seen that issue crop up as well in my testing of the enterprise version, what I am actually seeing is that deleted lines of code from a test file that exceed the minimum will drop the coverage, only after merge which means that the QG passes.
For example, the develop branch has nearly 88% code coverage, so I create a branch off of that and delete 27 lines of test coverage. The analysis then says I would have A LOT less coverage, but on no new lines. Here’s a screenshot
Clicking through the “0 New Lines” it does show the file that got changed, but doesn’t seem to realize that lines were deleted… which should be a change. Does Sonar not think that deleted lines are “changed” or “new”?
Condition coverage on new code ( new_branch_coverage )
Identical to Condition coverage but restricted to new / updated source code.
So from reading that it should work as you expected.
Because of that i supposed it may be related to the problem with the ‘fudge factor’ mentioned in the linked Jira ticket.
… we began ignoring QG conditions on coverage and duplications on new code when there are fewer than 20 new lines.
If deleted code is also treated as new code, maybe your changes are < 20 lines ?
Actually, @Ryan_Shatford (Welcome to the community!) is right. We don’t handle deleted lines well (at all). And we’re particularly not handling nicely the deletion of tests. We focus on lines of code that were added or changed in the PR and in this case there are none. So yes, after merge your “passing” PR will drop overall coverage.
I’ve flagged this for attention internally. And I can’t promise that there will be action any time soon.
As I expected, this is possibly a breaking change in how we utilize the community edition today versus us moving to the Enterprise version immediately. Please, when and if you have an external ticket to track this work, please update this thread.
I guess in Community Edition you’re analyzing each branch/PR as a new project & comparing manually? If so, I’m going to repeat the advice I gave earlier that using branch analysis may be what you need.
