Scans of new-style branches no longer fail quality gates

We are using SonarQube 7.9.1. We just went through a major upgrade from version 6.7.6.

In 6.7.6, we were using old-style branches, because new-style branches didn’t support the features we needed. In 7.9.1, we apparently now don’t have a choice, but I’ve been assured that 7.9.1 provides the features we need in the new-style scans.

After starting to use it, we’re discovering that scans of pull request branches don’t appear to be respecting the quality gate set in the base project. The quality gate clearly shows a required 80% test coverage, and I created a pull request that clearly shows the resulting coverage after merge would be well less than 80%, and the little box at the top left, underneath “Quality Gate”, says “Passed”.

Is it possible that we missed something in our configuration?

Hi,

In PRs and short-lived branches the ‘on New Code’ portions of your Quality Gate are enforced. Does this make sense?

 
Ann

Thanks for replying.

So you’re telling me that for short-lived branches, the quality gate enforces the code coverage percentage limit against the “Coverage on N New Lines to cover” percentage, instead of on the “Estimated after merge” percentage? There’s no configuration option that lets us apply it to the latter instead? I’m going to guess your answers are “yes” and “no”.

My test case adds an @Ignore annotation to a unit test class, which doesn’t add any new lines of code. This passes the quality gate, and results in a broken build after merge.

I can easily see other test cases that add a small number of new lines that are not covered by tests, which would fail the quality gate, even though the coverage after merge would still pass.

This is not what we need. It is critical for us that the scan prevents developers from breaking the build after merge. After this upgrade, we have now lost that ability.

Hi,

Your understanding is correct.

Would you mind sharing what condition is broken after merge?

 
Thx,
Ann

The unit test code coverage. To be specific, the percentage after merge is less than 80%, which will break the build after merge.

I see! Adding that one “non-code” line means your coverage reports… no longer reflect what that test covers(?) and so you actively decrease your coverage with no reflection of that in your PR/SLB.

Unfortunately, PRs/SLBs are only going to reflect changes to updated code, so even if you deleted the test altogether they wouldn’t currently show the decrease in coverage because the now-uncovered lines aren’t “new”.

However 8.1, E.T.A. next week, drops the concept of SLB so that you have just “Branches” (which all act like LLBs) and PRs. This means that you get the “full” treatment on them, and this will probably do what you want. I believe we’ve worked hard on the migration to make sure your previously-SLBs still get cleaned up as expected. So… I urge you to test (and adopt!) the new release as soon as it’s available.

 
HTH,
Ann

Our enterprise goes a long time between SonarQube upgrades, and they only support LTS versions. Is 8.1 going to be LTS?

The last LTS, 7.9, was released in July, so no.

A post was split to a new topic: New lines in PR aren’t lines of code?

Changing tack a bit, can you tell me whether the value of “coverage after merge” from the scan is available through the REST api? I’ve explored and used the REST api for other things. I can see that I can get the details of the quality gate, including what quality gate is set for each project, and I can get some of the details of the results of the last scan, but from a quick look it’s not clear whether I can extract that resulting “coverage after merge” value. Can you detail the path to that value in the api? If I can get this information, I can restore the functionality that we had before this upgrade.

Hi,

The easiest thing here is to use your browser’s developer console to ‘spy’ on the web services that are called to paint the interface.

 
HTH,
Ann

Thanks for the reminder about that. I managed to get the required info before I saw this.

I’ve managed to implement a “post-scan” block that fails the build if the resulting coverage violates the “coverage” threshold. The SonarQube info will still say the QGs passed, but I can’t help that now.

Hopefully this will get back to a more informative view when we eventually upgrade to 8.x, as you mentioned.

I wanted to pop this back up because I’m definitely running into a similar issue.

Running: SonarQube Enterprise 8.6
Problem: Deleting a unittest does not fail the Quality Gate set
Expected Result: The decrease in Coverage for the project should fail the Quality Gate
Actual Result: Deleted lines are not seen as “New” or “Changed”

Steps to reproduce:

  1. Make sure you have a current analysis of the “Main” branch, in my case “develop” that shows the current amount of code coverage
  2. Set the project “New Code” setting to use the “develop” branch as the reference branch. And then you have to set the “develop” branch “New Code” to “Previous Version”
  3. Create a new branch
  4. Delete a unittest
  5. Commit the change, push to origin and create a PR
  6. Once the analysis is done, all I get is an “Estimated coverage after merge” which is WAY below the reference branch coverage and that there are no new lines of code and the Quality Gate passes.

Here is a snapshot of the Quality Gate being used

This unfortunately doesn’t work for my use case, as this clearly shows it will not capture events where a test is removed or ignored. Now, I do have an Overall Condition on the Quality Gate, and the branch that gets analyzed does fail that check. So I guess I have to script around this during my build and fail that instead.

Welcome :slight_smile:

seems you hit this https://jira.sonarsource.com/browse/SONAR-10485

Gilbert

Thanks for the welcome Gilbert,

While I have seen that issue crop up as well in my testing of the enterprise version, what I am actually seeing is that deleted lines of code from a test file that exceed the minimum will drop the coverage, only after merge which means that the QG passes.

For example, the develop branch has nearly 88% code coverage, so I create a branch off of that and delete 27 lines of test coverage. The analysis then says I would have A LOT less coverage, but on no new lines. Here’s a screenshot

Clicking through the “0 New Lines” it does show the file that got changed, but doesn’t seem to realize that lines were deleted… which should be a change. Does Sonar not think that deleted lines are “changed” or “new”?

Defining New Code | SonarQube Docs has

Choose a specific branch to define your New Code. Any changes made from your reference branch are considered New Code.

Available at the project and branch level.

Metric Definitions | SonarQube Docs has

Condition coverage on new code ( new_branch_coverage )
Identical to Condition coverage but restricted to new / updated source code.

So from reading that it should work as you expected.
Because of that i supposed it may be related to the problem with the ‘fudge factor’ mentioned in the linked Jira ticket.

… we began ignoring QG conditions on coverage and duplications on new code when there are fewer than 20 new lines.

If deleted code is also treated as new code, maybe your changes are < 20 lines ?

Gilbert

Hi Gilbert,

Actually, @Ryan_Shatford (Welcome to the community!) is right. We don’t handle deleted lines well (at all). And we’re particularly not handling nicely the deletion of tests. We focus on lines of code that were added or changed in the PR and in this case there are none. So yes, after merge your “passing” PR will drop overall coverage.

I’ve flagged this for attention internally. And I can’t promise that there will be action any time soon.

 
:woman_shrugging:
Ann

Thank you for the reply @ganncamp

As I expected, this is possibly a breaking change in how we utilize the community edition today versus us moving to the Enterprise version immediately. Please, when and if you have an external ticket to track this work, please update this thread.

Thanks,
Ryan

Hi Ryan

I guess in Community Edition you’re analyzing each branch/PR as a new project & comparing manually? If so, I’m going to repeat the advice I gave earlier that using branch analysis may be what you need.

And sorry, but I don’t have a ticket for you.

 
Ann

was also a tremendous clever question as @Ryan_Shatford already mentioned the
deletion of 27 lines and 27 > 20 :partying_face::poop: :scream:

1 Like