Quality Gate passing when 'Coverage' metric should fail it - but not on all projects

SonarQube
1

I have created a Quality Gate mostly focussed on what has been happening ‘lately’ - but one metric is about the overall current coverage in the project. The Quality Gate is set as the default - and all projects use it. The Leak Period is set to previous_version.

I have several projects that (correctly) fail the quality gate - and fail it only due to low coverage, but I have several projects that are below the coverage minimum, but show as Passed for the quality gate.

Quality gate definition:

image
image1656×714 68.4 KB

I can see the following scenarios:

  1. A project with 36% coverage, and 100% coverage on new lines - gate passes instead of failing on the overall coverage criteria
  2. A project with 59% coverage, and 62% coverage on new lines - gate fails - but NOT on the overall coverage criteria
  3. A project with 59% coverage, and 71% coverage on new lines - gate fails on both new lines coverage and overall coverage criteria.

Looks like a bug to me. Any hints as to what I might be able to do to work around this?

Using SonarQube: 6.7.5 (build 38563)

2

Hi,

Please check the column on the right side of your project dashboard where the Quality Gate being applied to the project is shown:

Selection_160

Is your custom quality gate being used for your projects?

 
Ann

3

Thanks for the response Ann. Yes, I did check that - and have just validated again. The three projects I mentioned about all use the same “Default” quality gate - which is my custom one.

Two of them use the default Python Quality Profile, and one uses that as well as the default C# Quality Profile. But - the quality profile should be independent of the Quality Gate Coverage metric - correct?

4

Hi,

Thanks for reporting this. I’ve reproduced it in 6.7.5, but not in 7.4 (altho I can’t find the ticket related to the fix). It appears that the Warning condition is evaluated first, and evaluation stops if it’s true. We’re not likely to tackle this in an LTS Bug fix. The workaround would be to remove the Warning condition.

Note that we’re planning to drop the ability to set Warning conditions in the “near” future:

MMF-473 - Simplify the definition of Quality Gate

 
Ann

5

Thanks Ann. I may have actually found an issue on my side for a couple of the builds. I had missed the fact that a few of them had not built in quite a while - and I had only added the “Coverage” metric a few weeks ago (seems like forever ago). Rerunning these seemed to make most of them clear up.

As you say - I may still have an issue with the warnings… but I can work around that as you suggest!