Quality Gates Don't Fail

Running in Jenkins version 2.150.2
SonarQube version 6.7 (upgrade to latest LTS is planned for later this year).
Sonar Scanner 3.3
Jenkins SonarQube plugin version 2.9
Jenkins Quality Gates plugin version 2.5

Jobs are no longer failing using the following quality gate definition. We even introduced bad, new code to force it to fail but it did not.

Here is our gate definition:

Hi,

“Bad” how?

 
Ann

The development introduced code to force a blocker. According to the Quality Gate set up I provided earlier, any new blocker should cause a failure.

Hi,

What’s the origin date shown on the far right of that issue block?

 
Ann

There is no date on the far right but it was creaeted on 1/22, at about 5:01 PM GMT.

Hi,

Thanks for the full issue block. We seem to have a genuinely “new” blocker issue here (versus one recently raised but backdated to before the start of the New Code Period).

How about a screenshot of the part of your homepage that shows the start of the New Code period?

Also, what’s the response when you directly call the api/qualitygates/project_status?projectKey=[your project key] web service?

 
Ann

1 Like

We are on version 6.7, which I believe does not yet have New Code Period. If it does, I can’t find it. It’s no in Administration==>Configure==>General Setting.

We are using the Jenkins Sonarqube plugin and are not using a scripted pipeline for this so we don’t have code that directly calls this:

“Also, what’s the response when you directly call the api/qualitygates/project_status?projectKey=[your project key] web service?”

If you can give me some direction on how to do that, I will be glad too.

Hi,

The New Code Period is a very old concept in SonarQube.

You’ll point a browser to
[http://yourSonarQubeInstance]/api/qualitygates/project_status?projectKey=[your project key]

and post the results.

 
Ann

Here is the response to quality gate api call:

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“test_errors”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“test_failures”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“minor_violations”,“comparator”:“GT”,“periodIndex”:1}],“periods”:,“ignoredConditions”:false}}

I have looked all over for the New Code Period in version 6.7.7. Maybe it is called something else?
The Leak Period is closest I can come:

Hi,

:woman_facepalming: Yes. The concept got renamed. Sorry for the confusion.

Thanks for the screenshot and the web service response. I see 2 things here:

  • Your New Code/Leak Period is set to previous_analysis. That means that if I do introduce a Quality Gate-breaking change, I only have to run another new analysis for it to go away. Probably what you wanted here is previous_version, which ought to be a longer period - assuming you don’t feed your build number into the sonar.projectVersion analysis parameter.

  • The result of your web api call doesn’t match the Quality Gate Screenshot you sent. It reflects a Quality Gate that tests these conditions:

    • test errors
    • test failures
    • minor new issues

It looks like maybe you have multiple Quality Gates in play and something other than what you expect is being applied.

 
HTH,
Ann

1 Like

Hi Ann. First thank you for helping me. We changed the Leak Period to previous_version per your suggestion. The scan ran and did show one new blocker bug.

If I hover over the “E” in the over the New Bug category, it shows:
“Reliability rating is E when there is at least one blocker bug.

It should have failed per our quality gate, which is the one I pasted earlier despite what the api call said. I confirmed that.

Re-pasted here for ease of reference:

Hi,

What does the web service say? Unless you’ve changed which Quality Gate is applied to your project then it’s perfectly predictable that the project didn’t fail the QG. As a reminder, your previous web service call indicated that only the following criteria are applied:

  • test errors
  • test failures
  • minor new issues

So you need to double-check (in the right rail of the project hompage) which Quality Gate is being applied.

 
Ann

Hi Ann. I did check and it is ILI_Gate, the one I pasted. I just noticed this message: It didn’t include that in the log sent back to Jenkins.

image

In the Jenkins log the Sonar step ends with “Execution Successful”, which simply means the scan succeeded, but not reporting the result of the scan.

Our Jenkins job is set up to fail if the Sonar scan files but it doesn’t seem like Sonar is sending a failed scan message anywhere.
I’ve attached the Sonar output that shows up in Jenkins.Sonar Output.txt (8.6 KB)

Hi,

So your Quality Gate is failing!

It looks like this is a question of your Jenkins job configuration. I don’t see any indication in your Jenkins log that it’s waiting for the QG status to be calculated. Can you share your job/pipeline configuration? Note that it should look something like this.

 
Ann

1 Like

Hi,

You wrote you are using the https://github.com/jenkinsci/quality-gates-plugin
This is a community plugin, not supported by Sonarsource.

And with webhook delivery, introduced in Sonarqube 6.2 it’s not needed anymore.
After background task has finished, Sonarqube server sends JSON to Sonarqube Jenkins plugin, which consumes the payload and breaks the build if quality gate is violated.
This works only with Jenkins pipelines, not with classis jobs.

i would go
use only Jenkins pipelines, no classic jobs
deinstall Jenkins quality gate plugin
configure Sonarqube webhooks
make sure the communication Jenkins <-> Sonarqube server works both ways
then your pipeline will fail if quality gate violations

2 Likes

Thank you Gilbert. I’ve been reading up on the webhooks and am not sure how it would send the response back to each individual job because it’s URL based. A maximum of 20 webhooks can be created. Each job execution has it’s own URL so obviously we can’t create a webhook for each one. Is there a way to configure the webhooks so the response from Sonar can be read by the pipeline that triggered the scan?

Hi,

no, it’s not that every job execution has its own url, but his own unique analysisid.
e.g. in Sonarqube web ui see /admin/background_tasks , the ID column has the analysisid.

Our Sonarqube Enterprise instance uses only 1 global webhook for Jenkins, configured
like https://somejenkinshost/sonarqube-webhook/, it’s one static url for all jobs.

Everytime when CE on Sonarqube server has completed a background task of an analysis
(= computing the report archive uploaded by the Sonar Scanner on Jenkins) it triggers a webhook,
sending a JSON playload that contains the analysisid beside branch name, quality gate result …

You will notice two different scenarios in Jenkins pipelines. If the background task has finished
in no time, the quality gate result is available immediately (Sonar Scanner does an initial call for quality gate result after upload) - no need to wait for webhook.
Otherwise the waitForQualityGate() method of Sonarqube Jenkins plugin creates a listener for
this specific analysisid waiting for / consuming the payload with the matching analysisid.
The Jenkins log shows status PENDING in that case.
A timeout of 10mins is sufficient for all our jobs.

Gilbert