Quality Gate is unable to report status back to jenkins

SonarQube Server / Community Build
1
  • versions used (SonarQube, Scanner, Plugin, and any relevant extension)
    Sonarqube: 8.1-developer-beta
    Sonarscanner: 4.0.0
    Sonar Jenkins Plugin: 2.11

We have a Problem with the SonarQube Jenkins Plugin and the waitForQualityGate step where successful SonarQube checks are not forwarded to the waitForQualityGate step.

As far as I understood the logic of the Jenkins plugin it makes a POST request to the SonarQube QualityGate API endpoint and checks the status there, which either is SUCCESS or something like IN_PROGRESS. When it is still in progress Jenkins waits for the incoming webhook and reports back to waitForQualityGate step in the pipeline, which works fine.

When the request returns SUCCESS it seems like jenkins doesn’t wait for the webhook, which is fine I guess, but also the waitForQualityGate step in the pipeline doesn’t get notified.

Jenkins Logs:

Full details of the POST was
{"serverUrl":"https://ourhost.com",
"taskId":"AXDEniu_9bDjw4Tv_jLg",
"status":"SUCCESS",
"analysedAt":"2020-0310T13:24:18+0000",
"revision":"8629b3831a759a5d402903558fd33b015caf5045",
"changedAt":"2020-03-10T13:24:18+0000",
"project":{"key":"our-project-key",
"name":"Project name","url":"https://sonarqube.ourhost.com/dashboard?id=our-project-key"},"branch":{"name":"329","type":"PULL_REQUEST","isMain":false,"url":"https://sonarqube.ourhost.com/dashboard?id=our-project-key&pullRequest=329"},"qualityGate":{"name":"Quality Gate","status":"OK","conditions":[...]},"properties":{}}

waitForQualityGate step in Jenkinfile:

...
steps {
        timeout(time: 1, unit: 'HOURS') {
          waitForQualityGate abortPipeline: true
        }
}
...

Expected behaviour:
Jenkins gets notified about the Quality Gate result when the POST-Request had Status: SUCCESS
Actual Behaviour:
Jenkins is stuck in waitForQualityGate after POST-Request had Status: SUCCESS and runs into the timeout or in case Status was IN_PROGRES it waits for the webhook.

Please let us know a solution if there is one.
Thanks.

4

Hi @Linus_Boehm

I’m a bit surprised you are talking about a POST request. This is expected to be a GET, so I wonder if you are not confusing different things?

5

Hi @Julien_HENRY
Yeah you are right. Then I guess the log entry is from the incoming webhook from sonarqube. But Anyhow this specific build with the taskId from above is stuck in the waitForQualityGate step.
Any Idea why or how I can debug it further?

6

Can you give more details about your entire Jenkins pipeline? Do you run only a single analysis? Are you correctly using the withSonarQubeEnv closure to trigger your analysis?
Can you maybe dump the content of the file analysis-report.txt that is produced by the scanner, and verify that the taskId is consistent with the one in the webhook payload.

7

This is what the pipeline looks like:

pipeline {
  agent {
    kubernetes {
      label "${UUID.randomUUID().toString()}"
      defaultContainer 'tools'
      yamlFile 'buildpod.yaml'
    }
  }

  stages {

    ...

    stage('Sonarqube PR analysis') {
      when {
        changeRequest()
      }
      steps {
        withSonarQubeEnv('sonarqube') {
          sh """
            cp sonar-project.properties /tmp/sonar-project.properties
            echo "sonar.host.url=${SONAR_HOST_URL}
sonar.login=${SONAR_AUTH_TOKEN}
sonar.pullrequest.key=${env.CHANGE_ID}
sonar.pullrequest.base=${env.CHANGE_TARGET}
sonar.pullrequest.branch=${env.CHANGE_BRANCH}" >> /tmp/sonar-project.properties

            make jenkins-sonarqube
          """
        }
      }
    }


    stage('Sonarqube analysis') {
      when {
        branch 'master'
      }
      steps {
        withSonarQubeEnv('sonarqube') {
          sh """
            cp sonar-project.properties /tmp/sonar-project.properties
            echo "sonar.host.url=$SONAR_HOST_URL
sonar.login=$SONAR_AUTH_TOKEN
sonar.branch.name=$BRANCH_NAME" >> /tmp/sonar-project.properties

            make jenkins-sonarqube
          """
        }
      }
    }
    stage("Quality Gate") {
      when {
        anyOf {
          changeRequest()
          branch 'master'
        }
      }
      steps {
        timeout(time: 1, unit: 'HOURS') {
          waitForQualityGate abortPipeline: true
        }
      }
    }

    stage('Build and push dev docker image') {
      when {
        changeRequest()
      }
      environment {
        DOCKER_CONFIG = "/home/ci/.docker-development"
      }
      steps {
        sh '''
          make jenkins-skaffold
          skaffold build -p dev
        '''
      }
    }

    stage('Build and push docker image; deploy; create GitHub release') {
      when {
        branch 'master'
      }
      environment {
        DOCKER_CONFIG = "/home/ci/.docker-releases"
      }
      steps {
        withCredentials([usernamePassword(credentialsId: 'github', usernameVariable: 'GITHUB_USERNAME', passwordVariable: 'GITHUB_TOKEN')]) {
          sh '''
            # Setting Git credentials for release-version script
            git config credential.helper 'store --file /tmp/.gitcredentials'

            make jenkins-skaffold
            export VERSION=$(jx-release-version)
            skaffold run --namespace=cockpit-dev
            create-release -v "${VERSION}" -t "${GITHUB_TOKEN}"
          '''
        }
      }
    }
  }
}

There is a single SonarQube analysis being run. We juts use two different stages as master branch and PRs need different Parameters.

It might look a little complicated as we are running SonarQube within a docker container, but in the end it’s just an execution of sonar-scanner. To ensure that we have the results available on the build agent we copy the .scannerwork directory once the analysis is done.

This is the output of the SonarQube analysis:

#41 0.974 INFO: Scanner configuration file: /usr/lib/sonar-scanner/conf/sonar-scanner.properties
 #41 0.975 INFO: Project root configuration file: /scanner/sonar-project.properties
 #41 1.009 INFO: SonarQube Scanner 4.0.0.1744
 #41 1.010 INFO: Java 1.8.0_181 Oracle Corporation (64-bit)
 #41 1.010 INFO: Linux 4.15.0-1057-aws amd64
 #41 1.432 INFO: User cache: /root/.sonar/cache
 #41 2.886 INFO: SonarQube server 8.2.0
 #41 2.886 INFO: Default locale: "en_US", source code encoding: "UTF-8"
 #41 3.090 WARN: SonarScanner will require Java 11 to run starting in SonarQube 8.x
 #41 3.257 INFO: Load global settings
 #41 3.402 INFO: Load global settings (done) | time=144ms
 #41 3.408 INFO: Server id: ...
 #41 3.414 INFO: User cache: /root/.sonar/cache
 #41 3.416 INFO: Load/download plugins
 #41 3.416 INFO: Load plugins index
 #41 3.461 INFO: Load plugins index (done) | time=44ms
 #41 5.907 INFO: Load/download plugins (done) | time=2491ms
 #41 6.030 INFO: Loaded core extensions: developer-scanner
 #41 6.537 INFO: Process project properties
 #41 6.554 INFO: Process project properties (done) | time=17ms
 #41 6.558 INFO: Project key: cluster-manager
 #41 6.559 INFO: Base dir: /scanner
 #41 6.559 INFO: Working dir: /scanner/.scannerwork
 #41 6.564 INFO: Load project settings for component key: 'cluster-manager'
 #41 6.585 INFO: Load project settings for component key: 'cluster-manager' (done) | time=21ms
 #41 6.665 INFO: Load project branches
 #41 6.687 INFO: Load project branches (done) | time=22ms
 #41 6.689 INFO: Load project pull requests
 #41 6.732 INFO: Load project pull requests (done) | time=44ms
 #41 6.733 INFO: Load branch configuration
 #41 6.737 INFO: Load branch configuration (done) | time=4ms
 #41 6.753 INFO: Load quality profiles
 #41 6.804 INFO: Load quality profiles (done) | time=52ms
 #41 6.813 INFO: Load active rules
 #41 7.264 INFO: Load active rules (done) | time=451ms
 #41 7.281 INFO: Pull request 346 for merge into master from add-quality-gate-again
 #41 7.302 INFO: SCM collecting changed files in the branch
 #41 7.430 INFO: SCM collecting changed files in the branch (done) | time=128ms
 #41 7.449 INFO: Indexing files...
 #41 7.452 INFO: Project configuration:
 #41 7.452 INFO:   Excluded sources: /cmd/main.go, **/*_test.go, **/mocks.go, **/*_mocks.go, **/vendor/**, **/testdata/*, **/wire_gen.go, **/wire.go, /pkg/ui/api, **/*.yaml, **/*.md, **/Makefile, **/Dockerfile, **/*_test.go
 #41 7.452 INFO:   Included tests: **/*_test.go
 #41 7.750 INFO: 159 files indexed
 #41 7.751 INFO: 0 files ignored because of inclusion/exclusion patterns
 #41 7.752 INFO: 0 files ignored because of scm ignore settings
 #41 7.753 INFO: Quality profile for go: Sonar way
 #41 7.753 INFO: ------------- Run sensors on module Cluster Manager
 #41 7.826 INFO: Load metrics repository
 #41 7.853 INFO: Load metrics repository (done) | time=27ms
 #41 9.666 INFO: Sensor JavaXmlSensor [java]
 #41 9.670 INFO: Sensor JavaXmlSensor [java] (done) | time=4ms
 #41 9.670 INFO: Sensor JaCoCo XML Report Importer [jacoco]
 #41 9.682 INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=6ms
 #41 9.682 INFO: Sensor SonarGo [go]
 #41 9.682 INFO: 82 source files to be analyzed
 #41 12.20 INFO: Sensor SonarGo [go] (done) | time=2528ms
 #41 12.20 INFO: Sensor Go Unit Test Report [go]
 #41 12.20 INFO: 82/82 source files have been analyzed
 #41 12.23 INFO: Sensor Go Unit Test Report [go] (done) | time=27ms
 #41 12.23 INFO: Sensor Go Cover sensor for Go coverage [go]
 #41 12.23 INFO: Load coverage report from '/scanner/reports/coverage.out'
 #41 12.33 INFO: Sensor Go Cover sensor for Go coverage [go] (done) | time=94ms
 #41 12.33 INFO: ------------- Run sensors on project
 #41 12.33 INFO: Sensor Zero Coverage Sensor
 #41 12.34 INFO: Sensor Zero Coverage Sensor (done) | time=7ms
 #41 12.41 INFO: CPD Executor 21 files had no CPD blocks
 #41 12.41 INFO: CPD Executor Calculating CPD for 61 files
 #41 12.46 INFO: CPD Executor CPD calculation finished (done) | time=44ms
 #41 12.53 INFO: SCM writing changed lines
 #41 12.53 INFO: SCM writing changed lines (done) | time=2ms
 #41 12.54 INFO: Analysis report generated in 75ms, dir size=131 KB
 #41 12.63 INFO: Analysis report compressed in 88ms, zip size=84 KB
 #41 12.67 INFO: Analysis report uploaded in 39ms
 #41 12.67 INFO: ANALYSIS SUCCESSFUL, you can browse https://sonarqube.someurl/dashboard?id=cluster-manager&pullRequest=346
 #41 12.67 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
 #41 12.67 INFO: More about the report processing at https://sonarqube.someurl/api/ce/task?id=AXDoBrOuSB0dFw-7cLYE
 #41 12.68 INFO: Analysis total time: 6.652 s
 #41 12.68 INFO: ------------------------------------------------------------------------
 #41 12.68 INFO: EXECUTION SUCCESS
 #41 12.68 INFO: ------------------------------------------------------------------------
 #41 12.68 INFO: Total time: 11.719s
 #41 12.80 INFO: Final Memory: 27M/648M
 #41 12.80 INFO: ------------------------------------------------------------------------
 #41 DONE 13.0s

This is the content of report-task.txt

projectKey=cloud.syncier.cluster-manager
serverUrl=https://sonarqube.someurl
serverVersion=8.2.0.32929
dashboardUrl=https://sonarqube.someurl/dashboard?id=cloud.syncier.cluster-manager&pullRequest=346
ceTaskId=AXDoBrOuSB0dFw-7cLYE
ceTaskUrl=https://sonarqube.someurl/api/ce/task?id=AXDoBrOuSB0dFw-7cLYE

Checking status of SonarQube task ‘AXDoBrOuSB0dFw-7cLYE’ on server ‘sonarqube’
SonarQube task ‘AXDoBrOuSB0dFw-7cLYE’ status is ‘IN_PROGRESS’

Buil output shows

 Checking status of SonarQube task 'AXDoBrOuSB0dFw-7cLYE' on server 'sonarqube'
 SonarQube task 'AXDoBrOuSB0dFw-7cLYE' status is 'IN_PROGRESS'

So it looks like that it’s checking the correct task.

8

By chance, do you have configured many webhooks/email notifications on SonarQube? We have recently identified a kind of race condition, that is more likely to occur when task post-processing is long.

10

Just one Webhook and no mail notifications. Task processing is < 10 seconds.

11

We also did an upgrade to the sonarqube-8.2-developer editon. Also didn’t solved the issue

12

I noticed that the SonarQube Jenkins plugins writes some logs as well

@Julien_HENRY Do you know which logger we need to configure in Jenkins to be able to see those logs?

13

Hello,
I experience a similar issue. Versions used

  • Sonarqube 8.2.0.32929
  • Sonar Jenkins Plugin: 2.11

The jenkins job is a multi branch pipeline.
The webhook is configured in sonarqube, and I see the payload with an HTTP 200 status.
Response: 200
Duration: 113ms

Payload

{
“serverUrl”: “http://jenkins”,
“taskId”: “AXFWtbW7SOnNud6I6PxC”,
“status”: “SUCCESS”,
“analysedAt”: “2020-04-07T22:10:13+0000”,
“revision”: “907e1221d7759c9e9831897f16e2560627569d4c”,
“changedAt”: “2020-04-07T22:10:13+0000”,
“project”: {
“key”: “myproject:trunk”,
“name”: “myproject Trunk”,
“url”: “http://jenkins/dashboard?id=myproject%3Atrunk”
},
“branch”: {
“name”: “PR-3850”,
“type”: “PULL_REQUEST”,
“isMain”: false,
“url”: “http://jenkins/dashboard?id=myproject%3Atrunk&pullRequest=PR-3850”
},
“qualityGate”: {
“name”: “quality gate”,
“status”: “OK”,
“conditions”: [
{
“metric”: “new_blocker_violations”,
“operator”: “GREATER_THAN”,
“value”: “0”,
“status”: “OK”,
“errorThreshold”: “0”
},
{
“metric”: “new_critical_violations”,
“operator”: “GREATER_THAN”,
“value”: “0”,
“status”: “OK”,
“errorThreshold”: “0”
},
{
“metric”: “new_major_violations”,
“operator”: “GREATER_THAN”,
“value”: “0”,
“status”: “OK”,
“errorThreshold”: “0”
}
]
},
“properties”: {}
}

On Jenkins side, in the logs, I see the payload, that looks correct:

jenkins log

Apr 07, 2020 10:14:44 PM INFO org.sonarsource.scanner.jenkins.pipeline.SonarQubeWebHook doIndex

Received POST from xxx.xxx.xxx.xxx

Apr 07, 2020 10:14:44 PM FINE org.sonarsource.scanner.jenkins.pipeline.SonarQubeWebHook

Full details of the POST was {“serverUrl”:“http://jenkins”,“taskId”:“AXFWtbW7SOnNud6I6PxC”,“status”:“SUCCESS”,“analysedAt”:“2020-04-07T22:10:13+0000”,“revision”:“907e1221d7759c9e9831897f16e2560627569d4c”,“changedAt”:“2020-04-07T22:10:13+0000”,“project”:{“key”:“myproject:trunk”,“name”:“myproject Trunk”,“url”:“http://jenkins/dashboard?id=myproject%3Atrunk"},“branch”:{“name”:“PR-3850”,“type”:“PULL_REQUEST”,“isMain”:false,“url”:“http://jenkins/dashboard?id=myproject%3Atrunk&pullRequest=PR-3850”},“qualityGate”:{“name”:"Accounting quality gate”,“status”:“OK”,“conditions”:[{“metric”:“new_blocker_violations”,“operator”:“GREATER_THAN”,“value”:“0”,“status”:“OK”,“errorThreshold”:“0”},{“metric”:“new_critical_violations”,“operator”:“GREATER_THAN”,“value”:“0”,“status”:“OK”,“errorThreshold”:“0”},{“metric”:“new_major_violations”,“operator”:“GREATER_THAN”,“value”:“0”,“status”:“OK”,“errorThreshold”:“0”}]},“properties”:{}}

And the pipeline is blocked on the step ‘waitForQualityGate’ until the timeout (10 minutes) (note that I verified the quality gate was ready before the end of the timeout) . Here is the log from the pipeline:

Wait for SonarQube analysis to be completed and return quality gate status10m 0s
Checking status of SonarQube task 'AXFVVnzRSOnNud6I6Pwj' on server 'SonarQube'
SonarQube task 'AXFVVnzRSOnNud6I6Pwj' status is 'IN_PROGRESS'

Please note that if the quality gate is ready after the start of the step ‘waitForQualityGate’, then the initial request to sonarqube will return status OK for QG, and the step will pass.
And I see some pipeline execution where the quality gate was in progress when entering the step ‘waitForQualityGate’ and that then pass. ex:

Wait for SonarQube analysis to be completed and return quality gate status26s
Checking status of SonarQube task 'AXFVbj2QSOnNud6I6Pwr' on server 'SonarQube'
SonarQube task 'AXFVbj2QSOnNud6I6Pwr' status is 'IN_PROGRESS'
SonarQube task 'AXFVbj2QSOnNud6I6Pwr' status is 'SUCCESS'
SonarQube task 'AXFVbj2QSOnNud6I6Pwr' completed. Quality gate is 'OK'

So, I think my setup is correct, but it is more a race condition. I saw a post from Julien Henry talking about recently identified kind of race condition. In my case, task post-processing is in average between 10s and 1min.

Does anybody have an idea ?
Thank you.

14

Hi Torsten,

this is our logrecorder config for Sonarqube

jenkins_sq_logrecorder
jenkins_sq_logrecorder1136×256 5.82 KB

Gilbert

15

Welcome :slight_smile:

are you using Sonarqube scan with parallel in Jenkins pipelines ?
We noticed similar behaviour, see

Gilbert

16

Hi @glecocq

Indeed it might be the same issue I described here: [SONARJNKNS-320] - Jira

The longer the SonarQube compute engine task processing takes, the more likely this issue could appear.

For the record, any special reason why task post-processing would take longer than “normal”? Do you configure a lot of webhooks or notifications? Or maybe do you use a custom plugin using the PostProjectAnalysisTask extension point?

17

Thank you Julien.

I looked at the issue SONARJNKNS-320, and yes it might be the same issue.
I also had a look at my task post-processing history, and it take almost always 11s. I have a single webhook, and I didn’t add notifications. No usage of a custom plugin either.

The issue I described seems to occur more often on Pull request analysis than on branch analysis. But I don’t know if there is a correlation.

regards,
Guillaume.

18

Hi,

No, the sonar analysis is not done in a parallel pipeline

thx,
Guillaume.

19

That’s a really interesting information. I checked and PR decoration is done as a PostProjectAnalysisTask, so this certainly make the race condition more likely to occurs.

20

Oh, I didn’t know that PR decoration is done as a PostProjectAnalysisTask. I will follow the issue SONARJNKNS-320 then. Thanks a lot.

21

Ran into this question after a few days of looking, does anyone have a 20 on a fix for this. I have 3 environments all configured the same way one of them is running into this issue, infra and security groups are not a problem, however it seems that at the time jenkins wants to read that CE task url it hangs. I’m at a loss, throw me a line if you can if not, I’ll get some snacks and wait.