Quality Gate is unable to report status back to jenkins

Linus_Boehm · March 11, 2020, 10:53am

versions used (SonarQube, Scanner, Plugin, and any relevant extension)
Sonarqube: 8.1-developer-beta
Sonarscanner: 4.0.0
Sonar Jenkins Plugin: 2.11

We have a Problem with the SonarQube Jenkins Plugin and the waitForQualityGate step where successful SonarQube checks are not forwarded to the waitForQualityGate step.

As far as I understood the logic of the Jenkins plugin it makes a POST request to the SonarQube QualityGate API endpoint and checks the status there, which either is SUCCESS or something like IN_PROGRESS. When it is still in progress Jenkins waits for the incoming webhook and reports back to waitForQualityGate step in the pipeline, which works fine.

When the request returns SUCCESS it seems like jenkins doesn’t wait for the webhook, which is fine I guess, but also the waitForQualityGate step in the pipeline doesn’t get notified.

Jenkins Logs:

Full details of the POST was
{"serverUrl":"https://ourhost.com",
"taskId":"AXDEniu_9bDjw4Tv_jLg",
"status":"SUCCESS",
"analysedAt":"2020-0310T13:24:18+0000",
"revision":"8629b3831a759a5d402903558fd33b015caf5045",
"changedAt":"2020-03-10T13:24:18+0000",
"project":{"key":"our-project-key",
"name":"Project name","url":"https://sonarqube.ourhost.com/dashboard?id=our-project-key"},"branch":{"name":"329","type":"PULL_REQUEST","isMain":false,"url":"https://sonarqube.ourhost.com/dashboard?id=our-project-key&pullRequest=329"},"qualityGate":{"name":"Quality Gate","status":"OK","conditions":[...]},"properties":{}}

waitForQualityGate step in Jenkinfile:

...
steps {
        timeout(time: 1, unit: 'HOURS') {
          waitForQualityGate abortPipeline: true
        }
}
...

Expected behaviour:
Jenkins gets notified about the Quality Gate result when the POST-Request had Status: SUCCESS
Actual Behaviour:
Jenkins is stuck in waitForQualityGate after POST-Request had Status: SUCCESS and runs into the timeout or in case Status was IN_PROGRES it waits for the webhook.

Please let us know a solution if there is one.
Thanks.

Julien_HENRY · March 12, 2020, 8:16am

Hi @Linus_Boehm

I’m a bit surprised you are talking about a POST request. This is expected to be a GET, so I wonder if you are not confusing different things?

github.com

SonarSource/sonar-scanner-jenkins/blob/a929c69130a286f25a388a535cbc05685c5c8563/src/main/java/hudson/plugins/sonar/client/HttpClient.java#L37




public class HttpClient {
  public String getHttp(String url, @Nullable String token) {
    String baseUrl = StringUtils.substringBeforeLast(url, "/");
    String path = StringUtils.substringAfterLast(url, "/");
    HttpConnector httpConnector = HttpConnector.newBuilder()
      .userAgent("Scanner for Jenkins")
      .url(baseUrl)
      .credentials(token, null)
      .build();
    WsResponse response = httpConnector.call(new GetRequest(path));
    response.failIfNotSuccessful();
    return response.content();
  }
}

Linus_Boehm · March 16, 2020, 9:11am

Hi @Julien_HENRY
Yeah you are right. Then I guess the log entry is from the incoming webhook from sonarqube. But Anyhow this specific build with the taskId from above is stuck in the waitForQualityGate step.
Any Idea why or how I can debug it further?

Julien_HENRY · March 17, 2020, 8:27am

Can you give more details about your entire Jenkins pipeline? Do you run only a single analysis? Are you correctly using the withSonarQubeEnv closure to trigger your analysis?
Can you maybe dump the content of the file analysis-report.txt that is produced by the scanner, and verify that the taskId is consistent with the one in the webhook payload.

torstenwalter · March 17, 2020, 10:53am

This is what the pipeline looks like:

pipeline {
  agent {
    kubernetes {
      label "${UUID.randomUUID().toString()}"
      defaultContainer 'tools'
      yamlFile 'buildpod.yaml'
    }
  }

  stages {

    ...

    stage('Sonarqube PR analysis') {
      when {
        changeRequest()
      }
      steps {
        withSonarQubeEnv('sonarqube') {
          sh """
            cp sonar-project.properties /tmp/sonar-project.properties
            echo "sonar.host.url=${SONAR_HOST_URL}
sonar.login=${SONAR_AUTH_TOKEN}
sonar.pullrequest.key=${env.CHANGE_ID}
sonar.pullrequest.base=${env.CHANGE_TARGET}
sonar.pullrequest.branch=${env.CHANGE_BRANCH}" >> /tmp/sonar-project.properties

            make jenkins-sonarqube
          """
        }
      }
    }


    stage('Sonarqube analysis') {
      when {
        branch 'master'
      }
      steps {
        withSonarQubeEnv('sonarqube') {
          sh """
            cp sonar-project.properties /tmp/sonar-project.properties
            echo "sonar.host.url=$SONAR_HOST_URL
sonar.login=$SONAR_AUTH_TOKEN
sonar.branch.name=$BRANCH_NAME" >> /tmp/sonar-project.properties

            make jenkins-sonarqube
          """
        }
      }
    }
    stage("Quality Gate") {
      when {
        anyOf {
          changeRequest()
          branch 'master'
        }
      }
      steps {
        timeout(time: 1, unit: 'HOURS') {
          waitForQualityGate abortPipeline: true
        }
      }
    }

    stage('Build and push dev docker image') {
      when {
        changeRequest()
      }
      environment {
        DOCKER_CONFIG = "/home/ci/.docker-development"
      }
      steps {
        sh '''
          make jenkins-skaffold
          skaffold build -p dev
        '''
      }
    }

    stage('Build and push docker image; deploy; create GitHub release') {
      when {
        branch 'master'
      }
      environment {
        DOCKER_CONFIG = "/home/ci/.docker-releases"
      }
      steps {
        withCredentials([usernamePassword(credentialsId: 'github', usernameVariable: 'GITHUB_USERNAME', passwordVariable: 'GITHUB_TOKEN')]) {
          sh '''
            # Setting Git credentials for release-version script
            git config credential.helper 'store --file /tmp/.gitcredentials'

            make jenkins-skaffold
            export VERSION=$(jx-release-version)
            skaffold run --namespace=cockpit-dev
            create-release -v "${VERSION}" -t "${GITHUB_TOKEN}"
          '''
        }
      }
    }
  }
}

There is a single SonarQube analysis being run. We juts use two different stages as master branch and PRs need different Parameters.

It might look a little complicated as we are running SonarQube within a docker container, but in the end it’s just an execution of sonar-scanner. To ensure that we have the results available on the build agent we copy the .scannerwork directory once the analysis is done.

This is the output of the SonarQube analysis:

#41 0.974 INFO: Scanner configuration file: /usr/lib/sonar-scanner/conf/sonar-scanner.properties
 #41 0.975 INFO: Project root configuration file: /scanner/sonar-project.properties
 #41 1.009 INFO: SonarQube Scanner 4.0.0.1744
 #41 1.010 INFO: Java 1.8.0_181 Oracle Corporation (64-bit)
 #41 1.010 INFO: Linux 4.15.0-1057-aws amd64
 #41 1.432 INFO: User cache: /root/.sonar/cache
 #41 2.886 INFO: SonarQube server 8.2.0
 #41 2.886 INFO: Default locale: "en_US", source code encoding: "UTF-8"
 #41 3.090 WARN: SonarScanner will require Java 11 to run starting in SonarQube 8.x
 #41 3.257 INFO: Load global settings
 #41 3.402 INFO: Load global settings (done) | time=144ms
 #41 3.408 INFO: Server id: ...
 #41 3.414 INFO: User cache: /root/.sonar/cache
 #41 3.416 INFO: Load/download plugins
 #41 3.416 INFO: Load plugins index
 #41 3.461 INFO: Load plugins index (done) | time=44ms
 #41 5.907 INFO: Load/download plugins (done) | time=2491ms
 #41 6.030 INFO: Loaded core extensions: developer-scanner
 #41 6.537 INFO: Process project properties
 #41 6.554 INFO: Process project properties (done) | time=17ms
 #41 6.558 INFO: Project key: cluster-manager
 #41 6.559 INFO: Base dir: /scanner
 #41 6.559 INFO: Working dir: /scanner/.scannerwork
 #41 6.564 INFO: Load project settings for component key: 'cluster-manager'
 #41 6.585 INFO: Load project settings for component key: 'cluster-manager' (done) | time=21ms
 #41 6.665 INFO: Load project branches
 #41 6.687 INFO: Load project branches (done) | time=22ms
 #41 6.689 INFO: Load project pull requests
 #41 6.732 INFO: Load project pull requests (done) | time=44ms
 #41 6.733 INFO: Load branch configuration
 #41 6.737 INFO: Load branch configuration (done) | time=4ms
 #41 6.753 INFO: Load quality profiles
 #41 6.804 INFO: Load quality profiles (done) | time=52ms
 #41 6.813 INFO: Load active rules
 #41 7.264 INFO: Load active rules (done) | time=451ms
 #41 7.281 INFO: Pull request 346 for merge into master from add-quality-gate-again
 #41 7.302 INFO: SCM collecting changed files in the branch
 #41 7.430 INFO: SCM collecting changed files in the branch (done) | time=128ms
 #41 7.449 INFO: Indexing files...
 #41 7.452 INFO: Project configuration:
 #41 7.452 INFO:   Excluded sources: /cmd/main.go, **/*_test.go, **/mocks.go, **/*_mocks.go, **/vendor/**, **/testdata/*, **/wire_gen.go, **/wire.go, /pkg/ui/api, **/*.yaml, **/*.md, **/Makefile, **/Dockerfile, **/*_test.go
 #41 7.452 INFO:   Included tests: **/*_test.go
 #41 7.750 INFO: 159 files indexed
 #41 7.751 INFO: 0 files ignored because of inclusion/exclusion patterns
 #41 7.752 INFO: 0 files ignored because of scm ignore settings
 #41 7.753 INFO: Quality profile for go: Sonar way
 #41 7.753 INFO: ------------- Run sensors on module Cluster Manager
 #41 7.826 INFO: Load metrics repository
 #41 7.853 INFO: Load metrics repository (done) | time=27ms
 #41 9.666 INFO: Sensor JavaXmlSensor [java]
 #41 9.670 INFO: Sensor JavaXmlSensor [java] (done) | time=4ms
 #41 9.670 INFO: Sensor JaCoCo XML Report Importer [jacoco]
 #41 9.682 INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=6ms
 #41 9.682 INFO: Sensor SonarGo [go]
 #41 9.682 INFO: 82 source files to be analyzed
 #41 12.20 INFO: Sensor SonarGo [go] (done) | time=2528ms
 #41 12.20 INFO: Sensor Go Unit Test Report [go]
 #41 12.20 INFO: 82/82 source files have been analyzed
 #41 12.23 INFO: Sensor Go Unit Test Report [go] (done) | time=27ms
 #41 12.23 INFO: Sensor Go Cover sensor for Go coverage [go]
 #41 12.23 INFO: Load coverage report from '/scanner/reports/coverage.out'
 #41 12.33 INFO: Sensor Go Cover sensor for Go coverage [go] (done) | time=94ms
 #41 12.33 INFO: ------------- Run sensors on project
 #41 12.33 INFO: Sensor Zero Coverage Sensor
 #41 12.34 INFO: Sensor Zero Coverage Sensor (done) | time=7ms
 #41 12.41 INFO: CPD Executor 21 files had no CPD blocks
 #41 12.41 INFO: CPD Executor Calculating CPD for 61 files
 #41 12.46 INFO: CPD Executor CPD calculation finished (done) | time=44ms
 #41 12.53 INFO: SCM writing changed lines
 #41 12.53 INFO: SCM writing changed lines (done) | time=2ms
 #41 12.54 INFO: Analysis report generated in 75ms, dir size=131 KB
 #41 12.63 INFO: Analysis report compressed in 88ms, zip size=84 KB
 #41 12.67 INFO: Analysis report uploaded in 39ms
 #41 12.67 INFO: ANALYSIS SUCCESSFUL, you can browse https://sonarqube.someurl/dashboard?id=cluster-manager&pullRequest=346
 #41 12.67 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
 #41 12.67 INFO: More about the report processing at https://sonarqube.someurl/api/ce/task?id=AXDoBrOuSB0dFw-7cLYE
 #41 12.68 INFO: Analysis total time: 6.652 s
 #41 12.68 INFO: ------------------------------------------------------------------------
 #41 12.68 INFO: EXECUTION SUCCESS
 #41 12.68 INFO: ------------------------------------------------------------------------
 #41 12.68 INFO: Total time: 11.719s
 #41 12.80 INFO: Final Memory: 27M/648M
 #41 12.80 INFO: ------------------------------------------------------------------------
 #41 DONE 13.0s

This is the content of report-task.txt

projectKey=cloud.syncier.cluster-manager
serverUrl=https://sonarqube.someurl
serverVersion=8.2.0.32929
dashboardUrl=https://sonarqube.someurl/dashboard?id=cloud.syncier.cluster-manager&pullRequest=346
ceTaskId=AXDoBrOuSB0dFw-7cLYE
ceTaskUrl=https://sonarqube.someurl/api/ce/task?id=AXDoBrOuSB0dFw-7cLYE

Checking status of SonarQube task ‘AXDoBrOuSB0dFw-7cLYE’ on server ‘sonarqube’
SonarQube task ‘AXDoBrOuSB0dFw-7cLYE’ status is ‘IN_PROGRESS’

Buil output shows

 Checking status of SonarQube task 'AXDoBrOuSB0dFw-7cLYE' on server 'sonarqube'
 SonarQube task 'AXDoBrOuSB0dFw-7cLYE' status is 'IN_PROGRESS'

So it looks like that it’s checking the correct task.

Julien_HENRY · March 17, 2020, 11:09am

By chance, do you have configured many webhooks/email notifications on SonarQube? We have recently identified a kind of race condition, that is more likely to occur when task post-processing is long.

torstenwalter · March 17, 2020, 11:41am

Just one Webhook and no mail notifications. Task processing is < 10 seconds.

Linus_Boehm · March 17, 2020, 11:45am

We also did an upgrade to the sonarqube-8.2-developer editon. Also didn’t solved the issue

torstenwalter · March 17, 2020, 12:09pm

I noticed that the SonarQube Jenkins plugins writes some logs as well

github.com

jenkinsci/sonarqube-plugin/blob/master/src/main/java/org/sonarsource/scanner/jenkins/pipeline/WaitForQualityGateStep.java#L213-L216


private void log(String msg, Object... args) {
  getContextClass(TaskListener.class).getLogger().printf(msg, args);
  getContextClass(TaskListener.class).getLogger().println();
}

@Julien_HENRY Do you know which logger we need to configure in Jenkins to be able to see those logs?

glecocq · April 7, 2020, 10:42pm

Hello,
I experience a similar issue. Versions used

Sonarqube 8.2.0.32929
Sonar Jenkins Plugin: 2.11

The jenkins job is a multi branch pipeline.
The webhook is configured in sonarqube, and I see the payload with an HTTP 200 status.
Response: 200
Duration: 113ms

Payload

{
“serverUrl”: “http://jenkins”,
“taskId”: “AXFWtbW7SOnNud6I6PxC”,
“status”: “SUCCESS”,
“analysedAt”: “2020-04-07T22:10:13+0000”,
“revision”: “907e1221d7759c9e9831897f16e2560627569d4c”,
“changedAt”: “2020-04-07T22:10:13+0000”,
“project”: {
“key”: “myproject:trunk”,
“name”: “myproject Trunk”,
“url”: “http://jenkins/dashboard?id=myproject%3Atrunk”
},
“branch”: {
“name”: “PR-3850”,
“type”: “PULL_REQUEST”,
“isMain”: false,
“url”: “http://jenkins/dashboard?id=myproject%3Atrunk&pullRequest=PR-3850”
},
“qualityGate”: {
“name”: “quality gate”,
“status”: “OK”,
“conditions”: [
{
“metric”: “new_blocker_violations”,
“operator”: “GREATER_THAN”,
“value”: “0”,
“status”: “OK”,
“errorThreshold”: “0”
},
{
“metric”: “new_critical_violations”,
“operator”: “GREATER_THAN”,
“value”: “0”,
“status”: “OK”,
“errorThreshold”: “0”
},
{
“metric”: “new_major_violations”,
“operator”: “GREATER_THAN”,
“value”: “0”,
“status”: “OK”,
“errorThreshold”: “0”
}
]
},
“properties”: {}
}

On Jenkins side, in the logs, I see the payload, that looks correct:

jenkins log

Apr 07, 2020 10:14:44 PM INFO org.sonarsource.scanner.jenkins.pipeline.SonarQubeWebHook doIndex

Received POST from xxx.xxx.xxx.xxx

Apr 07, 2020 10:14:44 PM FINE org.sonarsource.scanner.jenkins.pipeline.SonarQubeWebHook

Full details of the POST was {“serverUrl”:“http://jenkins”,“taskId”:“AXFWtbW7SOnNud6I6PxC”,“status”:“SUCCESS”,“analysedAt”:“2020-04-07T22:10:13+0000”,“revision”:“907e1221d7759c9e9831897f16e2560627569d4c”,“changedAt”:“2020-04-07T22:10:13+0000”,“project”:{“key”:“myproject:trunk”,“name”:“myproject Trunk”,“url”:“http://jenkins/dashboard?id=myproject%3Atrunk"},“branch”:{“name”:“PR-3850”,“type”:“PULL_REQUEST”,“isMain”:false,“url”:“http://jenkins/dashboard?id=myproject%3Atrunk&pullRequest=PR-3850”},“qualityGate”:{“name”:"Accounting quality gate”,“status”:“OK”,“conditions”:[{“metric”:“new_blocker_violations”,“operator”:“GREATER_THAN”,“value”:“0”,“status”:“OK”,“errorThreshold”:“0”},{“metric”:“new_critical_violations”,“operator”:“GREATER_THAN”,“value”:“0”,“status”:“OK”,“errorThreshold”:“0”},{“metric”:“new_major_violations”,“operator”:“GREATER_THAN”,“value”:“0”,“status”:“OK”,“errorThreshold”:“0”}]},“properties”:{}}

And the pipeline is blocked on the step ‘waitForQualityGate’ until the timeout (10 minutes) (note that I verified the quality gate was ready before the end of the timeout) . Here is the log from the pipeline:

Wait for SonarQube analysis to be completed and return quality gate status10m 0s
Checking status of SonarQube task 'AXFVVnzRSOnNud6I6Pwj' on server 'SonarQube'
SonarQube task 'AXFVVnzRSOnNud6I6Pwj' status is 'IN_PROGRESS'

Please note that if the quality gate is ready after the start of the step ‘waitForQualityGate’, then the initial request to sonarqube will return status OK for QG, and the step will pass.
And I see some pipeline execution where the quality gate was in progress when entering the step ‘waitForQualityGate’ and that then pass. ex:

Wait for SonarQube analysis to be completed and return quality gate status26s
Checking status of SonarQube task 'AXFVbj2QSOnNud6I6Pwr' on server 'SonarQube'
SonarQube task 'AXFVbj2QSOnNud6I6Pwr' status is 'IN_PROGRESS'
SonarQube task 'AXFVbj2QSOnNud6I6Pwr' status is 'SUCCESS'
SonarQube task 'AXFVbj2QSOnNud6I6Pwr' completed. Quality gate is 'OK'

So, I think my setup is correct, but it is more a race condition. I saw a post from Julien Henry talking about recently identified kind of race condition. In my case, task post-processing is in average between 10s and 1min.

Does anybody have an idea ?
Thank you.

Rebse · April 8, 2020, 6:04am

Hi Torsten,

this is our logrecorder config for Sonarqube

Gilbert

Rebse · April 8, 2020, 6:10am

Welcome

are you using Sonarqube scan with parallel in Jenkins pipelines ?
We noticed similar behaviour, see

Gilbert

Julien_HENRY · April 8, 2020, 7:06am

Hi @glecocq

Indeed it might be the same issue I described here: [SONARJNKNS-320] - Jira

The longer the SonarQube compute engine task processing takes, the more likely this issue could appear.

For the record, any special reason why task post-processing would take longer than “normal”? Do you configure a lot of webhooks or notifications? Or maybe do you use a custom plugin using the PostProjectAnalysisTask extension point?

glecocq · April 8, 2020, 7:20am

Thank you Julien.

I looked at the issue SONARJNKNS-320, and yes it might be the same issue.
I also had a look at my task post-processing history, and it take almost always 11s. I have a single webhook, and I didn’t add notifications. No usage of a custom plugin either.

The issue I described seems to occur more often on Pull request analysis than on branch analysis. But I don’t know if there is a correlation.

regards,
Guillaume.

glecocq · April 8, 2020, 7:22am

Hi,

No, the sonar analysis is not done in a parallel pipeline

thx,
Guillaume.

Julien_HENRY · April 8, 2020, 7:54am

That’s a really interesting information. I checked and PR decoration is done as a PostProjectAnalysisTask, so this certainly make the race condition more likely to occurs.

glecocq · April 8, 2020, 8:36am

Oh, I didn’t know that PR decoration is done as a PostProjectAnalysisTask. I will follow the issue SONARJNKNS-320 then. Thanks a lot.

Pretorian · May 29, 2022, 7:00pm

Ran into this question after a few days of looking, does anyone have a 20 on a fix for this. I have 3 environments all configured the same way one of them is running into this issue, infra and security groups are not a problem, however it seems that at the time jenkins wants to read that CE task url it hangs. I’m at a loss, throw me a line if you can if not, I’ll get some snacks and wait.