waitForQualityGate timeout in Jenkins

performance
jenkins

(Idan Adar) #1

This is my stage:

stage ("SonarQube analysis") {  
        options {
            timeout(time: 5, unit: 'MINUTES')
            retry(2)
        }
    
        when {
          allOf {
            changeRequest()
          }
        }
      
        steps {
          script {
            STAGE_NAME = "SonarQube analysis"
      
            withSonarQubeEnv('****') {
              sh "../../../sonar-scanner/bin/sonar-scanner"
            }
      
            qualitygate = waitForQualityGate()
            if (qualitygate.status != "OK") {
              currentBuild.result = "FAILURE"
              slackSend (channel: '****', color: '#F01717', message: "*$JOB_NAME*, <$BUILD_URL|Build #$BUILD_NUMBER>: Code coverage threshold was not met! <http://****.com:9000/sonarqube/projects|Review in SonarQube>.")
            }
          }
        }
      
        post {
          aborted {
            slackSend (channel: '****', color: '#F01717', message: "*$JOB_NAME*, <$BUILD_URL|Build #$BUILD_NUMBER>: '$STAGE_NAME' stage timed out, re-start the job.")
          }
        }
      }

Many times, this gets stuck and the log shows “IN_PROGRESS”.
Because this happens so often, I’ve added the following to limit its duration and retrying twice before giving up…

options {
    timeout(time: 5, unit: 'MINUTES')
    retry(2)
}

Is there any way to debug these timeouts?


(Nicolas Bontoux) #2

Hi Idan,

Seems to me that you first need to understand at which precise point is the stage timing out.

I can also see that you’re using waitForQualityGate, which as documented:

The waitForQualityGate step will pause the pipeline until SonarQube analysis is completed and returns quality gate status.

So you’d have to check if it’s not just a matter of the Background Task being queued while other tasks are ongoing. Whether it’s that or something else, understanding when the timeout occurs will help you narrow this down.


(Idan Adar) #3

Hey @NicoB,

There are no other jobs running at the same time.
Can I see somewhere how long did a particular background task took? maybe the analysis takes longer than the allotted timeout in the pipelone.


(Idan Adar) #4

Seems they were completed quite fast, and the stage timeout is 5 minutes.
What am I missing in understanding the whole flow?

{
task: {
id: "AWWFqxfzLykuh2LH3Urg",
type: "REPORT",
componentId: "AVr0gAOjyohgD3UCfaIn",
componentKey: "****",
componentName: "****",
componentQualifier: "TRK",
analysisId: "AWWFqxwkZLXM20NZPMoT",
status: "SUCCESS",
submittedAt: "2018-08-29T15:31:47+0300",
submitterLogin: "admin",
startedAt: "2018-08-29T15:31:48+0300",
executedAt: "2018-08-29T15:31:50+0300",
executionTimeMs: 2652,
logs: false,
hasScannerContext: true,
organization: "default-organization"
}
}

Checking status of SonarQube task ‘AWWFqxfzLykuh2LH3Urg’ on server ‘****’
SonarQube task ‘AWWFqxfzLykuh2LH3Urg’ status is ‘IN_PROGRESS’
Cancelling nested steps due to timeout

It also happens several times in a row for the same job… until it for some reason decides to pass.


(Idan Adar) #6

@NicoB Anything you can add based on the above? Right now we cannot use sonar due to this.


(Nicolas Bontoux) #7

Do you feel observing similar situation as in the below thread ? Would be interesting to know if a pattern emerges for those separate reports.


(David Karr) #8

Just to chime in on a post that I hadn’t noticed before that my post was included in, I would emphasize that I saw the same behavior that you did. I don’t know what led me to try adding a “sleep(10)” between the steps, but it seemed like a situation where I was calling “waitForQualityGate” before something was properly ready, so adding the sleep seemed to make sense, and it appears to resolve the issue.

I’m only doing this in one project as an experiment. We were previously using the BuildBreaker, which has now been removed, and I’ve set up the webhook and the withSonarQubeEnv/waitForQualityGate pair (with the sleep) in this project. I’m currently mystified about this behavior with the required sleep, and I’m even more concerned about the webhook. The documentation I’ve read says this is part of the solution, but nothing describes exactly HOW it’s part of the solution. How is the payload sent by the webhook related to the management of these tasks? In fact, in one test case for this, I don’t even have the webhook installed at all, and I can’t see that it makes any difference at all.


(Idan Adar) #9

I will try with the sleep workaround; for us it is happening with all projects, no matter their size.
Will report back.

withSonarQubeEnv('...') {
  sh "../../../sonar-scanner/bin/sonar-scanner"
}

sleep(10)
qualitygate = waitForQualityGate()
if (qualitygate.status != "OK") {
    ...
}

(Idan Adar) #10

The sleep(10) tweak seems to have done the trick thus far.
Perhaps some delay option (configurable or not) could be introduced so we won’t need to add it ourselves.


(Julien Henry) #12

Hi folks,

As mentioned here the behavior you describe really looks like there is a problem regarding webhook configuration. Please ensure that Jenkins is correctly receiving a payload sent by the SonarQube server when an analysis report is processed.


Intermittent waitForQualityGate timeouts
(Idan Adar) #13

Can you please describe here (not via a link to docs) what you’re expecting our setup to be like, and how to properly verify the payload?


(David Karr) #14

@Julien_HENRY , I have some of the same questions that @IdanAdar has. I don’t administrate either our Jenkins or SonarQube servers, so I don’t have direct access to the system, but I haven’t seen anything in the Jenkins log that looks like a webhook call. We also are unfortunately using a version of SonarQube before they added more diagnostics on the SonarQube side about if or when the webhook is called. From what I can see, we’re blind on both sides to whether the webhook call is even made, whether it’s successful or not.


(Nicolas Bontoux) #15

Hi @David_Karr,

In case it can help: note that prior to the Webhook Administration console being available (since 7.1), you still had one way to dig a bit more on the SonarQube side: the api/webhooks/deliveries WebAPI. Check out the documentation embedded in your SonarQube Server, you’ll see that this API can give you visibility into delivery status and http error code if any.


(David Karr) #16

Excellent. That was exactly what I needed. Now I know why it’s not working (401).


(Idan Adar) #17

I think my webhook was incorrect as well.
I see in the webhooks console Server Unreachable.

Should the value be the Jenkins server URL or the SonarQube server URL?
I’ve changed it to the Jenkins server URL, e.g. https://****-jenkins.****.com/sonarqube-webhook in the webhook settings - that’s the only place I see a configuration of a webhook/URL.


(Julien Henry) #18

On Jenkins side, we are logging in INFO each time a payload is received.


(Julien Henry) #19

The webhook should be sent by SonarQube to Jenkins. So indeed you should go in SonarQube administration UI, and configure a new Webhook that point to a Jenkins server URL that is reachable by the SonarQube server.