waitForQualityGate timeout in Jenkins

SonarQube Server / Community Build
,
1

This is my stage:

stage ("SonarQube analysis") {  
        options {
            timeout(time: 5, unit: 'MINUTES')
            retry(2)
        }
    
        when {
          allOf {
            changeRequest()
          }
        }
      
        steps {
          script {
            STAGE_NAME = "SonarQube analysis"
      
            withSonarQubeEnv('****') {
              sh "../../../sonar-scanner/bin/sonar-scanner"
            }
      
            qualitygate = waitForQualityGate()
            if (qualitygate.status != "OK") {
              currentBuild.result = "FAILURE"
              slackSend (channel: '****', color: '#F01717', message: "*$JOB_NAME*, <$BUILD_URL|Build #$BUILD_NUMBER>: Code coverage threshold was not met! <http://****.com:9000/sonarqube/projects|Review in SonarQube>.")
            }
          }
        }
      
        post {
          aborted {
            slackSend (channel: '****', color: '#F01717', message: "*$JOB_NAME*, <$BUILD_URL|Build #$BUILD_NUMBER>: '$STAGE_NAME' stage timed out, re-start the job.")
          }
        }
      }

Many times, this gets stuck and the log shows “IN_PROGRESS”.
Because this happens so often, I’ve added the following to limit its duration and retrying twice before giving up…

options {
    timeout(time: 5, unit: 'MINUTES')
    retry(2)
}

Is there any way to debug these timeouts?

1 Like
2

Hi Idan,

Seems to me that you first need to understand at which precise point is the stage timing out.

I can also see that you’re using waitForQualityGate, which as documented:

The waitForQualityGate step will pause the pipeline until SonarQube analysis is completed and returns quality gate status.

So you’d have to check if it’s not just a matter of the Background Task being queued while other tasks are ongoing. Whether it’s that or something else, understanding when the timeout occurs will help you narrow this down.

3

Hey @NicoB,

There are no other jobs running at the same time.
Can I see somewhere how long did a particular background task took? maybe the analysis takes longer than the allotted timeout in the pipelone.

4

Seems they were completed quite fast, and the stage timeout is 5 minutes.
What am I missing in understanding the whole flow?

46
461656×436 45.5 KB

{
task: {
id: "AWWFqxfzLykuh2LH3Urg",
type: "REPORT",
componentId: "AVr0gAOjyohgD3UCfaIn",
componentKey: "****",
componentName: "****",
componentQualifier: "TRK",
analysisId: "AWWFqxwkZLXM20NZPMoT",
status: "SUCCESS",
submittedAt: "2018-08-29T15:31:47+0300",
submitterLogin: "admin",
startedAt: "2018-08-29T15:31:48+0300",
executedAt: "2018-08-29T15:31:50+0300",
executionTimeMs: 2652,
logs: false,
hasScannerContext: true,
organization: "default-organization"
}
}

Checking status of SonarQube task ‘AWWFqxfzLykuh2LH3Urg’ on server ‘****’
SonarQube task ‘AWWFqxfzLykuh2LH3Urg’ status is ‘IN_PROGRESS’
Cancelling nested steps due to timeout

It also happens several times in a row for the same job… until it for some reason decides to pass.

6

@NicoB Anything you can add based on the above? Right now we cannot use sonar due to this.

7

Do you feel observing similar situation as in the below thread ? Would be interesting to know if a pattern emerges for those separate reports.

8

Just to chime in on a post that I hadn’t noticed before that my post was included in, I would emphasize that I saw the same behavior that you did. I don’t know what led me to try adding a “sleep(10)” between the steps, but it seemed like a situation where I was calling “waitForQualityGate” before something was properly ready, so adding the sleep seemed to make sense, and it appears to resolve the issue.

I’m only doing this in one project as an experiment. We were previously using the BuildBreaker, which has now been removed, and I’ve set up the webhook and the withSonarQubeEnv/waitForQualityGate pair (with the sleep) in this project. I’m currently mystified about this behavior with the required sleep, and I’m even more concerned about the webhook. The documentation I’ve read says this is part of the solution, but nothing describes exactly HOW it’s part of the solution. How is the payload sent by the webhook related to the management of these tasks? In fact, in one test case for this, I don’t even have the webhook installed at all, and I can’t see that it makes any difference at all.

2 Likes
9

I will try with the sleep workaround; for us it is happening with all projects, no matter their size.
Will report back.

withSonarQubeEnv('...') {
  sh "../../../sonar-scanner/bin/sonar-scanner"
}

sleep(10)
qualitygate = waitForQualityGate()
if (qualitygate.status != "OK") {
    ...
}
3 Likes
10

The sleep(10) tweak seems to have done the trick thus far.
Perhaps some delay option (configurable or not) could be introduced so we won’t need to add it ourselves.

1 Like
12

Hi folks,

As mentioned here the behavior you describe really looks like there is a problem regarding webhook configuration. Please ensure that Jenkins is correctly receiving a payload sent by the SonarQube server when an analysis report is processed.

13

Can you please describe here (not via a link to docs) what you’re expecting our setup to be like, and how to properly verify the payload?

1 Like
14

@Julien_HENRY , I have some of the same questions that @IdanAdar has. I don’t administrate either our Jenkins or SonarQube servers, so I don’t have direct access to the system, but I haven’t seen anything in the Jenkins log that looks like a webhook call. We also are unfortunately using a version of SonarQube before they added more diagnostics on the SonarQube side about if or when the webhook is called. From what I can see, we’re blind on both sides to whether the webhook call is even made, whether it’s successful or not.

15

Hi @David_Karr,

In case it can help: note that prior to the Webhook Administration console being available (since 7.1), you still had one way to dig a bit more on the SonarQube side: the api/webhooks/deliveries WebAPI. Check out the documentation embedded in your SonarQube Server, you’ll see that this API can give you visibility into delivery status and http error code if any.

16

Excellent. That was exactly what I needed. Now I know why it’s not working (401).

1 Like
17

I think my webhook was incorrect as well.
I see in the webhooks console Server Unreachable.

Should the value be the Jenkins server URL or the SonarQube server URL?
I’ve changed it to the Jenkins server URL, e.g. https://****-jenkins.****.com/sonarqube-webhook in the webhook settings - that’s the only place I see a configuration of a webhook/URL.

1 Like
18

On Jenkins side, we are logging in INFO each time a payload is received.

1 Like
19

The webhook should be sent by SonarQube to Jenkins. So indeed you should go in SonarQube administration UI, and configure a new Webhook that point to a Jenkins server URL that is reachable by the SonarQube server.

1 Like
20

I am having the same issues and the sleep(10) did the trick as well!

21

Hi,

if all is configured properly, it works without sleep.
We use a timeout of 10 mins and it’s sufficient for all projects.
Server unreachable means the webhook could not be delivered from Sonarqube to Jenkins.
From my experience those problems stem from proxy configuration and firewall permissions.

Note that the communication has to work in both ways:
Jenkins > Sonarqube, upload of scan results, then Jenkins plugin does an initial poll for the scan result, if result is already available, webhook is not needed anymore - therefore sleep helps in some cases.
Sonarqube > Jenkins for webhook delivery.

Usualy it’s something like i.e.
We have one central Sonarqube server, that is accessed by a lot of Jenkins nodes from different
network areas. All webhooks worked fine, but the delivery to Jenkins in the same network did not.
You need to configure the nonProxyHosts settings in ../conf/sonar.properties for this Jenkins instances.
Before Sonarqube 7.8 this had to be done via sonar.ce.javaOpts ! now there’s a dedicated property
http.nonProxyHosts .
For the upload of the scanner results toSonarqube server, the proxy has to accept the contents
of the report archive.

Gilbert

22

It looks like webhooks are being sent before the status of job is set to SUCCES. Moreover pull request decorations are also between. So if e.g. decoration takes more time, sonar will be still in IN PROGRESS state even webhooks were already sent.
If meantime waitForQualityGate is started, the first check of job status will return IN PROGRESS. However hooks will not be sent once again. So it is why deadlock.
Can you confirm that this is the way sonar works?

3 Likes