I’ve been working on migrating our builds from using the BuildBreaker plugin to the webhook and the withSonarQubeEnv/waitForQualityGate pipeline steps. I have a concern about the purpose of the webhook url that I posted in a separate question, but I wanted to point out an issue I’m having with the two pipeline steps.
The process appears to be working, but ONLY if I have a “sleep(10)” (haven’t experimented with smaller delays) between them. If I don’t have the sleep, the “waitForQualityGate” prints “IN PROGRESS” and waits forever. If I add the sleep, the first thing it prints is “PENDING”, and then a few seconds later it reports completion.
First it checks whether the quality gate check is already complete. It immediately returns if it has.
If not it then waits until sonar sends the webhook call to jenkins server. (hence why adding the sleep seems to work because sonar has already finished its analysis by that time)
If the webhook never fires (isn’t configured correctly/ or cannot communicate/plugin bug) from sonar to jenkins the call stays stuck and hung until the timeout is hit.
@SMUnlimited: if that’s how it’s supposed to work, my experience is certainly odd.
The thing about the webhook is that in the documentation I’ve read, it’s not clear to me what the purpose of the webhook is. In fact, in the testcase I was working with yesterday, the SonarQube project doesn’t even have the webhook url set. A similar project using the same Jenkinsfile does have it set, but since we’re using SonarQube 6.7.5, before better webhook diagnostics were provided, we have no way to know whether the webhook call is successfull, or even being made at all.
@SMUnlimited is right. What you describe really look like the webhook is not correctly sent from SonarQube to Jenkins.
it’s not clear to me what the purpose of the webhook is
The old Build breaker used to work by polling the SonarQube server. As you certainly know, polling is not very efficient, especially if analysis duration is long.
The waitForQualityGate step polls only once (for various reasons) and then pause the pipeline, waiting for a webhook to be received.
Sorry, but I think you are incorrect about one detail here. The “waitForQualityGate” step is not waiting for “a webhook to be received”, it’s waiting for a task to be completed. I know this because I tested this with a project that does not have any webhook set at all. The task completed, and “waitForQualityGate” returned. No webhook involved at all.
I find this very mystifying. The documentation states that the webhook is key to getting this to work, but I don’t understand why. It’s clear that the sonar scan starts a task that “waitForQualityGate” is waiting for, which doesn’t have anything to do with the webhook.
I also heard from my sonarqube admin that he has the ability to set “how often” the webhook runs, which is not a config I can see as a normal user. I don’t understand that. I would think the point of the webhook would be for it to run instantly, as soon as the task is completed.
From the latest updates, I personally feel like the ‘mystifying’ part is explained by this:
Meaning that if the Background Task has already completed by the time that first poll is done, then that first poll will put an end to the waitForQualityGate step (independently of any webhook). Also explains why a sleep before that first poll increases the chance of success, as it lets more time for the Background Task to complete. If the task is still pending and/or in progress, then waitForQualityGate step would then wait for the webhook (and a timeout could be explained by a webhook delivery issue).
Just sharing my understanding of past updates, am sure @Julien_HENRY will correct me if needed.
You’re right @NicoB. If the analysis is fast enough to be completed by the time the waitForQualityGate starts, the pipeline will succeed, even with no webhook configured.
If I’m understanding this correctly, you’re saying that this is essentially the logic of “waitForQualityGate”:
If task is complete:
return task status
else
wait for webhook
return task status
fi
If that’s what you’re saying, then I can understand all of the behavior that I’m seeing.
I would think by now it should be obvious that the logic of this should be clearly documented somewhere so we can see it. I’ve not seen anything that describes this.
What’s still not clear to me, however, is the timing of when the webhook is called. My SonarQube admin said that they have the ability to set “how often” the webhook is called. That seems odd to me. I would think the webhook would be called instantly on completion of the task. Is that not how it works?
It seems odd to me too. I confirm that Webhooks are sent on task completion (can also happen in case of issue edit). It’s unclear to me what your admin refers to by ‘the ability to set “how often” the webhook is called’. I would invite you to get more details of what he means, or just get him to participate to this thread!
I’ll try to improve documentation. What do you think of:
Note that to prevent race conditions, when the step starts (or is restarted) a direct call is made to the server to check if the task is already completed.
I need to resurrect this discussion. We are seeing bad behavior again here. We are seeing the background task complete in a few seconds, but the Jenkins plugin gets ‘IN_PROGRESS’ and hangs forever. I’ve verified the task id. It started and completed at the time it was requested, but Jenkins never got a notice of the completion.
I’m thinking the only way to deal with this is to have multiple calls to “waitForQualityGate” in a row, with the first N having a very short timeout, and ignoring the timeout error in those first N calls. I don’t know what else to do here.
I tried that approach, and it didn’t help. I made the first call in a 1 minute timeout block. That got “PENDING” and timed out. The second call with a longer timeout also timed out, just getting “PENDING”.
Curiously, I noticed that the task itself was actually still in the PENDING state in SonarQube, which is very odd, as they usually complete within a minute. I’m trying it again without the odd retry mechanism.
Now that I realized that I had to change the Status from “All except Pending” to “All”, I’m seeing that the task gets a submitted timestamp, but never gets a “Started” timestamp.
I don’t know what was happening, but this was a local problem. I’ve now concluded that my error checking for “waitForQualityGate()” is going to have to respect the fact that this kind of failure is going to happen in our environment from time to time. I’m now wrapping it in a timeout block with a relatively short timeout, and not failing the build if it times out (but logging and notifying this fact).
Thanks for your discussions here! By configured the webhook, my pipelines are working great! Before that it is always paused in the waitForQualityGate step by claming “task is IN_PROGRESS”. My webhooks are configured as following:
I was having this problem too. If I let waitForQualityGate() run it would see “IN PROGRESS” and then hang until the timeout. Worked around it by adding a sleep until the analysis was complete, which defeats the purpose of using a webhook.
In case it help anyone else, my problem was that Jenkins wasn’t actually receiving the webook request. When I looked in Sonar I could see the recent deliveries of the webhook all showed “Response: Server Unreachable”.
I still don’t know exactly why because when I recreated the hook request manually using cURL from the sonar server everything worked fine. But I tweaked the URL a bit (I was using a hostname and port but changed it to the proper DNS name) and now the webook works as expected.
