SonarQube WebHook Fails with server unreachable until sonarQube Restart

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
  • what are you trying to achieve
  • what have you tried so far to achieve this

SonarQube version 8.7.1
WebHook delivery failure after running fine for 1-3 days

We have been seeing this issue since we started using sonarqube at 7.9.1. Increasing the memory
of the CE Java process does not seem to help. We have a webhook that will deliver payloads successfully for 2 hours to 4 days, then it stops with server unreachable and the only way to get successful deliveries is to restart the sonarqube services. If the delivery truly is failing due to a server
unreachable, it should recover on the next delivery if the server is reachable, but it does not ( to my knowledge the issue has never been that the server is unreachable ). I have seen various threads on this issue and the resolution is always fixed after restarting the sonarqube services ( not a great solution ).

I have opened several tickets and other than the suggestion to turn the CE logging to debug so we can see the failures there has been no solution ( other than to restart the sonarqube processes ).

Restarting the sonarqube server works 100% of the time once you detect the failure but turning on DEBUG to see the failures in the logs is not a suitable way to monitor for the problem.

If there is a way to query to find out the current status we are willing to use that as a solution vs logging into the UI to see if it is in a failed state.

Thanks

Hi,

Welcome to the community!

Is this relevant?

 
Ann

Our Server endpoint that will receive the webhook payload never changes ( it resolves to an AWS ALB which is static ). I can look at a netstat next time it fails ( which will only be a mater of a hours to a few days ).

Typically AWS load balancers aren’t static IPs. Amazon scales the LB up/down or rotates the backing instances as needed. That is the situation that we ran into where the webhooks would work until AWS changed the backing infrastructure and therefore the IPs returned from DNS lookups. We are running SonarQube in a docker container and the container for 8.9 (the new LTS version) has a change to make it so java only caches DNS lookups for 5 seconds instead of infinity. It may not be your issue but it certainly sounds very similar to what we were seeing.

2 Likes

Thanks Kyle, Goint to make the changes in our DEV environment as it only takes a few days ( or hours ) for it to occur, should be an easy test.

Making the changes ttl changes fixed the problem. THANKS
image

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.