Hello community ,
I’m using self-hosted SonarQube Enterprise edition 188.8.131.52563 and experiencing an odd behavior regarding Scanner uploads to SonarQube during the last few days. This started a day after having a server freeze that took SonarQube down out of the sudden. There were some automatic attempts to restart the application during the outage that ended up in Elasticsearch shard errors. Unfortunately, I don’t have these error logs anymore. After the server issue was resolved, we were able to start SonarQube without any issues and in general there are no issues.
That said, here’s the odd behavior we experienced twice since that server freeze (2 days ago as of writing):
- Jenkins generates a project analysis (for a pull request) using the official SonarScanner
- Jenkins successfully posts that generated report file to SonarQube
- Jenkins waits for the quality gate result from SonarQube and receives
404 Not Founderrors for the successfully pushed report
Searching for the pull request in the project within SonarQube reveals that there is no such pull request analysis. Looking into the background tasks of the whole instance there is no task matching the ID that Jenkins was waiting for.
My first thought: There is a network issue and the POST request got lost or dropped somewhere. Surprisingly there is no evidence that this is the case. NGINX that handles all requests show that Jenkins successfully posted the report file to SonarQube. I can see the request in the NGINX logs. SonarQube itself logged a success for that request in its
access.log but there is no entry inside
ce.log indicating that the report was actually processed. The latter explains why there is no background task in the UI, right?
What’s even more concerning: One of the two times this happened there was another report upload for the same project (different pull request) at the very same second which was processed properly. There is no error log around that time.
Interestingly the scenario for the other occurrence (different SonarQube project) is completely different. There wasn’t even another report upload +/- 5 minutes around the missing one and no other background task seem to be running during that time.
I have temporarily enabled debug log level for the instance and am waiting for another occurrence that hopefully never happen.
Some thoughts I also had:
- We have 2 workers for background tasks. Looks like this is no concurrency issue.
- Maybe the maximum number of connections is reached? - But this wouldn’t explain the success entry in the logs, right?
- Is there an undetected issue with the Elasticsearch data in es7 directory and we should’ve cleared it before starting SonarQube after the server issue?
I’d be extremely happy for any ideas how I can better troubleshoot this somewhat random issue. Is there anything I am missing or anything you’d need from me that helps?
I’ve seen Upload report fails. Please help! but that seem to not cover my situation.