Scanner report successfully uploaded but not processed in SonarQube

Hello community :wave:,

I’m using self-hosted SonarQube Enterprise edition 9.7.0.61563 and experiencing an odd behavior regarding Scanner uploads to SonarQube during the last few days. This started a day after having a server freeze that took SonarQube down out of the sudden. There were some automatic attempts to restart the application during the outage that ended up in Elasticsearch shard errors. Unfortunately, I don’t have these error logs anymore. After the server issue was resolved, we were able to start SonarQube without any issues and in general there are no issues.

That said, here’s the odd behavior we experienced twice since that server freeze (2 days ago as of writing):

  • Jenkins generates a project analysis (for a pull request) using the official SonarScanner
  • Jenkins successfully posts that generated report file to SonarQube
  • Jenkins waits for the quality gate result from SonarQube and receives 404 Not Found errors for the successfully pushed report

Searching for the pull request in the project within SonarQube reveals that there is no such pull request analysis. Looking into the background tasks of the whole instance there is no task matching the ID that Jenkins was waiting for.

My first thought: There is a network issue and the POST request got lost or dropped somewhere. Surprisingly there is no evidence that this is the case. NGINX that handles all requests show that Jenkins successfully posted the report file to SonarQube. I can see the request in the NGINX logs. SonarQube itself logged a success for that request in its access.log but there is no entry inside ce.log indicating that the report was actually processed. The latter explains why there is no background task in the UI, right?

What’s even more concerning: One of the two times this happened there was another report upload for the same project (different pull request) at the very same second which was processed properly. There is no error log around that time.

Interestingly the scenario for the other occurrence (different SonarQube project) is completely different. There wasn’t even another report upload +/- 5 minutes around the missing one and no other background task seem to be running during that time.

I have temporarily enabled debug log level for the instance and am waiting for another occurrence that hopefully never happen. :wink:

Some thoughts I also had:

  • We have 2 workers for background tasks. Looks like this is no concurrency issue.
  • Maybe the maximum number of connections is reached? - But this wouldn’t explain the success entry in the logs, right?
  • Is there an undetected issue with the Elasticsearch data in es7 directory and we should’ve cleared it before starting SonarQube after the server issue?

I’d be extremely happy for any ideas how I can better troubleshoot this somewhat random issue. Is there anything I am missing or anything you’d need from me that helps?

I’ve seen Upload report fails. Please help! but that seem to not cover my situation.

Best regards,
Steven

2 Likes

Hey there

Thanks for the detailed report. :slight_smile:

Is there any chance that after the freeze with various services restarts you accidentally ended up with two SonarQube instances/services both connected to the same database? This can cause some weird behavior?

You might check connections on the Database side, or just run a jps command on your server to get an idea of what processes are running.

Hi @Colin, thanks for the hint. I’ve checked connections on the database side for both of our instances (production, staging). The production instance was the one having the shutdown and auto-restarts, the staging instance is hosted on another server and was not affected by the incident and doesn’t show the odd behavior.

Both instances have the same amount of connections: 21 (as of writing this).
At that time of the day both instances are quite idle. Not sure if it is common connection count for idle instances?

We are using the official Docker image where jps seems to be missing. The openjdk alpine package should provide it though. Is there some SonarQube internal way of accessing the amount of processes?

Hey there.

Thanks for the follow-up.

If you’re using Docker, I’m not worried about there being multiple processes running within that container. Although it does make me wonder if somehow, around the time of the incident you’re describing, a second container was started somehow targeting the same database.

I wouldn’t worry about connection count as it’s displayed in SonarQube (it very rarely comes into play, even on huge instances), but more about connections to the database overall (which you find on the database-side, not SonarQube-side). This is what would indicate if something else, like another instance, was connecting to the database.

I also wouldn’t worry about Elasticsearch as it only comes into play much later during report processing.

Have you experienced the lost CE tasks since the 2 hiccups earlier this week (i.e. – is the issue still recurring?)