which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
We originally had Sonarqube functioning with version 7.3 community and then 8.2 developer editions but found that after a few months the background tasks started being canceled more often than not.
Plugins in use are the stock plugins of 8.2 and 8.3
The scans are initiated from Azure devops using the Sonarqube plugin yaml tasks
what are you trying to achieve
Trying to restore basic functionality of sonarqube
what have you tried so far to achieve this
Attempted to upgrade the sonarqube server to 8.3.
Re-built all indexes
Dropped the SQL server catalog and started from scratch
Redirected traffic to a staging SQL server - This allowed the scans to complete every time we tested, but we have not been able to identify why the change in sql server would cause the tasks to be canceled
I have attached today’s trace log and a screenshot of what we’ve been seeing. sonarqube_ce (9).log (553.6 KB)
I checked your logs and indeed we can see SQL queries which detects these tasks as worn outs and remove them. I’m not familiar with all details here but it looks that the CE worker changed while there were in progress (why? this is a good question), so they had been canceled.
I’m surprised that your logs show only SQL queries, we don’t see any regular INFO/DEBUG logs. Maybe no tasks were submitted during this time.
They will be useful here because we could see for a canceled tasks, which workers it was started with and when it changed. So we could understand what went wrong.
Could you run in TRACE mode on your Prod DB for a while, make sure some tasks are submitted then canceled during that time, so then we could do a complete analysis?
I believe I have attached a log that correlates with the 2020-07-14 13:46:33 - 13:45:56 tasks that are displayed in the screen shot. Is there another log I should be including.
Yes you did, but what’s weird is that your logs contain almost exclusively TRACE messages (SQL queries most of the time), there is no usual INFO/DEBUG logs saying when tasks are submitted, when it starts, each step starting/finishing…
So I don’t know why, but something is missing:
either your logs are not the original SQ logs and had been filtered somewhat
none of the tasks were actually submitted during that 13:42 > 14:03 window
What I’d like to see is the whole lifecycle of a task which ends up being canceled: submission, when it starts (with which worker!), what happened before it gets canceled, etc.
Here we don’t have that.
That makes a lot of sense @rloic, thanks for sharing (and thanks @Cameron for the pointer ).
Indeed @Tarig0, this would also explain why your logs looked weird without any task submissions. Likely one instance is creating tasks while the other cancel them!
You should check that only one single SQ instance is connected to your database, so there is no concurrent instances doing concurrent work.
One thing that I would ask to be added to the product. Could there be a check added to ensure that multiple compute engines are not running against the same database, unless setup for enterprise?
May I ask you please to mark my comment or yours as the solution so it properly close the topic?
We’ve seen the issue so many times that we thought about it indeed. The truth is that a robust solution would be on the DB side to only allow connections from one host where SQ is installed, assuming 2 instances are not installed on one host. It is doable but it has to be implemented on the DB side anyway.
From the SQ application perspective, there is no real way to know that something else is doing stuff in the DB, so we can’t strictly prevent this to happen.
).