Slow background tasks process - Persist issues

Hi,

We’re using Sonar 8.2.0 developer edition and have ±1.3M LOC, for some time now we’re seeing slowness in processing background tasks and it’s becoming a major bottleneck lately, reaching for delays in process of up to two days.
We’ve made improvements to the Postgres DB, the sonarqube instance and also I’ve increased the Xmx and Xmx of the CE worker, both instances are currently working under normal load but still the issue persists.
I’ve added the log of the CE where I saw that the main cause for the slow process of background tasks is the persist part, and specifically the Persist issues with the longest duration.

When a task is processed normally for us it takes under 40s, and the next task of the same build (so the code is the same) can take even up to an hour.
What can I do in order to improve the performance? where is this issue might be coming from?

2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=2409ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=7ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=10ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Execute DB migrations for current project | status=SUCCESS | time=5ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=18ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=12ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Verify billing | status=SUCCESS | time=0ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=5378 | status=SUCCESS | time=387ms
2021.04.14 02:58:16 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=11ms
2021.04.14 02:58:17 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load quality profiles | status=SUCCESS | time=149ms
2021.04.14 02:58:17 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load Quality gate | status=SUCCESS | time=8ms
2021.04.14 02:58:17 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load new code period | status=SUCCESS | time=0ms
2021.04.14 02:58:17 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Detect file moves | status=SUCCESS | time=0ms
2021.04.14 02:58:17 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load duplications | duplications=629 | status=SUCCESS | time=163ms
2021.04.14 02:58:17 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute cross project duplications | status=SUCCESS | time=0ms
2021.04.14 02:58:17 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute size measures | status=SUCCESS | time=289ms
2021.04.14 02:58:18 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=SUCCESS | time=943ms
2021.04.14 02:58:18 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute coverage measures | status=SUCCESS | time=399ms
2021.04.14 02:58:18 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute comment measures | status=SUCCESS | time=11ms
2021.04.14 02:58:18 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Copy custom measures | status=SUCCESS | time=3ms
2021.04.14 02:58:18 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute duplication measures | status=SUCCESS | time=14ms
2021.04.14 02:58:18 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute size measures on new code | status=SUCCESS | time=43ms
2021.04.14 02:58:18 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute language distribution | status=SUCCESS | time=10ms
2021.04.14 02:58:19 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute test measures | status=SUCCESS | time=8ms
2021.04.14 02:58:19 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute complexity measures | status=SUCCESS | time=18ms
2021.04.14 02:58:19 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load measure computers | status=SUCCESS | time=0ms
2021.04.14 02:58:19 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute Quality Profile status | status=SUCCESS | time=5ms
2021.04.14 03:04:40 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Execute component visitors | status=SUCCESS | time=381202ms
2021.04.14 03:04:40 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Checks executed after computation of measures | status=SUCCESS | time=132ms
2021.04.14 03:04:40 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute Quality Gate measures | status=SUCCESS | time=10ms
2021.04.14 03:04:40 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute Quality profile measures | status=SUCCESS | time=3ms
2021.04.14 03:04:40 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Generate Quality profile events | status=SUCCESS | time=5ms
2021.04.14 03:04:40 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Generate Quality gate events | status=SUCCESS | time=0ms
2021.04.14 03:04:47 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=6721ms
2021.04.14 03:04:47 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=6ms
2021.04.14 03:04:47 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=14ms
2021.04.14 03:04:47 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=86 | status=SUCCESS | time=16ms
2021.04.14 03:05:38 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist live measures | insertsOrUpdates=260021 | status=SUCCESS | time=51740ms
2021.04.14 03:05:38 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist duplication data | insertsOrUpdates=303 | status=SUCCESS | time=88ms
2021.04.14 03:05:38 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist new ad hoc Rules | status=SUCCESS | time=0ms
2021.04.14 03:17:50 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist issues | inserts=9638 | updates=0 | merged=0 | untouched=0 | status=SUCCESS | time=731093ms
2021.04.14 03:17:50 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist project links | status=SUCCESS | time=0ms
2021.04.14 03:17:50 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist events | status=SUCCESS | time=18ms
2021.04.14 03:18:07 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=17676ms
2021.04.14 03:18:07 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=SUCCESS | time=0ms
2021.04.14 03:18:07 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Enable analysis | status=SUCCESS | time=73ms
2021.04.14 03:18:07 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Update last usage date of quality profiles | status=SUCCESS | time=19ms
2021.04.14 03:18:08 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Purge db | status=SUCCESS | time=139ms
2021.04.14 03:18:09 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Index analysis | status=SUCCESS | time=1298ms
2021.04.14 03:18:09 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Send issue notifications | status=SUCCESS | time=0ms
2021.04.14 03:18:09 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Publish task results | status=SUCCESS | time=0ms
2021.04.14 03:18:09 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Trigger refresh of Portfolios and Applications | status=SUCCESS | time=0ms
2021.04.14 03:18:09 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Webhooks | globalWebhooks=0 | projectWebhooks=0 | status=SUCCESS | time=9ms
2021.04.14 03:18:09 INFO  ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Pull Request decoration | status=SUCCESS | time=199ms

Hey there.

  1. There have been heaps of performance improvements over the v8.x series including SONAR-13093 which address the Persist Issues step
  2. A good first step would be to look at your database and understand if there are any long-running queries, Memory/CPU spikes. A VACUUM FULL ANALYZE can often do the trick as well. If queries are running quickly, we should turn our eyes to your SonarQube instance itself.

In any case, you should make your way to a newer, supported version of SonarQube (v8.8, v8.9 LTS in the next few weeks) A non-LTS version of SonarQube is only supported until a newer version is released.

@Colin We have recently upgraded from 7.9 to 8.9 and we are actually having this same issue on

  • Execute component visitors | status=SUCCESS | time=66506ms
  • Persist issues | cacheSize=157 MB | inserts=1 | updates=166439 | merged=0 | status=SUCCESS | time=40220ms

We have tried to check what’s wrong in the Database, but nothing seems to be wrong,