Hi,
We’re using Sonar 8.2.0 developer edition and have ±1.3M LOC, for some time now we’re seeing slowness in processing background tasks and it’s becoming a major bottleneck lately, reaching for delays in process of up to two days.
We’ve made improvements to the Postgres DB, the sonarqube instance and also I’ve increased the Xmx and Xmx of the CE worker, both instances are currently working under normal load but still the issue persists.
I’ve added the log of the CE where I saw that the main cause for the slow process of background tasks is the persist part, and specifically the Persist issues with the longest duration.
When a task is processed normally for us it takes under 40s, and the next task of the same build (so the code is the same) can take even up to an hour.
What can I do in order to improve the performance? where is this issue might be coming from?
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=2409ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=7ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=10ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Execute DB migrations for current project | status=SUCCESS | time=5ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=18ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=12ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Verify billing | status=SUCCESS | time=0ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=5378 | status=SUCCESS | time=387ms
2021.04.14 02:58:16 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=11ms
2021.04.14 02:58:17 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load quality profiles | status=SUCCESS | time=149ms
2021.04.14 02:58:17 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load Quality gate | status=SUCCESS | time=8ms
2021.04.14 02:58:17 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load new code period | status=SUCCESS | time=0ms
2021.04.14 02:58:17 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Detect file moves | status=SUCCESS | time=0ms
2021.04.14 02:58:17 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load duplications | duplications=629 | status=SUCCESS | time=163ms
2021.04.14 02:58:17 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute cross project duplications | status=SUCCESS | time=0ms
2021.04.14 02:58:17 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute size measures | status=SUCCESS | time=289ms
2021.04.14 02:58:18 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=SUCCESS | time=943ms
2021.04.14 02:58:18 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute coverage measures | status=SUCCESS | time=399ms
2021.04.14 02:58:18 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute comment measures | status=SUCCESS | time=11ms
2021.04.14 02:58:18 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Copy custom measures | status=SUCCESS | time=3ms
2021.04.14 02:58:18 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute duplication measures | status=SUCCESS | time=14ms
2021.04.14 02:58:18 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute size measures on new code | status=SUCCESS | time=43ms
2021.04.14 02:58:18 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute language distribution | status=SUCCESS | time=10ms
2021.04.14 02:58:19 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute test measures | status=SUCCESS | time=8ms
2021.04.14 02:58:19 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute complexity measures | status=SUCCESS | time=18ms
2021.04.14 02:58:19 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Load measure computers | status=SUCCESS | time=0ms
2021.04.14 02:58:19 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute Quality Profile status | status=SUCCESS | time=5ms
2021.04.14 03:04:40 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Execute component visitors | status=SUCCESS | time=381202ms
2021.04.14 03:04:40 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Checks executed after computation of measures | status=SUCCESS | time=132ms
2021.04.14 03:04:40 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute Quality Gate measures | status=SUCCESS | time=10ms
2021.04.14 03:04:40 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Compute Quality profile measures | status=SUCCESS | time=3ms
2021.04.14 03:04:40 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Generate Quality profile events | status=SUCCESS | time=5ms
2021.04.14 03:04:40 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Generate Quality gate events | status=SUCCESS | time=0ms
2021.04.14 03:04:47 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=6721ms
2021.04.14 03:04:47 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=6ms
2021.04.14 03:04:47 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=14ms
2021.04.14 03:04:47 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=86 | status=SUCCESS | time=16ms
2021.04.14 03:05:38 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist live measures | insertsOrUpdates=260021 | status=SUCCESS | time=51740ms
2021.04.14 03:05:38 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist duplication data | insertsOrUpdates=303 | status=SUCCESS | time=88ms
2021.04.14 03:05:38 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist new ad hoc Rules | status=SUCCESS | time=0ms
2021.04.14 03:17:50 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist issues | inserts=9638 | updates=0 | merged=0 | untouched=0 | status=SUCCESS | time=731093ms
2021.04.14 03:17:50 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist project links | status=SUCCESS | time=0ms
2021.04.14 03:17:50 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist events | status=SUCCESS | time=18ms
2021.04.14 03:18:07 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=17676ms
2021.04.14 03:18:07 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=SUCCESS | time=0ms
2021.04.14 03:18:07 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Enable analysis | status=SUCCESS | time=73ms
2021.04.14 03:18:07 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Update last usage date of quality profiles | status=SUCCESS | time=19ms
2021.04.14 03:18:08 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Purge db | status=SUCCESS | time=139ms
2021.04.14 03:18:09 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Index analysis | status=SUCCESS | time=1298ms
2021.04.14 03:18:09 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Send issue notifications | status=SUCCESS | time=0ms
2021.04.14 03:18:09 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Publish task results | status=SUCCESS | time=0ms
2021.04.14 03:18:09 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.s.ComputationStepExecutor] Trigger refresh of Portfolios and Applications | status=SUCCESS | time=0ms
2021.04.14 03:18:09 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Webhooks | globalWebhooks=0 | projectWebhooks=0 | status=SUCCESS | time=9ms
2021.04.14 03:18:09 INFO ce[AXjLT8Rf_OC4pcvzcu6D][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Pull Request decoration | status=SUCCESS | time=199ms