Post upgradation from7.9 to 8.9 background tasks taking more time

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension) : sonarqube 8.9
  • what are you trying to achieve - Reduce time taken to run the background task
  • what have you tried so far to achieve this - increased worker count to 2, current CE value - sonar.ce.javaOpts=-Xmx4G -Xms4G -XX:+HeapDumpOnOutOfMemoryError

CE log:
2021.06.17 12:31:48 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=11339ms
2021.06.17 12:31:48 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=4ms
2021.06.17 12:31:48 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
2021.06.17 12:31:48 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2021.06.17 12:31:48 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=8ms
2021.06.17 12:31:48 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=8ms
2021.06.17 12:31:49 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=21544 | status=SUCCESS | time=876ms
2021.06.17 12:31:49 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=3ms
2021.06.17 12:31:49 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Load quality profiles | status=SUCCESS | time=216ms
2021.06.17 12:31:49 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Load Quality gate | status=SUCCESS | time=7ms
2021.06.17 12:31:49 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Load new code period | status=SUCCESS | time=3ms
2021.06.17 12:31:49 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Detect file moves | status=SUCCESS | time=0ms
2021.06.17 12:31:50 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Load duplications | duplications=2741 | status=SUCCESS | time=239ms
2021.06.17 12:31:50 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute cross project duplications | status=SUCCESS | time=0ms
2021.06.17 12:31:51 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute size measures | status=SUCCESS | time=818ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=SUCCESS | time=55346ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute coverage measures | status=SUCCESS | time=388ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute comment measures | status=SUCCESS | time=35ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Copy custom measures | status=SUCCESS | time=2ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute duplication measures | status=SUCCESS | time=43ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute size measures on new code | status=SUCCESS | time=41ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute language distribution | status=SUCCESS | time=37ms
2021.06.17 12:32:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute test measures | status=SUCCESS | time=35ms
2021.06.17 12:32:47 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute complexity measures | status=SUCCESS | time=63ms
2021.06.17 12:32:47 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Load measure computers | status=SUCCESS | time=0ms
2021.06.17 12:32:47 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute Quality Profile status | status=SUCCESS | time=9ms
2021.06.17 12:35:15 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Execute component visitors | status=SUCCESS | time=148132ms
2021.06.17 12:35:15 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Checks executed after computation of measures | status=SUCCESS | time=18ms
2021.06.17 12:35:15 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute Quality Gate measures | status=SUCCESS | time=0ms
2021.06.17 12:35:15 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Compute Quality profile measures | status=SUCCESS | time=6ms
2021.06.17 12:35:15 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Generate Quality profile events | status=SUCCESS | time=3ms
2021.06.17 12:35:15 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Generate Quality gate events | status=SUCCESS | time=2ms
2021.06.17 12:35:15 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Check upgrade possibility for not analyzed code files. | status=SUCCESS | time=1ms
2021.06.17 12:35:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=31632ms
2021.06.17 12:35:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=3ms
2021.06.17 12:35:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=4ms
2021.06.17 12:35:46 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=52 | status=SUCCESS | time=15ms
2021.06.17 12:37:38 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist live measures | insertsOrUpdates=506055 | status=SUCCESS | time=111511ms
2021.06.17 12:37:38 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist duplication data | insertsOrUpdates=404 | status=SUCCESS | time=396ms
2021.06.17 12:37:38 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist new ad hoc Rules | status=SUCCESS | time=0ms
2021.06.17 12:38:34 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist issues | cacheSize=140 MB | inserts=166445 | updates=0 | merged=0 | status=SUCCESS | time=55281ms
2021.06.17 12:38:34 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist project links | status=SUCCESS | time=0ms
2021.06.17 12:38:34 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist events | status=SUCCESS | time=5ms
2021.06.17 12:40:12 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=98057ms
2021.06.17 12:40:12 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=SUCCESS | time=0ms
2021.06.17 12:40:12 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Enable analysis | status=SUCCESS | time=64ms
2021.06.17 12:40:12 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Update last usage date of quality profiles | status=SUCCESS | time=13ms
2021.06.17 12:40:12 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Purge db | status=SUCCESS | time=134ms
2021.06.17 12:40:41 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Index analysis | status=SUCCESS | time=28609ms
2021.06.17 12:40:41 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Update need issue sync for branch | status=SUCCESS | time=4ms
2021.06.17 12:40:41 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Send issue notifications | newIssuesNotifs=0 | newIssuesDeliveries=0 | myNewIssuesNotifs=0 | myNewIssuesDeliveries=0 | changesNotifs=0 | changesDeliveries=0 | status=SUCCESS | time=3ms
2021.06.17 12:40:41 INFO ce[AXoZ7kqRBZAtIs07GGM1][o.s.c.t.s.ComputationStepExecutor] Publish task results | status=SUCCESS | time=0ms

Time taken for
Execute component visitors | status=SUCCESS | time=148132ms
Persist issues | cacheSize=140 MB | inserts=166445 | updates=0 | merged=0 | status=SUCCESS | time=55281ms

how do i reduce time for this

@dmeneses we are currently facing this issue in our project. We want this operation to be reduced because we do have multiple developers submitting sonar scans, and for a specific project, there will be only 1 scan in processing, the rest are pending.

After analysis, it appears that mainly these operations are taking time:

  • Execute component visitors | status=SUCCESS | time=66506ms
  • Persist issues | cacheSize=157 MB | inserts=1 | updates=166439 | merged=0 | status=SUCCESS | time=40220ms
  • Extract report | status=SUCCESS | time=11339ms

Is there any recommendation to have mainly this time reduced to only a few seconds instead of minutes? I want to highlight that sonar.sources=“is only the changed files

Thank you for your help in advance :slight_smile:

Hi,
The time to get an analysis done depends on some many variables: size of the project, hardware used, languages being analyzed, DB, etc. So the duration of those operations doesn’t say much without the context. In any case, I don’t think there’s anything we can do to reduce the time from minutes to seconds. What I can say is that “Persist issues” is mostly spend writing to the DB while “Extract report” is mostly hard drive I/O operations.
I also don’t recommend setting sonar.sources to changed files only. You might miss issues in those changed files by not having other related files in the analysis.

Hi @dmeneses Thank you for your reply, actually, if we set source=All accepted folders, the scan takes more than 25mn, because in the project we have more than 500K lines and large files.

With the new Code scan we believe that for developers we need to provide feedback only on what was updated and scan only what was changed, how can this be wrong?

regarding the Persist issues, we’ll see if this can be improved somehow.