SonarQube Enterprise - Background tasks processing is stuck and requires restart

SonarQube Server / Community Build
1

Dear SonarQube community,

We are currently facing a recurring issue with our SonarQube server. For an unknown reason, after some time (can be days or weeks), a background task will be stuck and will run for hours without being processed (more than 8h and it seems to be infinite), which will completely stuck CE process and fail webhook calls.

We tried to increase the number of workers to 2, unfortunately, it’s just delaying the problem as the first worker will be stuck and after some time, the second worker will also be stuck on a second task.

  • SonarQube Enterprise v9.9.5 (build 90363)
  • how is SonarQube deployed: zip on Ubuntu 22.04 VM

We understood that the problem is located in the CE service and then enabled DEBUG logs on it to try to understand, here is the result of a faulty task. As you can see the zip report is never uncompress on the disk and the CE server will be stuck forever. The only workaround we found is to perform a “systemctl restart sonarqube”, it will take an abnormal amount of time (almost 5 minutes) when the problem occured but fix the issue, the CE server will process the “stucked task” and then continue processing the rest of the pending tasks.

2024.07.03 01:08:52 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=CENSORED-PROJECT | type=REPORT | branch=develop | branchType=BRANCH | id=AZB2IxbWo8ApsgAChpeV | submitter=admin
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.a.AnnotationConfigApplicationContext] Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@7e00408a
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'org.springframework.context.annotation.internalConfigurationAnnotationProcessor'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'org.springframework.context.event.internalEventListenerProcessor'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.container.LazyUnlessEagerAnnotationStrategy-org.sonar.ce.task.container.LazyUnlessEagerAnnotationStrategy@708be152AZB2IyIonq1Unc-ljnBJ'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'org.springframework.context.event.internalEventListenerFactory'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'org.springframework.context.annotation.internalAutowiredAnnotationProcessor'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'org.springframework.context.annotation.internalCommonAnnotationProcessor'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.core.platform.StartableBeanPostProcessor-org.sonar.core.platform.StartableBeanPostProcessor@668796a3AZB2IyIonq1Unc-ljnBK'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.setting.SettingsLoader'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.setting.SettingsLoader' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.server.setting.ThreadLocalSettings'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.step.ComputationStepExecutor'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.step.ReportComputationSteps-org.sonar.ce.task.projectanalysis.step.ReportComputationSteps@8644522AZB2IyIonq1Unc-ljnBM'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.CeTask-CeTask{type=REPORT, uuid=AZB2IxbWo8ApsgAChpeV, component=Component{uuid='AXmKANqKHgFsaAb2t50G', key='CENSORED-PROJECT', name='Core CENSORED-PROJECT'}, mainComponent=Component{uuid='AXmKANqKHgFsaAb2t50G', key='CENSORED-PROJECT', name='Core CENSORED-PROJECT'}, submitter=User{uuid='AW7qaq8u2r2kF2AfC77I', login='admin'}}'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.analysis.AnalysisMetadataHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.analysis.AnalysisMetadataHolderImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.core.platform.PlatformEditionProvider'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.qualitygate.QualityGateHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.qualitygate.QualityGateStatusHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.batch.BatchReportReaderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.batch.BatchReportDirectoryHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.batch.BatchReportReaderImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.batch.BatchReportDirectoryHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.server.webhook.WebhookPayloadFactoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.server.webhook.WebHooksImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.Z'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.ConfigurationRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.TreeRootHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.ConfigurationRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.TreeRootHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.ConfigurationRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.analysis.AnalysisMetadataHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.ConfigurationRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.analysis.ProjectConfigurationFactory'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.Z' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.ConfigurationRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.Z' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.c'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.Z' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.H'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.W'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.issue.RuleRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.issue.RuleRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.issue.AdHocRuleCreator'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.issue.RuleRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.db.DbClient'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.locations.flow.FlowGenerator'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.locations.flow.FlowGenerator' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.TreeRootHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.W' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.analysis.AnalysisMetadataHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.W' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.H'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.W' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.issue.RuleRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.W' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.locations.flow.FlowGenerator'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.measure.MeasureRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.MetricRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.MetricRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.db.DbClient'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.ReportMetricValidatorImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.core.metric.ScannerMetrics'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.core.metric.ScannerMetrics' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.server.metric.UnanalyzedLanguageMetrics'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.core.metric.ScannerMetrics' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.A.D.C.E.A'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.ReportMetricValidatorImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.core.metric.ScannerMetrics'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.measure.MeasureRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.db.DbClient'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.measure.MeasureRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.batch.BatchReportReaderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.measure.MeasureRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.MetricRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.measure.MeasureRepositoryImpl' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.ReportMetricValidatorImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.Z'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.async.SynchronousAsyncExecution'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.W'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.measure.MeasureRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.TreeRootHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.MetricRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.db.DbClient'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.J'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.J' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.Y'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.J' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.H'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.J'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.async.SynchronousAsyncExecution'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.W'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.measure.MeasureRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.component.TreeRootHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.metric.MetricRepositoryImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.CeTask-CeTask{type=REPORT, uuid=AZB2IxbWo8ApsgAChpeV, component=Component{uuid='AXmKANqKHgFsaAb2t50G', key='CENSORED-PROJECT', name='Core CENSORED-PROJECT'}, mainComponent=Component{uuid='AXmKANqKHgFsaAb2t50G', key='CENSORED-PROJECT', name='Core CENSORED-PROJECT'}, submitter=User{uuid='AW7qaq8u2r2kF2AfC77I', login='admin'}}'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.analysis.AnalysisMetadataHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.qualitygate.QualityGateHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.qualitygate.QualityGateStatusHolderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.batch.BatchReportReaderImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.d'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-com.sonarsource.G.D.k'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.step.ComputationStepExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.step.ReportComputationSteps-org.sonar.ce.task.projectanalysis.step.ReportComputationSteps@8644522AZB2IyIonq1Unc-ljnBM'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.step.ComputationStepExecutor' via constructor to bean named 'CeTaskInterrupter'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.step.ComputationStepExecutor' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.step.ExtractReportStep'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'ComputationTempFolder'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Creating shared instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.filesystem.ComputationTempFolderProvider-org.sonar.ce.task.projectanalysis.filesystem.ComputationTempFolderProvider@749c50b4AZB2IyIonq1Unc-ljnBN'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'ComputationTempFolder' via factory method to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.server.platform.ServerFileSystemImpl'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.step.ExtractReportStep' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.db.DbClient'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.step.ExtractReportStep' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.CeTask-CeTask{type=REPORT, uuid=AZB2IxbWo8ApsgAChpeV, component=Component{uuid='AXmKANqKHgFsaAb2t50G', key='CENSORED-PROJECT', name='Core CENSORED-PROJECT'}, mainComponent=Component{uuid='AXmKANqKHgFsaAb2t50G', key='CENSORED-PROJECT', name='Core CENSORED-PROJECT'}, submitter=User{uuid='AW7qaq8u2r2kF2AfC77I', login='admin'}}'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.step.ExtractReportStep' via constructor to bean named 'ComputationTempFolder'
2024.07.03 01:08:52 DEBUG ce[AZB2IxbWo8ApsgAChpeV][o.s.c.p.PriorityBeanFactory] Autowiring by type from bean name 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.step.ExtractReportStep' via constructor to bean named 'jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27-org.sonar.ce.task.projectanalysis.batch.BatchReportDirectoryHolderImpl'
2024.07.03 01:09:04 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Before cleanup stats (total=3, active=2, idle=1, waiting=0)
2024.07.03 01:09:04 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - After cleanup  stats (total=3, active=2, idle=1, waiting=0)
2024.07.03 01:09:04 DEBUG ce[][o.postgresql.Driver] Connecting with URL: jdbc:postgresql://my-censored-db-url/sonar
2024.07.03 01:09:04 DEBUG ce[][o.p.c.v.ConnectionFactoryImpl] Trying to establish a protocol version 3 connection to my-censored-db-url:5432
2024.07.03 01:09:04 DEBUG ce[][o.p.c.v.ConnectionFactoryImpl] Receive Buffer Size is 65,536
2024.07.03 01:09:04 DEBUG ce[][o.p.c.v.ConnectionFactoryImpl] Send Buffer Size is 535,296
2024.07.03 01:09:04 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@5235bc8a
2024.07.03 01:09:04 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - After adding stats (total=4, active=2, idle=2, waiting=0)
2024.07.03 01:09:34 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Before cleanup stats (total=4, active=2, idle=2, waiting=0)
2024.07.03 01:09:34 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - After cleanup  stats (total=4, active=2, idle=2, waiting=0)
2024.07.03 01:09:34 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled (queueDepth=0).
2024.07.03 01:10:04 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Before cleanup stats (total=4, active=2, idle=2, waiting=0)
2024.07.03 01:10:04 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - After cleanup  stats (total=4, active=2, idle=2, waiting=0)
2024.07.03 01:10:04 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled (queueDepth=0).
2024.07.03 01:10:34 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Before cleanup stats (total=4, active=2, idle=2, waiting=0)
2024.07.03 01:10:34 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - After cleanup  stats (total=4, active=2, idle=2, waiting=0)
2024.07.03 01:10:34 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled (queueDepth=0).
2024.07.03 01:10:40 DEBUG ce[][c.z.h.p.HikariPool] HikariPool-1 - keepalive: connection org.postgresql.jdbc.PgConnection@3ab8d7db is alive

Thanks in advance for your help,
Thibault

2

Hey @S0obi

I can think of 3 possible sources of the issue (well, I can think of more than 3, but these are the ones that stand out to me.

1. Resources availability on the SQ Server / DB Server

When this happens, I would suggest monitoring and taking note of the following:

  • CPU / Memory / Disk space utilization on the SonarQube server
  • CPU / Memory / Disk space on the Postgres database server

2. Something related to the analysis report

If you know what specific scan triggered the issue (you should be able to trace it back the Compute Engine task ID to the scanner logs), it would be good to know:

  • Does this always happen on the same project? Different projects?
  • Is there anything about the scanner report that stands out (specifically, size?) The scanner logs will include logs like:
INFO: Analysis report generated in 66ms, dir size=155.9 kB
INFO: Analysis report compressed in 15ms, zip size=18.2 kB

3. Something related to Database connections

In SonarQube v9.8 (SONAR-17762), we introduced some additional settings to manage the JDBC connection pool. They are documented here.

In some configurations when there is a firewall between SonarQube and the data you may experience timeout issues. The firewall may interrupt idle DB connections after a specific timeout which can lead to resetting connections.

This is particularly interesting to me because I see database connections being made in the logs you shared after the task stalls. If you have a firewall in between SonarQube and your database, this definitely seems relevant.

I am not an expert in these settings or with database connections, but maybe you have a DBA who is!

3

Hello @Colin, thanks for your fast response and help.

1. Resources availability on the SQ Server / DB Server
Our SonarQube server is well monitored and we can’t find any strange values during the background task event, on CPU/Memory/Disk.
Our DB server has not yet this kind of monitoring, we will make sure to add it, just in case.

2. Something related to the analysis report
Our issue is unfortunately not easily reproducible, it seems to be different project every time. I will try to note down the affected project and time to see if I can extract a pattern. The issue is only randomly triggered and it can take few weeks until it reappear.

The last “stuck background task” info on the scanner side are :

[INFO] 01:08:38.385 Analysis report generated in 62376ms, dir size=28.9 MB
[INFO] 01:08:54.813 Analysis report compressed in 16427ms, zip size=19.1 MB
[INFO] 01:08:56.205 Analysis report uploaded in 1390ms

At our level, it’s quiet a “big analysis” but it doesn’t seem crazy.

3. Something related to Database connections

We don’t have any firewall between our DB and SonarQube server, however if we are out of options it may definitely something we may try. I will try to take a look with our DBA team.

4

Hello @Colin and all,

Finally we were able to fix our issue, the theory adapted to our issue was actually “3. Something related to Database connections”.

For those that are reading this post, check your sonar.jdbc.* properties in your sonar.properties file. Before the fix, our config was :

sonar.jdbc.maxActive=60
sonar.jdbc.maxIdle=5
sonar.jdbc.minIdle=2
sonar.jdbc.maxWait=5000
sonar.jdbc.minEvictableIdleTimeMillis=600000
sonar.jdbc.timeBetweenEvictionRunsMillis=30000
sonar.jdbc.removeAbandoned=true
sonar.jdbc.removeAbandonedTimeout=60

This obsolete config was kept from version to version upgrade. The issue was linked to this particular configuration sonar.jdbc.removeAbandoned and sonar.jdbc.removeAbandonedTimeout.

Remove abandoned connection flag can be defined as follow :

Flag to remove abandoned connections if they exceed the removeAbandonedTimeout . If set to true a connection is considered abandoned and eligible for removal if it has been in use longer than the removeAbandonedTimeout Setting this to true can recover db connections from applications that fail to close a connection.

We found out that removeAbandonedTimeout set to 60 seconds were not sufficient in our case (it was at the begining of our use of SonarQube) and was causing SonarQube CE component to be completely stuck as the database connection was abandonned.

As a fix, we completely removed our specific JDBC configurations and rely on SonarQube defaults. After SonarQube restart, we were able to confirm that the issue didn’t reoccur since then.

Best regards,
Thibault

1 Like
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.