Scanning broken on SonarQube 9.9 LTS + PostgreSQL 14

Since we upgraded to SonarQube 9.9 LTS and PostgreSQL 14, from 8.9 LTS with PostgreSQL 13, all our scans are failing. We use the developer edition deployed in Kubernetes.

From TeamCity, the scans fail with:

java.lang.IllegalStateException: Unable to load component class org.sonar.scanner.scan.filesystem.ProjectFileIndexer
19:02:01     at org.sonar.core.platform.SpringComponentContainer.getComponentByType(SpringComponentContainer.java:139)
19:02:01     at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:363)
19:02:01     at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
19:02:01     at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
19:02:01     at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:137)
19:02:01     at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
19:02:01     at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
19:02:01     at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
19:02:01     at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
19:02:01     at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
19:02:01     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
19:02:01     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
19:02:01     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
19:02:01     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
19:02:01     at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
19:02:01     at jdk.proxy1/jdk.proxy1.$Proxy0.execute(Unknown Source)
19:02:01     at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
19:02:01     at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
19:02:01     at org.sonarsource.scanner.cli.Main.execute(Main.java:112)
19:02:01     at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
19:02:01     at org.sonarsource.scanner.cli.Main.main(Main.java:61)
19:02:01   Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.sonarsource.scanner.api.internal.IsolatedClassloader@61df66b6-org.sonar.scanner.scan.filesystem.ProjectFileIndexer': Unsatisfied dependency expressed through constructor parameter 7; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.sonarsource.scanner.api.internal.IsolatedClassloader@61df66b6-org.sonar.scanner.scan.filesystem.FileIndexer': Unsatisfied dependency expressed through constructor parameter 6; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.sonarsource.scanner.api.internal.IsolatedClassloader@61df66b6-org.sonar.scanner.scan.filesystem.MetadataGenerator': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.sonarsource.scanner.api.internal.IsolatedClassloader@61df66b6-org.sonar.scanner.scan.filesystem.StatusDetection': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'projectRepositories' defined in org.sonar.scanner.repository.ProjectRepositoriesProvider: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.sonar.scanner.repository.ProjectRepositories]: Factory method 'projectRepositories' threw exception; nested exception is java.lang.IllegalStateException: Fail to request url: https://sonarqube.company.com/batch/project.protobuf?key=PROJECT&branch=develop
19:02:01     at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
19:02:01     at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
19:02:01     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
19:02:01     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
19:02:01     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
19:02:01     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
19:02:01     at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
19:02:01     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
19:02:01     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
19:02:01     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:233)
19:02:01     at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1284)
19:02:01     at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1245)
19:02:01     at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:494)
19:02:01     at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:349)
19:02:01     at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:342)
19:02:01     at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1172)
19:02:01     at org.sonar.core.platform.SpringComponentContainer.getComponentByType(SpringComponentContainer.java:137)
19:02:01     ... 20 more
19:02:01   Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.sonarsource.scanner.api.internal.IsolatedClassloader@61df66b6-org.sonar.scanner.scan.filesystem.FileIndexer': Unsatisfied dependency expressed through constructor parameter 6; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.sonarsource.scanner.api.internal.IsolatedClassloader@61df66b6-org.sonar.scanner.scan.filesystem.MetadataGenerator': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.sonarsource.scanner.api.internal.IsolatedClassloader@61df66b6-org.sonar.scanner.scan.filesystem.StatusDetection': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'projectRepositories' defined in org.sonar.scanner.repository.ProjectRepositoriesProvider: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.sonar.scanner.repository.ProjectRepositories]: Factory method 'projectRepositories' threw exception; nested exception is java.lang.IllegalStateException: Fail to request url:

If I run the same call via curl, it fails after 30 seconds with:

* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* HTTP/2 stream 1 was not closed cleanly before end of the underlying stream
* Connection #0 to host sonarqube.company.com left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly before end of the underlying stream
 
real        0m30.042s
user        0m0.011s
sys          0m0.015s

This is likely because of the ingress controller (Traefik) so I went ahead and did a port-forward so I ran the same call directly connecting to the SonarQube service. The request takes more than 2 minutes and 30 seconds to complete. But eventually it completes.

I traced down the issue to a long SQL query that runs in the database:

SELECT p.uuid as uuid,
  p.uuid_path as uuidPath,
  p.branch_uuid as branchUuid,
  p.module_uuid as moduleUuid,
  p.module_uuid_path as moduleUuidPath,
  p.main_branch_project_uuid as mainBranchProjectUuid,
  p.kee as kee,
  p.name as name,
  p.long_name as longName,
  p.description as description,
  p.qualifier as qualifier,
  p.scope as scope,
  p.language as language,
  p.root_uuid as rootUuid,
  p.path as path,
  p.enabled as enabled,
  p.copy_component_uuid as copyComponentUuid,
  p.private as isPrivate,
  p.created_at as createdAt
  FROM components p
  INNER JOIN components module ON
  module.branch_uuid = p.branch_uuid
  and module.uuid = $1
  and module.scope='PRJ' AND module.enabled = true
  where
  p.scope = $2
  and p.enabled = true
  and
  p.module_uuid_path LIKE module.module_uuid_path

This query takes longer than 2m30s to complete. We noticed this used to take seconds before the upgrade. Not the exact query, but at least the project listing.

We have performed investigation on the following areas:

  • Investigated the 30s timeout on ingress controller → we agreed this would not solve the issue, since the actual query takes too much time
  • Investigated database performance. Our PostgreSQL DBs run as a cluster and they are spread across different nodes and different underlying storage systems. We have tested the performance of the DB and found no issues. Additionally, we confirmed that TeamCity – using the same cluster, and being much more DB intensive, has no performance issues. We have also moved the master nodes of the DB to new nodes, running on different hardware and storage and the issue is the same.

Can anyone help us understand what is going on?

1 Like

While we haven’t been able to find the exact root case of this, we were able to fix the performance issue. Since SQL tracing would have been time consuming, we decided to attempt to reproduce the error on a fresh server with new PostgreSQL database using our PROD dump. As we have not been able to reproduce, we dropped our PROD PostgreSQL database and re-imported the backup.

Performance is back to normal again.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.