Errors from sonarqube:10.1.0-community onwards

sub · May 6, 2024, 12:00pm

version: sonarqube:10.1.0-community
deployment: Docker, sonarqube:10.1.0-community image
trying to achieve: errorless migration to 10.1.0
tried so far: multiple migration jumps, including patch-to-patch

Hello.

I have been having problems with upgrade/migration to sonarqube:10.1.0-community and onward for the past few months.

I have inherited the handling of SonarQube from a colleague who left the company before I joined. He left the server at version 7.x. I have approached upgrading the server as suggested in the migration path, first raising it to 7 LTS, then migrating to 8 LTS, and then to 9 LTS. This landed me on 9.9.3 at the time. I then raised it to 9.9.4 and 9.9.5 respectively in time. All of this passed well.

Due to us requiring some features that were released only in 10.2 (HCL analysis, manual choice of a main branch per project via the GUI, etc.), I wanted to upgrade onward. A migration from 9 LTS (9.9.3, 9.9.4, 9.9.5) to 10.0.0 seemingly succeeeded. At least all the smoke tests I did passed, no errors were caught when clicking around. However, going from 10.0.0 to 10.1.0 started producing errors.

For example, accessing a project throws a status 500 on the path /api/ce/component?component=<projectKey>, with the JSON response of: {"errors":[{"msg":"An error has occurred. Please contact your administrator"}]}.

The following log is produced on the standard output during that error:

2024.05.06 11:52:48 ERROR web[AY9No9nppev+csf5AADc][o.s.s.w.WebServiceEngine] Fail to process request http://<domain>/api/ce/component?component=<projectKey>
org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.postgresql.util.PSQLException: ERROR: syntax error at or near "$3"
  Position: 1424
### The error may exist in org.sonar.db.ce.CeActivityMapper
### The error may involve org.sonar.db.ce.CeActivityMapper.selectByQuery-Inline
### The error occurred while setting parameters
### SQL: select                   ca.uuid,     ca.node_name as nodeName,     ca.task_type as taskType,     ca.component_uuid as componentUuid,     ca.main_component_uuid as mainComponentUuid,     ca.analysis_uuid as analysisUuid,     ca.status as status,     ca.submitter_uuid as submitterUuid,     ca.submitted_at as submittedAt,     ca.worker_uuid as workerUuid,     ca.started_at as startedAt,     ca.executed_at as executedAt,     ca.created_at as createdAt,     ca.updated_at as updatedAt,     ca.is_last as isLast,     ca.is_last_key as isLastKey,     ca.main_is_last as mainIsLast,     ca.main_is_last_key as mainIsLastKey,     ca.execution_time_ms as executionTimeMs,     ca.error_message as errorMessage,     ca.error_type as errorType,     ctm.uuid as ctm_uuid,     ctm.task_uuid as ctm_taskUuid,     ctm.message as ctm_message,     ctm.message_type as ctm_type,     ctm.created_at as ctm_createdAt    ,           csc.task_uuid is not null as hasScannerContext                     from ce_activity ca     left outer join ce_task_message ctm on ca.uuid = ctm.task_uuid     left outer join ce_scanner_context csc on csc.task_uuid = ca.uuid         where ca.uuid in (                   select ca.uuid     from ce_activity ca      WHERE  ca.main_is_last=true                       and ca.main_component_uuid in         (           ?         )          order by ca.created_at desc, uuid desc     offset (?-1) rows fetch next ? rows only         )     order by ca.created_at desc, ca.uuid desc
### Cause: org.postgresql.util.PSQLException: ERROR: syntax error at or near "$3"
  Position: 1424
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:156)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
        at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
        at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
        at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:142)
        at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
        at jdk.proxy2/jdk.proxy2.$Proxy120.selectByQuery(Unknown Source)
        at org.sonar.db.ce.CeActivityDao.selectByQuery(CeActivityDao.java:84)
        at org.sonar.server.ce.ws.ComponentAction.handle(ComponentAction.java:89)
        at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:111)
        at org.sonar.server.platform.web.WebServiceFilter.doFilter(WebServiceFilter.java:84)
        at org.sonar.server.platform.web.MasterServletFilter$JavaxFilterAdapter.doFilter(MasterServletFilter.java:227)
        at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:198)
        at org.sonar.server.platform.web.MasterServletFilter$HttpFilterChainAdapter.doFilter(MasterServletFilter.java:241)
        at org.sonar.server.platform.web.SonarLintConnectionFilter.doFilter(SonarLintConnectionFilter.java:66)
        at org.sonar.server.platform.web.MasterServletFilter$JavaxFilterAdapter.doFilter(MasterServletFilter.java:227)
        at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:198)
        at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:146)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:83)
        at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:70)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.sonar.server.platform.web.CspFilter.doFilter(CspFilter.java:63)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:60)
        at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:47)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:57)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.sonar.server.platform.web.RequestIdFilter.doFilter(RequestIdFilter.java:66)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:65)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:115)
        at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
        at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:262)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "$3"
  Position: 1424
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415)
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190)
        at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:177)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
        at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
        at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:80)
        at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:62)
        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:333)
        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:158)
        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:110)
        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:90)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:154)
        ... 153 common frames omitted

Since the database migration did not produce any visible errors, I thought that maybe the version 10.1.0 specifically was buggy, so I decided to continue with upgrades, patch-to-patch, minor-to-minor. As incrementally as I could. However, not only did the error not go away, some others arose further. From some sync errors, to being unable to list users under the administration.

But to leave those perhaps on the side and tackle this initial one first, what would be the issue? I’ve turned half of the internet around, and have seen that some others have also had similar issues, with no solution being presented.

Thank you for any sort of assistance.

ganncamp · May 7, 2024, 12:28pm

Hi,

First, welcome to the community, and congrats on your epic migration journey!

Did you run analyses after any of your migrations? I have very vague, fuzzy memories that there are possible bad DB states that only show up after earlier migrations after you start trying to use the upgraded DB.

Can you bump up your server log level briefly? You can do this through the UI, Administration → System, and it should give us the SQL of the query that’s failing.

Spoiler: this is probably going to need some manual DB updates to fix.

Also, have you upgraded the underlying DB version along the way? I don’t remember off-hand, but I’m pretty confident that SonarQube 7.x didn’t run on the same version of Postgres (because it didn’t exist!) that 10.x does.

Ann

sub · May 8, 2024, 11:39am

Hello Ann,

Thank you for the response and the warm welcome.

The database was indeed of an older version, thank you for the heads-up. I have migrated a few versions of Postgres now as well (dump → removal of volumes → up of a new version → import).

Almost everything seems to work great now on sonarqube:10.4.1-community. Not on the latest version yet because some community plugins haven’t caught up with their support yet.

I say almost, because after a slight smoke test, one thing is still a bit off. All the data seems to be here, but the projects aren’t displaying on the main projects page (/projects). They are accessible under Administration → Projects → Management (/admin/projects_management), however. I can open the projects from there and see all the past analyses, details, etc.

Permissions shouldn’t be the issue. Auth is handled via SAML LDAP, which works. 1:1 group mappings exist, and the projects have permission templates applies to them that are active (Project Settings → Permissions on a project list groups that I am a part of). All projects are private, but on 9.9.5 all with explicit permissions were able to view them on /projects. Has something changed, or is somthing still missing?

I have also bumped the SONAR_LOG_LEVEL from INFO to DEBUG. Nothing is specifically logged when visiting /projects, I’m just getting periodic ce and web logs of HikariPool-1.

All the best.

ganncamp · May 8, 2024, 12:33pm

Hi,

I’m glad things are settling down. The empty Projects page sounds like a bad Elasticsearch index.

SonarQube is backed by your database, but some pages are painted from an Elasticsearch index for faster response time, and Projects is one of them. Try forcing an ElasticSearch reindex, and let us know if that doesn’t do it for you.

Ann

sub · May 8, 2024, 12:55pm

Ann,

This was it indeed! I wiped the contents of the es8 dir, redployed SonarQube, and it now works. Unsure how I didn’t think of this, since I had to wipe those same indexes every time I downgraded SonarQube previously.

Thank you once more, hopefully everything indeed stabilizes now. Otherwise you might hear from me again.

Topic		Replies	Views
Sonar upgrade from 9.X to 10.1 - sonar and postgres errors SonarQube Server / Community Build	13	629	May 24, 2024
Sonar upgrade from 9.9 to 10.3 - DB error SonarQube Server / Community Build sonarqube , bump	10	281	March 22, 2024
SonarQube 8.4.1 Analysis Bug SonarQube Server / Community Build upgrade , bug-no-category	20	3052	April 27, 2022
Questions for migrating the sonarqube v10.6 from centos7 to rocky8 with current projects SonarQube Server / Community Build	25	186	December 4, 2024
Upgrade Sonar qube SonarQube Server / Community Build sonarqube , settings	10	332	April 15, 2024

Errors from sonarqube:10.1.0-community onwards

Related topics