Project data reload errors after 10.6 upgrade

SonarQube Server / Community Build
1

Hi there,

We upgraded our SonarQube Enterprise server from 10.4.1 to 10.6 today. We intentionally skipped 10.5 due to a reported bug on these forums that I believe would have negatively impacted us. After performing the database migration we ran into a slew of project data reload errors that all look the same, I’ll share one below.

Did I make a mistake in skipping 10.5 and going straight to 10.6? Or is there something else going on? Scanning is functional on the projects that failed to reload, but filtering on issues in the UI is currently disabled.

java.lang.IllegalStateException: Fail to prepare SQL request to select all issues
	at org.sonar.server.issue.index.IssueIteratorForSingleChunk.<init>(IssueIteratorForSingleChunk.java:72)
	at org.sonar.server.issue.index.IssueIteratorFactory.createForBranch(IssueIteratorFactory.java:39)
	at org.sonar.server.issue.index.IssueIndexer.indexOnAnalysis(IssueIndexer.java:126)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.lambda$execute$1(IndexIssuesStep.java:52)
	at java.base/java.util.Optional.ifPresent(Unknown Source)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.execute(IndexIssuesStep.java:48)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IssueSyncTaskProcessor.process(IssueSyncTaskProcessor.java:60)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.postgresql.util.PSQLException: ERROR: column ii.uuid does not exist
  Position: 803
### The error may involve org.sonar.db.issue.IssueMapper.scrollIssuesForIndexation-Inline
### The error occurred while setting parameters
### SQL: select       i.kee as issueKey,       i.assignee,       i.line,       i.resolution,       i.severity,       i.status,       i.effort,       i.author_login as authorLogin,       i.issue_close_date as issueCloseDate,       i.issue_creation_date as issueCreationDate,       i.issue_update_date as issueUpdateDate,       r.uuid as ruleUuid,       r.language as language,       i.clean_code_attribute as cleanCodeAttribute,       r.clean_code_attribute as ruleCleanCodeAttribute,       c.uuid as componentUuid,       c.path,       c.scope,       c.branch_uuid as branchUuid,       pb.is_main as isMain,       pb.project_uuid as projectUuid,       i.tags,       i.issue_type as issueType,       r.security_standards as securityStandards,       c.qualifier,       i.code_variants as codeVariants,             ii.uuid as "ii_uuid",     ii.software_quality as "ii_softwareQuality",     ii.severity as "ii_severity",                 rdi.uuid as "rdi_uuid",     rdi.software_quality as "rdi_softwareQuality",     rdi.severity as "rdi_severity",                 n.uuid is not null as isNewCodeReferenceIssue         from issues i     inner join rules r on r.uuid = i.rule_uuid     inner join components c on c.uuid = i.component_uuid     inner join project_branches pb on c.branch_uuid = pb.uuid     left join new_code_reference_issues n on n.issue_key = i.kee     left outer join issues_impacts ii on i.kee = ii.issue_key     left outer join rules_default_impacts rdi on r.uuid = rdi.rule_uuid      WHERE  c.branch_uuid = ? and i.project_uuid = ?      order by i.kee
### Cause: org.postgresql.util.PSQLException: ERROR: column ii.uuid does not exist
  Position: 803
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectCursor(DefaultSqlSession.java:129)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectCursor(DefaultSqlSession.java:117)
	at org.apache.ibatis.binding.MapperMethod.executeForCursor(MapperMethod.java:166)
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:84)
	at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141)
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
	at jdk.proxy2/jdk.proxy2.$Proxy63.scrollIssuesForIndexation(Unknown Source)
	at org.sonar.db.issue.IssueDao.scrollIssuesForIndexation(IssueDao.java:99)
	at org.sonar.server.issue.index.IssueIteratorForSingleChunk.<init>(IssueIteratorForSingleChunk.java:68)
	... 24 more
Caused by: org.postgresql.util.PSQLException: ERROR: column ii.uuid does not exist
  Position: 803
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190)
	at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:177)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.queryCursor(PreparedStatementHandler.java:72)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.queryCursor(RoutingStatementHandler.java:85)
	at org.apache.ibatis.executor.ReuseExecutor.doQueryCursor(ReuseExecutor.java:71)
	at org.apache.ibatis.executor.BaseExecutor.queryCursor(BaseExecutor.java:180)
	at org.apache.ibatis.executor.CachingExecutor.queryCursor(CachingExecutor.java:82)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectCursor(DefaultSqlSession.java:125)
	... 32 more
2

Hey @frittsy

There should be no problems upgrading directly from v10.4 to v10.6. This is supported.

It seems like a SQL query from SonarQube v10.4 is being used against the v10.6 database (unless we really messed up). We dropped a column in SonarQube v10.5 that appears in the logs you shared (uuid in the issue_impacts table), but we had no reports on v10.5 of a similar error.

I would suggest two things:

  • Make sure that there are no other SonarQube instances connected to the database
  • Delete the data/es8 folder from your SonarQube server and restart
3

Hey @Colin, thanks for the reply and confirming that my upgrade path is supported. I figured as much but just wanted to double check.

Due to that nature of our web server infrastructure (containers in AWS ECS), there is always a brief overlap where 2 instances of SonarQube are running, since ECS brings up a new container before completely killing the old one. But this shouldn’t be longer than a minute or two, and is not currently the case. It’s possible that the 10.4 container was still running when I did the database migration though, if you think that could have played a part.

I will attempt deleting that es8 folder and/or relaunching a new container off-hours tonight and see what happens.

4

The SQ 10.4 instance being alive during the upgrade could definitely cause these symptoms. Keep me posted after trashing the index and restarting.

5

We experienced a similar issue when upgrading from 10.5.1-enterprise to 10.6.0-enterprise. We have a ECS Fargate deployment, and some of the issues from the projects are no longer visible in the UI under the Issues tab.

6

Just want to confirm that launching a new 10.6 task/container in ECS resolved this for us, did not need to manually wipe anything since the storage on these containers are ephemeral.

7

I also confirm that index issues are resolved with force reindexing or restarting.