Microsoft SQLServer DB migration fails when upgrading SonarQube DE from 9.9.0 LTS to 10.4.1

Hi,

I’m facing a DB migration error when upgrading SonarQube Developer Edition from 9.9.0.65466 to 10.4.1.88267.
As far as i understand the upgrade path examples I should be able to directly upgade from 9.9 LTS to > 10.1+?

2024.04.11 13:24:12 INFO  web[][DbMigrations] Executing DB migrations...
2024.04.11 13:24:12 INFO  web[][DbMigrations] #100000 'Remove orphan rules in Quality Profiles'...
2024.04.11 13:24:13 INFO  web[][o.s.s.p.d.m.s.MassUpdate] 0 rows processed (0 items/sec)
2024.04.11 13:24:13 INFO  web[][DbMigrations] #100000 'Remove orphan rules in Quality Profiles': success | time=187ms
2024.04.11 13:24:13 INFO  web[][DbMigrations] #100001 'Drop index 'projects_module_uuid' in the 'Components' table'...
2024.04.11 13:24:13 INFO  web[][DbMigrations] #100001 'Drop index 'projects_module_uuid' in the 'Components' table': success | time=125ms
2024.04.11 13:24:13 INFO  web[][DbMigrations] #100002 'Drop column 'module_uuid' in the 'Components' table'...
2024.04.11 13:24:13 ERROR web[][DbMigrations] #100002 'Drop column 'module_uuid' in the 'Components' table': failure | time=141ms
2024.04.11 13:24:13 ERROR web[][DbMigrations] Executed DB migrations: failure | time=453ms
2024.04.11 13:24:13 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=485ms
2024.04.11 13:24:13 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #100002 'Drop column 'module_uuid' in the 'Components' table' failed
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
	at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:54)
	at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
	at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalStateException: Fail to execute ALTER TABLE components DROP COLUMN module_uuid
	at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:106)
	at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:86)
	at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:128)
	at org.sonar.server.platform.db.migration.step.DropColumnChange.execute(DropColumnChange.java:49)
	at org.sonar.server.platform.db.migration.step.DdlChange.execute(DdlChange.java:45)
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
	... 9 common frames omitted
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The index 'IX_components__branch_uuid' is dependent on column 'module_uuid'.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:259)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1695)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:920)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:814)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7675)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4137)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:272)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:246)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStatement.java:787)
	at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94)
	at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
	at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:91)
	... 14 common frames omitted

The last thing that can be seen is this:
grafik

Thanks in advance.
Thomas

Hey there

Could a rogue DBA have added this index? You might want to manually drop it from the database, and see if the upgrade proceeds.

Normally, no alterations should be made to the database (like adding an index) outside of the upgrade process.

Hi Collin,

we deleted the index, and the colomn got dropped successfully.
Why did this happen to the database, as far as I know we’ve done no manual modifications.

Yet we try and error the upgrade by removing the “blocking” index iteratively.

org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #100002 'Drop column 'module_uuid' in the 'Components' table' failed
...
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The index 'IX_components__branch_uuid' is dependent on column 'module_uuid'.
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #102024 'Drop 'purge_status' column in 'snapshots' table' failed
...
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The index 'IX_snapshots__component_uuid_islast_purge_status' is dependent on column 'purge_status'.

After we deleted those two indices the Database was up-tp-date. But still the SonarQube Server is not starting:

2024.04.12 14:17:26 INFO  web[][o.s.s.p.DetectPluginChange] Detect plugin changes
2024.04.12 14:17:26 INFO  web[][o.s.s.p.DetectPluginChange] No plugin change detected
2024.04.12 14:17:27 INFO  web[][o.h.v.i.util.Version] HV000001: Hibernate Validator null
2024.04.12 14:17:28 WARN  web[][o.s.c.a.AnnotationConfigApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd-org.sonar.server.es.IndexCreator': Initialization of bean failed; nested exception is java.lang.NullPointerException: Cannot invoke "org.elasticsearch.common.document.DocumentField.getValue()" because "field" is null
2024.04.12 14:17:28 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd-org.sonar.server.es.IndexCreator': Initialization of bean failed; nested exception is java.lang.NullPointerException: Cannot invoke "org.elasticsearch.common.document.DocumentField.getValue()" because "field" is null
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:628)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:920)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:222)
	at org.sonar.server.platform.platformlevel.PlatformLevel.start(PlatformLevel.java:80)
	at org.sonar.server.platform.platformlevel.PlatformLevel4.start(PlatformLevel4.java:687)
	at org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:214)
	at org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:194)
	at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:362)
	at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:113)
	at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:346)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "org.elasticsearch.common.document.DocumentField.getValue()" because "field" is null
	at org.sonar.server.es.metadata.MetadataIndexImpl.getMetadata(MetadataIndexImpl.java:97)
	at org.sonar.server.es.metadata.MetadataIndexImpl.getDbVendor(MetadataIndexImpl.java:83)
	at org.sonar.server.es.metadata.EsDbCompatibilityImpl.hasSameDbVendor(EsDbCompatibilityImpl.java:37)
	at org.sonar.server.es.IndexCreator.checkDbCompatibility(IndexCreator.java:183)
	at org.sonar.server.es.IndexCreator.start(IndexCreator.java:87)
	at org.sonar.core.platform.StartableBeanPostProcessor.postProcessBeforeInitialization(StartableBeanPostProcessor.java:33)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
	... 17 common frames omitted
2024.04.12 14:17:29 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2024.04.12 14:17:29 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2024.04.12 14:17:29 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.

The issue seems to be known Sonarqube10 deployment error but the resolution does not fit to our usecase.

Thanks in advance
Thomas