Upgrade from 7.9 LTS to 8.9 LTS - Migration step 3433 failed

migration step #3433 ‘Populate ‘uuid’ column for ‘PROJECT_MEASURES’’ failed

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    SonarQube 7.9.6-community (upgraded from 7.9.1-developer to benefit from the dockerhub published image for this upgrade).
    PostgreSQL 10.7

  • what are you trying to achieve
    Upgrade to SonarQube 8.9.6-developer

  • what have you tried so far to achieve this
    So far followed the instructions, and when executing the migrations in /setup the following error occurs:

sonarqube-lts 2022.01.13 10:55:25 INFO  web[][DbMigrations] #3433 'Populate 'uuid' column for 'PROJECT_MEASURES''...
sonarqube-lts 2022.01.13 10:55:46 INFO  app[][o.s.a.SchedulerImpl] Stopping SonarQube
sonarqube-lts 2022.01.13 10:55:46 INFO  web[][o.s.p.ProcessEntryPoint] Gracefully stopping process
sonarqube-lts 2022.01.13 10:55:56 WARN  web[][o.s.s.p.d.m.DatabaseMigrationExecutorServiceImpl] Pool DatabaseMigrationExecutorServiceImpl did not terminate
sonarqube-lts 2022.01.13 10:55:56 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [DB_migration-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
    java.base@11.0.11/java.net.SocketInputStream.socketRead0(Native Method)
    java.base@11.0.11/java.net.SocketInputStream.socketRead(Unknown Source)
    java.base@11.0.11/java.net.SocketInputStream.read(Unknown Source)
    java.base@11.0.11/java.net.SocketInputStream.read(Unknown Source)
    app//org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
    app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
    app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
    app//org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
    app//org.postgresql.core.PGStream.receiveChar(PGStream.java:443)
    app//org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057)
    app//org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:521)
    app//org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:878)
    app//org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:901)
    app//org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1644)
    app//org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
    app//org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
    app//org.sonar.server.platform.db.migration.step.UpsertImpl.addBatch(UpsertImpl.java:56)
    app//org.sonar.server.platform.db.migration.step.MassUpdate.callSingleHandler(MassUpdate.java:120)
    app//org.sonar.server.platform.db.migration.step.MassUpdate.lambda$execute$0(MassUpdate.java:93)
    app//org.sonar.server.platform.db.migration.step.MassUpdate$$Lambda$584/0x00000001005ea840.handle(Unknown Source)
    app//org.sonar.server.platform.db.migration.step.SelectImpl.scroll(SelectImpl.java:78)
    app//org.sonar.server.platform.db.migration.step.MassUpdate.execute(MassUpdate.java:93)
    app//org.sonar.server.platform.db.migration.version.v84.projectmeasures.PopulateProjectMeasureUuid.execute(PopulateProjectMeasureUuid.java:44)
    app//org.sonar.server.platform.db.migration.step.DataChange.execute(DataChange.java:44)
    app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
    app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
    app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl$$Lambda$573/0x00000001005edc40.accept(Unknown Source)
    app//com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
    app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
    app//org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
    app//org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
    app//org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
    app//org.sonar.server.platform.db.migration.DatabaseMigrationImpl$$Lambda$568/0x00000001005ee840.run(Unknown Source)
    java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.base@11.0.11/java.lang.Thread.run(Unknown Source)
sonarqube-lts 2022.01.13 10:55:56 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [ProgressLogger[MassUpdate]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
    java.base@11.0.11/java.lang.Object.wait(Native Method)
    java.base@11.0.11/java.util.TimerThread.mainLoop(Unknown Source)
    java.base@11.0.11/java.util.TimerThread.run(Unknown Source)
sonarqube-lts 2022.01.13 10:55:56 INFO  web[][o.s.s.app.WebServer] WebServer stopped
sonarqube-lts 2022.01.13 10:56:08 INFO  web[][o.s.s.p.d.m.s.MassUpdate] 532520 rows processed (8875 items/sec)
sonarqube-lts 2022.01.13 10:56:08 ERROR web[][DbMigrations] #3433 'Populate 'uuid' column for 'PROJECT_MEASURES'': failure | time=43051ms
sonarqube-lts 2022.01.13 10:56:08 ERROR web[][DbMigrations] Executed DB migrations: failure | time=44523ms
sonarqube-lts 2022.01.13 10:56:08 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=44581ms
sonarqube-lts 2022.01.13 10:56:08 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
sonarqube-lts org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #3433 'Populate 'uuid' column for 'PROJECT_MEASURES'' failed
sonarqube-lts 	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
sonarqube-lts 	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
sonarqube-lts 	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
sonarqube-lts 	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
sonarqube-lts 	at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
sonarqube-lts 	at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
sonarqube-lts 	at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
sonarqube-lts 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
sonarqube-lts 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
sonarqube-lts 	at java.base/java.lang.Thread.run(Unknown Source)
sonarqube-lts Caused by: java.lang.IllegalStateException: Failed to insert row with value 3433 in table schema_migrations
sonarqube-lts 	at org.sonar.server.platform.db.migration.history.MigrationHistoryImpl.done(MigrationHistoryImpl.java:91)
sonarqube-lts 	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:76)
sonarqube-lts 	... 9 common frames omitted
sonarqube-lts Caused by: java.sql.SQLException: Data source is closed
sonarqube-lts 	at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:521)
sonarqube-lts 	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753)
sonarqube-lts 	at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
sonarqube-lts 	at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)
sonarqube-lts 	at org.sonar.server.platform.db.migration.history.MigrationHistoryImpl.done(MigrationHistoryImpl.java:82)
sonarqube-lts 	... 10 common frames omitted

Another thread indicated that the UUIDs are generated by the service and I identified some CPU throttling during the migration (Kubernetes hosted) so I removed that bottleneck with no change.

I have also tried the upgrade from 7.9.1-developer edition. I noted no migrations to run when moving to 7.9.6-community. The one thing that comes to mind is going from 7.9.6-developer rather than 7.9.6-community though it would be good to have some reasoning if this would help.

Thanks

1 Like

Hello @amccague . I’m reacting a bit late to this, did you manage to solve the issue? It looks like the connexion to the database was lost during the migration, so most likely, re-running the database migration would help.

Database migrations are designed to be re-entrant, so if it fails at some point, you can re-run it safely.