Upgrade SonarQube from 8.5 to 8.6 failure on DB migration

krimitalel · January 11, 2021, 6:37pm

I am trying to upgrade SonarQube From 8.5 to 8.6 but i get this error when I perform DB Migration ( / setup )

2021.01.11 14:48:30 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #4122 'Migrate view definitions from xml to db' failed
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
        at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Failed to migrate views definitions property.
        at org.sonar.server.platform.db.migration.version.v86.MigrateApplicationDefinitionsFromXmlToDb.execute(MigrateApplicationDefinitionsFromXmlToDb.java:110)
        at org.sonar.server.platform.db.migration.step.DataChange.execute(DataChange.java:44)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
        ... 9 common frames omitted
Caused by: java.sql.BatchUpdateException: Batch entry 0 insert into app_projects(uuid, application_uuid, project_uuid, created_at) values ('AXbx63vxHeot98TA_hoF', 'AXKHLGE6K7Ly74qVE6X_', 'AXJ6u5APK7Ly74qVE6Wb', 1610376510438) was aborted: ERROR: duplicate key value violates unique constraint "uniq_app_projects"
  Detail: Key (application_uuid, project_uuid)=(AXKHLGE6K7Ly74qVE6X_, AXJ6u5APK7Ly74qVE6Wb) already exists.  Call getNextException to see other errors in the batch.
        at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:169)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2286)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:521)
        at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:870)
        at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:893)
        at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1644)
        at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
        at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
        at org.sonar.server.platform.db.migration.step.UpsertImpl.execute(UpsertImpl.java:68)
        at org.sonar.server.platform.db.migration.version.v86.MigrateApplicationDefinitionsFromXmlToDb.insertApplicationProjects(MigrateApplicationDefinitionsFromXmlToDb.java:163)

Any help please ?

jacek.poreda · January 12, 2021, 7:27am

Hello @krimitalel,

Welcome to the community!

Is that first error you occurred during migration or is it the one you are having continuously when trying to upgrade?

If that’s not an issue do you mind sharing results of this query (this will show xml content of apps and portfolios definition):

select * from internal_properties where kee='views.def';

ybechtold · January 15, 2021, 5:00pm

Hello @jacek.poreda,

we’re facing the same problem as @krimitalel with Sonarqube EE 8.5 -> 8.6, identical error message.
Since this is hindering the upgrade for our productive instance I’d like to jump in as he has not yet responded. Sadly, the output of your command contains sensitive data to our clients but I can provide a version of the XML contained after swapping the data for dummy data.

For us, the error occured on multiple tries to update and was perfectly reproducable. There were no other errors to be found in any of the other log files, only a few warnings in the es.log concerning ES-Configuration that can, to the best of my knowledge of ES, be ignored (deprecated configuration directive naming).

I’ve left the syntax of the following XML as is and just changed the contained data but kept things that seemed to be referenced using common names, in hope of introducing no error in the xml through this.

<views> <vw key="Dummy" def="false"> <name><![CDATA[Invalid projects]]></name> <desc><![CDATA[random desc]]></desc> <regexp><![CDATA[foo.bar.*]]></regexp> <qualifier><![CDATA[VW]]></qualifier> </vw> <vw key="foo.bar.testapplication" def="false"> <name><![CDATA[Test application]]></name> <desc><![CDATA[several projects together, that will be deployed together]]></desc> <qualifier><![CDATA[APP]]></qualifier> foo.bar:db foo.bar.d </vw> <vw key="foo.bar.a.website" def="false"> <name><![CDATA[A Website]]></name> <desc><![CDATA[]]></desc> <qualifier><![CDATA[APP]]></qualifier> foo.bar.a.website.a foo.bar.a.website.b foo.bar.a.website.c foo.bar.a.website.d foo.bar.a.website.e </vw> <vw key="foo.bar.b.website" def="false"> <name><![CDATA[b Website]]></name> <desc><![CDATA[All Website projects]]></desc> <qualifier><![CDATA[APP]]></qualifier> foo.bar.b.website.a foo.bar.b.website.b foo.bar.b.website.c foo.bar.b.website.d foo.bar.b.website.e foo.bar.b.website.f foo.bar.b.website.g foo.bar.b.cms.a foo.bar.b.cms.b foo.bar.b.cms.c foo.bar.b.website.h </vw> <vw key="foo.bar.c.website" def="false"> <name><![CDATA[C Website]]></name> <desc><![CDATA[Apps for C Website]]></desc> <qualifier><![CDATA[APP]]></qualifier> foo.bar.c.a foo.bar.c.b foo.bar.c.c foo.bar.c.d foo.bar.c.e foo.bar.c.f foo.bar.c.g foo.bar.c.h foo.bar.c.cms.a foo.bar.c.cms.b foo.bar.c.cms.c foo.bar.c.i foo.bar.c.j foo.bar.c.k foo.bar.c.l foo.bar.c.app.a foo.bar.c.app.b foo.bar.c.app.c foo.bar.c.m </vw> <vw key="Portfolio_X" def="false"> <name><![CDATA[Portfolio X]]></name> <desc><![CDATA[]]></desc> <regexp><![CDATA[foo.bar.x.*]]></regexp> <qualifier><![CDATA[VW]]></qualifier> </vw> <vw key="Portfolio_C" def="false"> <name><![CDATA[Portfolio C]]></name> <desc><![CDATA[Apps for C]]></desc> <regexp><![CDATA[foo.bar.c.*]]></regexp> <qualifier><![CDATA[VW]]></qualifier> </vw> <vw key="foo.bar.b.cms.generationexecutable" def="false"> <name><![CDATA[b cms Generation Executable]]></name> <desc><![CDATA[b cms Generation Executable]]></desc> <qualifier><![CDATA[APP]]></qualifier> </vw> </views>

I hope this can help shed some light on this issue and will also help the original poster.

Thanks for any help,
Yuri

jacek.poreda · February 15, 2021, 1:17pm

Hey @ybechtold,

Welcome to to the community and thanks for sharing.

There is an issue with this migration and it will be fixed in 8.6.1 see.

The problem is that application definition in XML contains projects which no longer exist in database. Thus migration is failing the first time with error similar to:

2020.12.22 15:38:10 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #4122 'Migrate view definitions from xml to db' failed
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
	at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
	at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
	at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Failed to migrate views definitions property.
	at org.sonar.server.platform.db.migration.version.v86.MigrateApplicationDefinitionsFromXmlToDb.execute(MigrateApplicationDefinitionsFromXmlToDb.java:110)
	at org.sonar.server.platform.db.migration.step.DataChange.execute(DataChange.java:44)
	at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
	... 9 common frames omitted
Caused by: org.postgresql.util.PSQLException: No value specified for parameter 1.
	at org.postgresql.core.v3.SimpleParameterList.checkAllParametersSet(SimpleParameterList.java:275)

Then after re-trying it will show the error you are seeing about duplication.

For now I would recommend waiting for bugfix release.

system · February 15, 2021, 1:17pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.