Database migration failing - Execution of migration step #3204 'Rename table 'PROJECTS' to 'COMPONENTS'' failed

Versions:

  • SonarQube Docker image version: sonarqube:8.9-community
  • PostgreSQL 12

Error:
Upon starting up a container running an aformentioned SonaQube image, it fills the public schema of Postgres database sonar with the following tables:

|table_name               |
|-------------------------|
|schema_migrations        |
|active_rule_parameters   |
|active_rules             |
|alm_app_installs         |
|analysis_properties      |
|ce_activity              |
|ce_queue                 |
|ce_scanner_context       |
|ce_task_characteristics  |
|ce_task_input            |
|ce_task_message          |
|default_qprofiles        |
|deprecated_rule_keys     |
|duplications_index       |
|es_queue                 |
|event_component_changes  |
|events                   |
|file_sources             |
|group_roles              |
|groups                   |
|internal_component_props |
|issue_changes            |
|issues                   |
|live_measures            |
|manual_measures          |
|metrics                  |
|notifications            |
|org_qprofiles            |
|org_quality_gates        |
|organization_alm_bindings|
|project_branches         |
|perm_templates_groups    |
|perm_templates_users     |
|perm_tpl_characteristics |
|permission_templates     |
|plugins                  |
|project_alm_bindings     |
|organizations            |
|project_links            |
|project_mappings         |
|project_measures         |
|project_qprofiles        |
|projects                 |
|qprofile_changes         |

When I then attempt to enter SonarQube web interface from browser, it prompts me to upgrade, so do the logs:

The database must be manually upgraded. Please backup the database and browse /setup. For more information: https://docs.sonarqube.org/latest/setup/upgrading

When I try to upgrade I am faced with the following error in the logs:

2021.09.08 09:35:14 INFO  web[][o.s.s.p.d.m.DatabaseMigrationImpl] Starting DB Migration and container restart
2021.09.08 09:35:14 INFO  web[][DbMigrations] Executing DB migrations...
2021.09.08 09:35:14 INFO  web[][DbMigrations] #3204 'Rename table 'PROJECTS' to 'COMPONENTS''...
2021.09.08 09:35:14 ERROR web[][DbMigrations] #3204 'Rename table 'PROJECTS' to 'COMPONENTS'': failure | time=13ms
2021.09.08 09:35:14 ERROR web[][DbMigrations] Executed DB migrations: failure | time=17ms
2021.09.08 09:35:14 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=105ms
2021.09.08 09:35:14 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #3204 'Rename table 'PROJECTS' to 'COMPONENTS'' failed
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
        at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Fail to execute ALTER TABLE projects RENAME TO components
        at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:106)
        at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:86)
        at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:128)
        at org.sonar.server.platform.db.migration.version.v82.RenameProjectsTableToComponents.execute(RenameProjectsTableToComponents.java:33)
        at org.sonar.server.platform.db.migration.step.DdlChange.execute(DdlChange.java:45)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
        ... 9 common frames omitted
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:350)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
        at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:279)
        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
        at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:91)
        ... 14 common frames omitted
Caused by: java.io.EOFException: null
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:445)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
        ... 23 common frames omitted

I attempted to grant accesses to the database user sonarqube uses, I also attempted to fit the sonarqube with the admin database user to no avail. Can you point me to what I might be doing wrong?

I have the same problem. Yesterday I tried to install 8.9.2-community and lts-community from scratch in kubernetes.
The DB is empty and I tried both Postgres 10 and 12.
When that migration statement is used the connection to the DB somehow gets disconnected and the SQL statement fails.
Today, I tried 9.0.1 from scratch with an empty DB (the same Postgres server) and it had no problems.
I would like to use 8.9.2 since it is LTS.

As an additonal information it might be interessting to know that when I execute that statement in pgAdmin4 when I am connected to the half-setup database the connection gets closed as well. It seems like the database has a problem with that statement. That is why I tried two versions.

I am wondering why this does not happen with 9.0.1. Maybe someone knows if this statement will be executed when 9.0.1 is run for the first time.

Was 8.9.2 tested with a Postgres database? I would assume it was.

Unfortunately, I have no database logs since it is a cloud database (as a service).

The stacktrace running the lts-community image is:

2021.09.08 22:00:59 ERROR web[DbMigrations] #3204 ‘Rename table ‘PROJECTS’ to ‘COMPONENTS’’: failure | time=11ms
2021.09.08 22:00:59 ERROR web[DbMigrations] Executed DB migrations: failure | time=14ms
2021.09.08 22:00:59 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=237ms
2021.09.08 22:00:59 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #3204 ‘Rename table ‘PROJECTS’ to ‘COMPONENTS’’ failed
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Fail to execute ALTER TABLE projects RENAME TO components
at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:106)
at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:86)
at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:128)
at org.sonar.server.platform.db.migration.version.v82.RenameProjectsTableToComponents.execute(RenameProjectsTableToComponents.java:33)
at org.sonar.server.platform.db.migration.step.DdlChange.execute(DdlChange.java:45)
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
… 9 common frames omitted
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:350)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:279)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:91)
… 14 common frames omitted
Caused by: java.io.EOFException: null
at org.postgresql.core.PGStream.receiveChar(PGStream.java:445)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
… 23 common frames omitted

When I run
ALTER TABLE projects RENAME TO components
against the database in pgAdmin I get this:

The application has lost the database connection:
⁃ If the connection was idle it may have been forcibly disconnected.
⁃ The application server or database server may have been restarted.
⁃ The user session may have timed out.
Do you want to continue and establish a new session?

Seems like the statement itself is crashing the database connection.

The database is set to use UTF8 encoding and both Collation and Character type is set to C.UTF-8 (which seems to be the default anyway).

Any help appreciated.

Thanks, Thomas

#8-9-lts-upgrade

Hello @peter_vyboch and @Thomas_Hofmann , welcome to the community :slight_smile: .

Could you provide more details where your DB is hosted, so that we could try to reproduce that issue?

Also is it possible for you to provide more details about what exact PostgreSQL version is used on that service?

@Thomas_Hofmann For the k8s, are you using our official helmchart or something else?

Hi @jacek.poreda,

the database is hosted on IBM Cloud. Versions are:
server_version_num Preset Options 120007
server_version_num Preset Options 100017

I adapted the offical hemlchart for our purposes.

I could setup such a database for you and give you access I think.

@Thomas_Hofmann

When I run
ALTER TABLE projects RENAME TO components
against the database in pgAdmin I get this: (…)

According to what you write here, it seems like there is an issue with your database. Do you have any resource limits set on your DB? Have you tried to see DB logs what is going on when you execute this specific query?

@jacek.poreda It is a resource constrained DB with 1 GB of RAM.
I also tried 2GB and the migration failed at exactly the same position so I figured that this would not happen at exactly the same time.

Now, the statement does not seem to be so expensive but I do not know what it will actually trigger.
Do you have any special resource constrains in mind which I could search for?

I will need to ask support to get DB logs.

I have requested access to the DB logs now.

1 Like

Looking at the database logs I see that a DB process is segfaulting:

LOG: server process (PID 23715) was terminated by signal 11: Segmentation fault

DETAIL: Failed process was running: ALTER TABLE projects RENAME TO components

LOG: terminating any other active server processes

stderr F 2021-09-10 19:26:59 UTC [PostgreSQL JDBC Driver] [57P02] [23717]: [1-1] user=xxx,db=sonarqube-8-9-2,client=172.30.223.192 WARNING: terminating connection because of crash of another server process

stderr F 2021-09-10 19:26:59 UTC [PostgreSQL JDBC Driver] [57P02] [23717]: [2-1] user=xxx,db=sonarqube-8-9-2,client=172.30.223.192 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

stderr F 2021-09-10 19:26:59 UTC [PostgreSQL JDBC Driver] [57P02] [23717]: [3-1] user=xxx,db=sonarqube-8-9-2,client=172.30.223.192 HINT: In a moment you should be able to reconnect to the database and repeat your command.

stderr F 2021-09-10 19:26:59 UTC [pgAdmin 4 - DB:sonarqube-8-9-2] [57P02] [23663]: [1-1] user=xxx,db=sonarqube-8-9-2,client=172.30.223.192 WARNING: terminating connection because of crash of another server process

stderr F 2021-09-10 19:26:59 UTC [pgAdmin 4 - DB:sonarqube-8-9-2] [57P02] [23663]: [2-1] user=xxx,db=sonarqube-8-9-2,client=172.30.223.192 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

I created a postgres database running in a container and then exported a script to create the schema after sonarqube has created it.

Now when I use that script to create the database in the cloud DB it works but sonarqube still tries to run the database creation. Probably there is a switch I can use to prevent this. I will read the docs…
→ actually I did not find a switch in docs. Do you have a hint @jacek.poreda ?
UPDATE: After creating all rows in schema_migrations sonarqube starts up without trying to create the DB.

2021.09.10 22:25:39 INFO web[o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2021.09.10 22:25:39 INFO web[o.s.s.p.d.m.AutoDbMigration] Automatically perform DB migration on fresh install
2021.09.10 22:25:40 INFO web[DbMigrations] Executing DB migrations…
2021.09.10 22:25:40 INFO web[DbMigrations] #1 ‘Create initial schema’…
2021.09.10 22:25:40 ERROR web[DbMigrations] #1 ‘Create initial schema’: failure | time=41ms
2021.09.10 22:25:40 ERROR web[DbMigrations] Executed DB migrations: failure | time=52ms
2021.09.10 22:25:40 ERROR web[o.s.s.p.PlatformImpl] Web server startup failed
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #1 ‘Create initial schema’ failed

@Thomas_Hofmann Such operations are not intended to be done by users and if not done carefully can lead to future issues with product.

Have you tried to see what are resource allocation during that query? Maybe there is still not enough memory.

@jacek.poreda Yes, I know. I was just trying to find a workaround. In the meantime the issue has been reported to the cloud provider and was fixed. I don’t have all the details but it sounded like a combination of some plugin in DB and access rights problems led to the crash of the process. Thanks for your support!

Thomas

2 Likes

Thanks a lot for providing details and I’m happy that you have managed to resolve issue!

In my case it was as Postgres in IBM Cloud as well, I was not using a Helm chart but rather a Docker image from Dockerhub, configured only with available environment variables. I have moved on to the latest version of the SonarQube image available on Docker Hub which resolved it for me.

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.