DB Migration fails when updating sonar from 6.7.1 to 7.8

SonarQube
,
1

MY SQL version : 5.6.10 (aurora mysql)

DB Migration fails when updating from 6.7.1 to 7.8.

When the sonarqube.com/setup was hit , the DB migration started and failed with below error.

NOTE: I checked the memory, disk space everything was fine.

LOG:

2019.11.27 22:18:31 INFO web[DbMigrations] #2105 ‘Add LINE_HASHES_VERSION to table FILE_SOURCES’…
2019.11.27 22:37:27 ERROR web[DbMigrations] #2105 ‘Add LINE_HASHES_VERSION to table FILE_SOURCES’: failure | time=1135824ms
2019.11.27 22:37:27 ERROR web[DbMigrations] Executed DB migrations: failure | time=45101782ms
2019.11.27 22:37:27 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=45101911ms
2019.11.27 22:37:27 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #2105 ‘Add LINE_HASHES_VERSION to table FILE_SOURCES’ failed
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Fail to execute ALTER TABLE file_sources ADD (line_hashes_version INTEGER NULL)
at org.sonar.server.platform.db.migration.step.DdlChange$Context.execute(DdlChange.java:97)
at org.sonar.server.platform.db.migration.step.DdlChange$Context.execute(DdlChange.java:77)
at org.sonar.server.platform.db.migration.version.v72.AddLineHashesVersionToFileSources.execute(AddLineHashesVersionToFileSources.java:36)
at org.sonar.server.platform.db.migration.step.DdlChange.execute(DdlChange.java:45)
at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
… 9 common frames omitted
Caused by: java.sql.SQLException: Incorrect key file for table ‘file_sources’; try to repair it
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2482)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2440)
at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:175)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:175)
at org.sonar.server.platform.db.migration.step.DdlChange$Context.execute(DdlChange.java:82)
… 13 common frames omitted

2

Welcome to the community forum!

This issue occurs when the temporary folder is running out of space. Usually it’s the /tmp/ folder, that can be on its own partition. Monitor the free space on this partition while the migration is running.

3

Hi Pierre,
Thanks for the direction. I have triggered DB migration again and checking the tmp partition.
image
The attached is the partitions currently I have. devtmpfs and tmpfs looks good for now. Are
you mentioning these partitions?

I will monitor this till the migration runs and update you the result.

4

Hi Pierre,

The DB migration failed again as expected but I was not able to see the tmp partition becoming full. It was almost fully free only.
image
Do we see this error if the disk is full ?

Caused by: java.sql.SQLException: Incorrect key file for table ‘file_sources’; try to repair it

5

Hi , Can someone provide shed some light on what needs to be done further?

6

Everything point to a space issue. You should dig more on that subject : try to increase the tmp_table_size, check your my.cnf where the tmp folder is, try to move it somewhere else with more space.

1 Like