Sonarqube db migration 6.7 -> 7.6

(Karl Murray) #1

Hi
Currently we use version 6.7 and are trying to upgrade to 7.6
We’ve run into issues were the upgrade has failed after 1 hour, we have upgraded the size of undo tablespace several times which has let the upgrade continue for up to 20 hours until it fails yet again with below error:

2019.04.19 06:08:49 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=71187631ms
2019.04.19 06:08:49 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #1907 'Populate table live_measures' failed
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
        at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Error during processing of row: [uuid=AWSzTvhKFrfKIfpn7V2M,project_uuid=AWSzTvBAuLayKLuXFDMh,metric_id=136,value=null,text_value=null,variation_value_1=null,measure_data=oracle.sql.BLOB@7e882158]
        at org.sonar.server.platform.db.migration.step.SelectImpl.newExceptionWithRowDetails(SelectImpl.java:89)
        at org.sonar.server.platform.db.migration.step.SelectImpl.scroll(SelectImpl.java:81)
        at org.sonar.server.platform.db.migration.step.MassUpdate.execute(MassUpdate.java:92)
        at org.sonar.server.platform.db.migration.version.v70.PopulateLiveMeasures.execute(PopulateLiveMeasures.java:57)
        at org.sonar.server.platform.db.migration.step.DataChange.execute(DataChange.java:45)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
        ... 9 common frames omitted
Caused by: java.sql.SQLException: ORA-01555: snapshot too old: rollback segment number 5 with name "_SYSSMU5_720104719$" too small

        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494)
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:441)
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:436)
        at oracle.jdbc.driver.T4C8TTILob.processError(T4C8TTILob.java:758)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
        at oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)
        at oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:3027)
        at oracle.sql.BLOB.getBytes(BLOB.java:340)
        at oracle.sql.BLOB.getBytes(BLOB.java:222)
        at oracle.jdbc.driver.T4CBlobAccessor.getBytes(T4CBlobAccessor.java:202)
        at oracle.jdbc.driver.GeneratedStatement.getBytes(GeneratedStatement.java:143)
        at oracle.jdbc.driver.GeneratedScrollableResultSet.getBytes(GeneratedScrollableResultSet.java:164)
        at org.apache.commons.dbcp2.DelegatingResultSet.getBytes(DelegatingResultSet.java:398)
        at org.apache.commons.dbcp2.DelegatingResultSet.getBytes(DelegatingResultSet.java:398)
        at org.sonar.server.platform.db.migration.step.Select$Row.getNullableBytes(Select.java:101)
        at org.sonar.server.platform.db.migration.version.v70.PopulateLiveMeasures.lambda$execute$0(PopulateLiveMeasures.java:65)
        at org.sonar.server.platform.db.migration.step.MassUpdate.callSingleHandler(MassUpdate.java:118)
        at org.sonar.server.platform.db.migration.step.MassUpdate.lambda$execute$0(MassUpdate.java:92)
        at org.sonar.server.platform.db.migration.step.SelectImpl.scroll(SelectImpl.java:78)
        ... 13 common frames omitted
2019.04.19 06:19:07 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:100)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.<init>(DefaultProcessCommands.java:34)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.secondary(DefaultProcessCommands.java:52)
        at org.sonar.server.app.WebServer.isOperational(WebServer.java:68)
        at org.sonar.server.app.WebServer.getStatus(WebServer.java:60)
        at org.sonar.process.ProcessEntryPoint.waitForOperational(ProcessEntryPoint.java:145)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:120)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:100)
        at org.sonar.server.app.WebServer.main(WebServer.java:91)
Caused by: java.io.FileNotFoundException: /opt/sonarqube/sonarqube-7.6/temp/sharedmemory (Too many open files in system)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:97)
        ... 8 common frames omitted

Is there a better way to fix this than upgrading undo table size?
I’ve read on the forum and on upgrade guide that we should be upgrading to LTS version first, so should we be upgrading from: 6.7 -> 6.7.7, then upgrading to 7.6 ?

(G Ann Campbell) #2

Hi,

The error is coming from your database, so I think you’ll have to address it at your database.

Regarding 6.7.7, no you don’t have to upgrade to it first. It’s only important that you hit a 6.7 in your upgrade path.

 
Ann

(Karl Murray) #3

Hi thanks for your response!
Could there be a reason that it takes so long though?
Are there ways to improve the performance of it? Such as adding specific indexes

(G Ann Campbell) #4

Hi,

There was recent discussion about drive type affecting upgrade performance:

Beyond that, there’s a note about Oracle cleanup in the docs starting with 6.6 that might be relevant:

https://docs.sonarqube.org/latest/setup/upgrading/#header-2

 
HTH,
Ann