Error during update sq v7.9.1(LTS) to sq v.8.9.1(LTS)

Hello,
Ubuntu 20.04 LTS, postgresql 12,java “11.0.14.1” 2022-02-08
I’m working on updating sonarqube v7.9.1(LTS) to sonarqube v.8.9.1(LTS). Everyrhing does according Upgrade Guide | SonarQube Docs.
But I got the following:

2022.04.14 21:04:09 INFO  web[][DbMigrations] #3542 'Rename table 'LIVE_MEASURES_COPY' to 'LIVE_MEASURES''...
2022.04.14 21:04:10 INFO  web[][DbMigrations] #3542 'Rename table 'LIVE_MEASURES_COPY' to 'LIVE_MEASURES'': success | time=77ms
2022.04.14 21:04:10 INFO  web[][DbMigrations] #3543 'Add primary key on 'uuid' column of 'LIVE_MEASURES' table'...
2022.04.14 21:05:43 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :

2022.04.14 21:04:10 INFO  web[][DbMigrations] #3543 'Add primary key on 'uuid' column of 'LIVE_MEASURES' table'...
2022.04.14 21:05:43 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:103)
2022.04.14 21:12:51 ERROR web[][DbMigrations] #3543 'Add primary key on 'uuid' column of 'LIVE_MEASURES' table': failure | time=521630ms
2022.04.14 21:12:51 ERROR web[][DbMigrations] Executed DB migrations: failure | time=7270665ms
2022.04.14 21:12:51 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=7270756ms
2022.04.14 21:12:51 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #3543 'Add primary key on 'uuid' column of 'LIVE_MEASURES' table' failed
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)

image
As far as I see the table “live_measures_copy” exists before upgrade.

I’m starting the following
postgres=# \c sonardevel
You are now connected to database “sonardevel” as user “postgres”.
sonardevel=# DROP INDEX “live_measures_component”;
ERROR: index “live_measures_component” does not exist

sonardevel=# CREATE INDEX live_measures_component ON live_measures (component_uuid,metric_uuid);

Hi,

Welcome to the community!

Can we have that full stacktrace in the web.log, please?

 
HTH,
Ann

2022.04.14 20:23:59 INFO  web[][o.s.s.p.d.m.s.MassUpdate] 65 rows processed (1 items/sec)
2022.04.14 20:23:59 INFO  web[][o.s.s.p.d.m.s.MassUpdate] 0 rows processed (0 items/sec)
2022.04.14 20:23:59 INFO  web[][DbMigrations] #3538 'Populate 'metric_uuid' for 'QUALITY_GATE_CONDITIONS'': success | time=8ms
2022.04.14 20:23:59 INFO  web[][DbMigrations] #3539 'Make 'metric_uuid' column not nullable for 'QUALITY_GATE_CONDITIONS''...
2022.04.14 20:23:59 INFO  web[][DbMigrations] #3539 'Make 'metric_uuid' column not nullable for 'QUALITY_GATE_CONDITIONS'': success | time=3ms
2022.04.14 20:23:59 INFO  web[][DbMigrations] #3540 'Copy 'LIVE_MEASURES' table to 'LIVE_MEASURES_COPY''...
2022.04.14 21:04:08 INFO  web[][DbMigrations] #3540 'Copy 'LIVE_MEASURES' table to 'LIVE_MEASURES_COPY'': success | time=2409746ms
2022.04.14 21:04:08 INFO  web[][DbMigrations] #3541 'Drop 'LIVE_MEASURES' table'...
2022.04.14 21:04:09 INFO  web[][DbMigrations] #3541 'Drop 'LIVE_MEASURES' table': success | time=1039ms
2022.04.14 21:04:09 INFO  web[][DbMigrations] #3542 'Rename table 'LIVE_MEASURES_COPY' to 'LIVE_MEASURES''...
2022.04.14 21:04:10 INFO  web[][DbMigrations] #3542 'Rename table 'LIVE_MEASURES_COPY' to 'LIVE_MEASURES'': success | time=77ms
2022.04.14 21:04:10 INFO  web[][DbMigrations] #3543 'Add primary key on 'uuid' column of 'LIVE_MEASURES' table'...
2022.04.14 21:05:43 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:103)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.<init>(DefaultProcessCommands.java:34)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.secondary(DefaultProcessCommands.java:52)
        at org.sonar.server.app.WebServer.isOperational(WebServer.java:69)
        at org.sonar.server.app.WebServer.getStatus(WebServer.java:61)
        at org.sonar.process.ProcessEntryPoint.waitForStatus(ProcessEntryPoint.java:121)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:104)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:81)
        at org.sonar.server.app.WebServer.main(WebServer.java:99)
Caused by: java.io.FileNotFoundException: /opt/sonarqube-8.9.7.52159/temp/sharedmemory (Too many open files in system)
        at java.base/java.io.RandomAccessFile.open0(Native Method)
        at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
        at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
        at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:100)
        ... 8 common frames omitted
2022.04.14 21:05:43 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2022.04.14 21:05:44 WARN  web[][o.s.p.ProcessEntryPoint$HardStopperThread] Can not stop in 1000ms
2022.04.14 21:05:44 WARN  web[][o.s.s.a.EmbeddedTomcat] Failed to stop web server
org.apache.catalina.LifecycleException: A child container failed during stop
        at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:1010)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
        at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:491)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
        at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:791)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
        at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:450)
        at org.sonar.server.app.EmbeddedTomcat.terminate(EmbeddedTomcat.java:112)
        at org.sonar.server.app.WebServer.hardStop(WebServer.java:83)
        at org.sonar.process.ProcessEntryPoint$HardStopperThread.lambda$new$0(ProcessEntryPoint.java:219)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022.04.14 21:05:53 WARN  web[][o.s.s.p.d.m.DatabaseMigrationExecutorServiceImpl] Pool DatabaseMigrationExecutorServiceImpl did not terminate
2022.04.14 21:05:53 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [DB_migration-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14.1/java.net.SocketInputStream.socketRead0(Native Method)\n java.base@11.0.14.1/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)\n java.base@11.0.14.1/java.net.SocketInputStream.read(SocketInputStream.java:168)\n java.base@11.0.14.1/java.net.SocketInputStream.read(SocketInputStream.java:140)\n java.base@11.0.14.1/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)\n java.base@11.0.14.1/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)\n java.base@11.0.14.1/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n java.base@11.0.14.1/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1449)\n java.base@11.0.14.1/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1060)\n app//org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)\n app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)\n app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)\n app//org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)\n app//org.postgresql.core.PGStream.receiveChar(PGStream.java:443)\n app//org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057)\n app//org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)\n app//org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)\n app//org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)\n app//org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)\n app//org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)\n app//org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)\n app//org.postgresql.jdbc.PgStatement.execute(PgStatement.java:279)\n app//org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)\n app//org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)\n app//org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:91)\n app//org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:86)\n app//org.sonar.server.platform.db.migration.version.v84.metrics.livemeasures.AddPKeyOnUuidOfLiveMeasuresTable.execute(AddPKeyOnUuidOfLiveMeasuresTable.java:36)\n app//org.sonar.server.platform.db.migration.step.DdlChange.execute(DdlChange.java:45)\n app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)\n app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)\n app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl$$Lambda$531/0x00000008404b3040.accept(Unknown Source)\n app//com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)\n app//org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)\n app//org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)\n app//org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)\n app//org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)\n app//org.sonar.server.platform.db.migration.DatabaseMigrationImpl$$Lambda$526/0x00000008404b4c40.run(Unknown Source)\n java.base@11.0.14.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n java.base@11.0.14.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14.1/java.lang.Thread.run(Thread.java:829)
2022.04.14 21:12:51 ERROR web[][DbMigrations] #3543 'Add primary key on 'uuid' column of 'LIVE_MEASURES' table': failure | time=521630ms
2022.04.14 21:12:51 ERROR web[][DbMigrations] Executed DB migrations: failure | time=7270665ms
2022.04.14 21:12:51 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=7270756ms
2022.04.14 21:12:51 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #3543 'Add primary key on 'uuid' column of 'LIVE_MEASURES' table' failed
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
        at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Failed to insert row with value 3543 in table schema_migrations
        at org.sonar.server.platform.db.migration.history.MigrationHistoryImpl.done(MigrationHistoryImpl.java:91)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:76)
        ... 9 common frames omitted
Caused by: java.sql.SQLException: Data source is closed
        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:521)
        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753)
        at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
        at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)
        at org.sonar.server.platform.db.migration.history.MigrationHistoryImpl.done(MigrationHistoryImpl.java:82)
        ... 10 common frames omitted

Hi,

This looks like the problem:

That’s going to be a limit set at the OS level.

 
HTH,
Ann

bash-4.1$ sysctl vm.max_map_count
vm.max_map_count = 262144
bash-4.1$ sysctl fs.file-max
fs.file-max = 175536
bash-4.1$ ulimit -u
3048
bash-4.1$ ulimit -n
125536

Should I increase those?

Hi,

This external article may help.

 
Ann

Thank you very much