Upgrade from 5.7 to 6.2 failed - DB migration issue - Error during processing of row root_project_id resource_id id

SonarQube
,
1

Trying to upgrade from 5.6.6 to 6.2. First upgraded to 5.6.7, this went well without any problems.
Later trying to upgrade to 6.2 facing this issue.

Both DB and application running in the same VM. Database connectivity is not a problem.

mysql> show variables like '%net_write_timeout%';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| net_write_timeout | 60000 |
+-------------------+-------+

How to rectify this problem?

2018.08.12 13:16:53 INFO  web[][o.s.p.ProcessEntryPoint] Starting web
2018.08.12 13:16:53 INFO  web[][o.s.s.a.TomcatContexts] Webapp directory: /sonarapplication/sonarqube-6.2/web
2018.08.12 13:16:53 INFO  web[][o.a.c.h.Http11NioProtocol] Initializing ProtocolHandler ["http-nio-135.249.45.51-8085"]
2018.08.12 13:16:53 INFO  web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
2018.08.12 13:16:54 INFO  web[][o.e.plugins] [Ghaur] modules [], plugins [], sites []
2018.08.12 13:16:55 INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2018.08.12 13:16:55 INFO  web[][o.s.s.p.LogServerVersion] SonarQube Server / 6.2 / 4a28f29f95254b58f3cf0a0871bc632e998403f5
2018.08.12 13:16:55 INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:mysql://localhost:3306/sonarqube?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance
2018.08.12 13:16:56 WARN  web[][o.s.s.p.DatabaseServerCompatibility] Database must be upgraded. Please backup database and browse /setup
2018.08.12 13:16:56 INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /sonarapplication/sonarqube-6.2
2018.08.12 13:16:56 INFO  web[][o.s.s.p.w.RailsAppsDeployer] Deploying Ruby on Rails applications
2018.08.12 13:16:56 INFO  web[][o.s.s.p.Platform] DB needs migration, entering safe mode
2018.08.12 13:16:56 INFO  web[][jruby.rack] jruby 1.7.9 (ruby-1.8.7p370) 2013-12-06 87b108a on OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14 [linux-amd64]
2018.08.12 13:16:56 INFO  web[][jruby.rack] using a shared (threadsafe!) runtime
2018.08.12 13:17:01 INFO  web[][jruby.rack] keeping custom (config.logger) Rails logger instance
2018.08.12 13:18:24 INFO  web[][o.a.c.u.SessionIdGeneratorBase] Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [82,980] milliseconds.
2018.08.12 13:18:24 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@7cafe548 [pattern=org.sonar.api.web.ServletFilter$UrlPattern@7a9f10d1]
2018.08.12 13:18:24 INFO  web[][o.a.c.h.Http11NioProtocol] Starting ProtocolHandler ["http-nio-135.249.45.51-8085"]
2018.08.12 13:18:24 INFO  web[][o.s.s.a.TomcatAccessLog] Web server is started
2018.08.12 13:18:24 INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 8085
2018.08.12 13:26:05 INFO  web[][o.s.s.p.d.m.PlatformDatabaseMigration] Starting DB Migration
2018.08.12 13:26:06 INFO  web[][DbMigration] ==  CreatePermTemplatesCharacteristics: migrating =============================
2018.08.12 13:26:06 INFO  web[][DbMigration] -- create_table("perm_tpl_characteristics", {})
2018.08.12 13:26:06 INFO  web[][DbMigration]    -> 0.0720s
2018.08.12 13:26:06 INFO  web[][DbMigration]    -> 0 rows
2018.08.12 13:26:06 INFO  web[][DbMigration] -- index_exists?("perm_tpl_characteristics", ["template_id", "permission_key"], {:name=>"uniq_perm_tpl_charac"})
2018.08.12 13:26:06 INFO  web[][DbMigration]    -> 0.0080s
2018.08.12 13:26:06 INFO  web[][DbMigration] -- add_index("perm_tpl_characteristics", ["template_id", "permission_key"], {:name=>"uniq_perm_tpl_charac", :unique=>true})
2018.08.12 13:26:06 INFO  web[][DbMigration]    -> 0.0660s
2018.08.12 13:26:06 INFO  web[][DbMigration]    -> 0 rows
2018.08.12 13:26:06 INFO  web[][DbMigration] ==  CreatePermTemplatesCharacteristics: migrated (0.1470s) ====================
2018.08.12 13:26:06 INFO  web[][DbMigration]
2018.08.12 13:26:06 INFO  web[][DbMigration] ==  AddColumnsWithUuidsToResourceIndex: migrating =============================
2018.08.12 13:50:45 INFO  web[][DbMigration] ==  AddColumnsWithUuidsToResourceIndex: migrated (1478.6660s) =================
2018.08.12 13:50:45 INFO  web[][DbMigration]
2018.08.12 13:50:45 INFO  web[][DbMigration] ==  PopulateUuidColumnsOfResourceIndex: migrating =============================
2018.08.12 13:55:56 INFO  web[][o.s.d.v.MassUpdate] 207499 resource index entries processed (3458 items/sec)
2018.08.12 13:56:56 INFO  web[][o.s.d.v.MassUpdate] 421749 resource index entries processed (3570 items/sec)
2018.08.12 13:57:56 INFO  web[][o.s.d.v.MassUpdate] 632749 resource index entries processed (3516 items/sec)
2018.08.12 13:58:56 INFO  web[][o.s.d.v.MassUpdate] 846749 resource index entries processed (3566 items/sec)
2018.08.12 13:59:56 INFO  web[][o.s.d.v.MassUpdate] 1057749 resource index entries processed (3516 items/sec)
2018.08.12 14:00:56 INFO  web[][o.s.d.v.MassUpdate] 1266249 resource index entries processed (3475 items/sec)
2018.08.12 14:01:56 INFO  web[][o.s.d.v.MassUpdate] 1479249 resource index entries processed (3550 items/sec)
2018.08.12 14:02:56 INFO  web[][o.s.d.v.MassUpdate] 1720499 resource index entries processed (4020 items/sec)
2018.08.12 14:03:56 INFO  web[][o.s.d.v.MassUpdate] 1933499 resource index entries processed (3550 items/sec)
2018.08.12 14:04:56 INFO  web[][o.s.d.v.MassUpdate] 2141499 resource index entries processed (3466 items/sec)
2018.08.12 14:05:56 INFO  web[][o.s.d.v.MassUpdate] 2357999 resource index entries processed (3608 items/sec)
2018.08.12 14:06:56 INFO  web[][o.s.d.v.MassUpdate] 2567249 resource index entries processed (3487 items/sec)
2018.08.12 14:07:56 INFO  web[][o.s.d.v.MassUpdate] 2781249 resource index entries processed (3566 items/sec)
2018.08.12 14:08:56 INFO  web[][o.s.d.v.MassUpdate] 2988499 resource index entries processed (3454 items/sec)
2018.08.12 14:09:56 INFO  web[][o.s.d.v.MassUpdate] 3196749 resource index entries processed (3470 items/sec)
2018.08.12 14:10:56 INFO  web[][o.s.d.v.MassUpdate] 3402999 resource index entries processed (3437 items/sec)
2018.08.12 14:11:56 INFO  web[][o.s.d.v.MassUpdate] 3620749 resource index entries processed (3629 items/sec)
2018.08.12 14:12:56 INFO  web[][o.s.d.v.MassUpdate] 3827499 resource index entries processed (3445 items/sec)
2018.08.12 14:13:56 INFO  web[][o.s.d.v.MassUpdate] 4035999 resource index entries processed (3475 items/sec)
2018.08.12 14:14:56 INFO  web[][o.s.d.v.MassUpdate] 4245999 resource index entries processed (3500 items/sec)
2018.08.12 14:15:56 INFO  web[][o.s.d.v.MassUpdate] 4454999 resource index entries processed (3483 items/sec)
2018.08.12 14:16:56 INFO  web[][o.s.d.v.MassUpdate] 4661999 resource index entries processed (3450 items/sec)
2018.08.12 14:17:56 INFO  web[][o.s.d.v.MassUpdate] 4874749 resource index entries processed (3545 items/sec)
2018.08.12 14:18:56 INFO  web[][o.s.d.v.MassUpdate] 5082999 resource index entries processed (3470 items/sec)
2018.08.12 14:19:56 INFO  web[][o.s.d.v.MassUpdate] 5293749 resource index entries processed (3512 items/sec)
2018.08.12 14:20:56 INFO  web[][o.s.d.v.MassUpdate] 5499249 resource index entries processed (3425 items/sec)
2018.08.12 14:21:56 INFO  web[][o.s.d.v.MassUpdate] 5707999 resource index entries processed (3479 items/sec)
2018.08.12 14:22:56 INFO  web[][o.s.d.v.MassUpdate] 5915249 resource index entries processed (3454 items/sec)
2018.08.12 14:23:56 INFO  web[][o.s.d.v.MassUpdate] 6115249 resource index entries processed (3333 items/sec)
2018.08.12 14:24:56 INFO  web[][o.s.d.v.MassUpdate] 6323499 resource index entries processed (3470 items/sec)
2018.08.12 14:25:56 INFO  web[][o.s.d.v.MassUpdate] 6521749 resource index entries processed (3304 items/sec)
2018.08.12 14:26:56 INFO  web[][o.s.d.v.MassUpdate] 6727249 resource index entries processed (3425 items/sec)
2018.08.12 14:27:56 INFO  web[][o.s.d.v.MassUpdate] 6929499 resource index entries processed (3370 items/sec)
2018.08.12 14:28:56 INFO  web[][o.s.d.v.MassUpdate] 7130999 resource index entries processed (3358 items/sec)
2018.08.12 14:29:56 INFO  web[][o.s.d.v.MassUpdate] 7331749 resource index entries processed (3345 items/sec)
2018.08.12 14:30:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (312 items/sec)
2018.08.12 14:31:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:32:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:33:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:34:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:35:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:36:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:37:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:38:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:39:56 INFO  web[][o.s.d.v.MassUpdate] 7350499 resource index entries processed (0 items/sec)
2018.08.12 14:40:56 INFO  web[][o.s.d.v.MassUpdate] 7627999 resource index entries processed (4625 items/sec)
2018.08.12 14:41:06 ERROR web[][o.s.s.p.d.m.DatabaseMigrator] Fail to execute database migration: org.sonar.db.version.v60.PopulateUuidColumnsOfResourceIndex
java.lang.IllegalStateException: Error during processing of row: [id=372524828,resource_id=958516,root_project_id=237388]
        at org.sonar.db.version.SelectImpl.newExceptionWithRowDetails(SelectImpl.java:89)
        at org.sonar.db.version.SelectImpl.scroll(SelectImpl.java:81)
        at org.sonar.db.version.MassUpdate.execute(MassUpdate.java:91)
        at org.sonar.db.version.v60.PopulateUuidColumnsOfResourceIndex.populateUuidColumns(PopulateUuidColumnsOfResourceIndex.java:63)
        at org.sonar.db.version.v60.PopulateUuidColumnsOfResourceIndex.execute(PopulateUuidColumnsOfResourceIndex.java:44)
        at org.sonar.db.version.BaseDataChange.execute(BaseDataChange.java:45)
        at org.sonar.server.platform.db.migrations.DatabaseMigrator.executeMigration(DatabaseMigrator.java:95)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:455)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:316)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:61)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
        at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:190)
        at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:81)
        at org.jruby.RubyClass.finvoke(RubyClass.java:527)
        at org.jruby.RubyBasicObject.send(RubyBasicObject.java:2590)
        at org.jruby.RubyKernel.send(RubyKernel.java:2223)
        at org.jruby.RubyKernel$INVOKER$s$send.call(RubyKernel$INVOKER$s$send.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:350)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
        at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
        at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:384)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:303)
        at org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
        at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:190)
        at org.jruby.internal.runtime.methods.WrapperMethod.call(WrapperMethod.java:74)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:206)
        at org.jruby.RubyClass.finvoke(RubyClass.java:567)
        at org.jruby.RubyBasicObject.send(RubyBasicObject.java:2595)
        at org.jruby.RubyKernel.send(RubyKernel.java:2227)
        at org.jruby.RubyKernel$INVOKER$s$send.call(RubyKernel$INVOKER$s$send.gen)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:356)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:213)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:217)
        at org.jruby.ast.CallSpecialArgBlockPassNode.interpret(CallSpecialArgBlockPassNode.java:66)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:384)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:336)
        at org.jruby.runtime.BlockBody.call(BlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:224)
        at org.jruby.RubyProc$INVOKER$i$0$0$call.call(RubyProc$INVOKER$i$0$0$call.gen)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.IfNode.interpret(IfNode.java:118)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:190)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
        at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:384)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:357)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:312)
        at org.jruby.runtime.Block.yield(Block.java:142)
        at org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
        at org.jruby.RubyArray.each(RubyArray.java:1613)
        at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:182)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:214)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)
        at org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.WhenOneArgNode.whenNoTest(WhenOneArgNode.java:27)
        at org.jruby.ast.WhenOneArgNode.when(WhenOneArgNode.java:45)
        at org.jruby.ast.CaseNode.interpret(CaseNode.java:132)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:182)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:112)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:164)
        at RbCallUpgrade_369004465.callUpgrade(RbCallUpgrade_369004465.gen:13)
        at org.sonar.server.platform.db.migrations.PlatformDatabaseMigration.doUpgradeDb(PlatformDatabaseMigration.java:126)
        at org.sonar.server.platform.db.migrations.PlatformDatabaseMigration.doDatabaseMigration(PlatformDatabaseMigration.java:108)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Application was streaming results when the connection failed. Consider raising value of 'net_write_timeout' on the server.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:988)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3552)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3452)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3893)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:875)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1991)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:374)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:354)
        at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:6337)
        at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
        at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
        at org.sonar.db.version.SelectImpl.scroll(SelectImpl.java:77)
        ... 157 common frames omitted
Caused by: java.io.EOFException: Can not read response from server. Expected to read 24 bytes, read 22 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3512)
        ... 167 common frames omitted
2018.08.12 14:41:06 INFO  web[][o.s.s.p.d.m.PlatformDatabaseMigration] DB migration failed | time=4501110ms
2018.08.12 14:41:06 ERROR web[][o.s.s.p.d.m.PlatformDatabaseMigration] DB Migration or container restart failed. Process ended with an exception
org.jruby.exceptions.RaiseException: (StandardError) An error has occurred, all later migrations canceled:
2

I restarted sonarqube multiple times. Common patterns I observed are:

Every time it fails for different IDs.
DB migration fails approximately after 50 min.
Same resource id repeats with “resource index entries processed (0 items/sec)” before failing

3

Just before the failure DB process list shows as below:

mysql> show full processlist;
+-----+-----------------+-----------------+-----------+------------+------+------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Id  | User            | Host            | db        | Command    | Time | State                        | Info                                                                                                                                      |
+-----+-----------------+-----------------+-----------+------------+------+------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
|   7 | sonarqubedbuser | localhost:44670 | sonarqube | Sleep      | 2177 |                              | NULL                                                                                                                                      |
|   8 | sonarqubedbuser | localhost:44672 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|   9 | sonarqubedbuser | localhost:44674 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  10 | sonarqubedbuser | localhost:44676 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  11 | sonarqubedbuser | localhost:44678 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  12 | sonarqubedbuser | localhost:44680 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  13 | sonarqubedbuser | localhost:44682 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  14 | sonarqubedbuser | localhost:44684 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  15 | sonarqubedbuser | localhost:44686 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  16 | sonarqubedbuser | localhost:44688 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  17 | sonarqubedbuser | localhost:44690 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  18 | sonarqubedbuser | localhost:44692 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  19 | sonarqubedbuser | localhost:44694 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  20 | sonarqubedbuser | localhost:44696 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  21 | sonarqubedbuser | localhost:44698 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  22 | sonarqubedbuser | localhost:44700 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  23 | sonarqubedbuser | localhost:44702 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  24 | sonarqubedbuser | localhost:44704 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  25 | sonarqubedbuser | localhost:44706 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  26 | sonarqubedbuser | localhost:44708 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  27 | sonarqubedbuser | localhost:44710 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  28 | sonarqubedbuser | localhost:44712 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  29 | sonarqubedbuser | localhost:44714 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  30 | sonarqubedbuser | localhost:44716 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  31 | sonarqubedbuser | localhost:44718 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  32 | sonarqubedbuser | localhost:44720 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  33 | sonarqubedbuser | localhost:44722 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  34 | sonarqubedbuser | localhost:44724 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  35 | sonarqubedbuser | localhost:44726 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  36 | sonarqubedbuser | localhost:44728 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  37 | sonarqubedbuser | localhost:44730 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  38 | sonarqubedbuser | localhost:44732 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  39 | sonarqubedbuser | localhost:44734 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  40 | sonarqubedbuser | localhost:44736 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  41 | sonarqubedbuser | localhost:44738 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  42 | sonarqubedbuser | localhost:44740 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  43 | sonarqubedbuser | localhost:44742 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  44 | sonarqubedbuser | localhost:44744 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  45 | sonarqubedbuser | localhost:44746 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  46 | sonarqubedbuser | localhost:44748 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  47 | sonarqubedbuser | localhost:44750 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  48 | sonarqubedbuser | localhost:44752 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  49 | sonarqubedbuser | localhost:44754 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  50 | sonarqubedbuser | localhost:44756 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  51 | sonarqubedbuser | localhost:44758 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  52 | sonarqubedbuser | localhost:44760 | sonarqube | Sleep      | 2124 |                              | NULL                                                                                                                                      |
|  93 | sonarqubedbuser | localhost:44842 | sonarqube | Query      | 1821 | Sending to client            | SELECT ri.id, ri.resource_id, ri.root_project_id from resource_index ri where ri.component_uuid is null or ri.root_component_uuid is null |
|  94 | sonarqubedbuser | localhost:44844 | sonarqube | Query      |  519 | Waiting for global read lock | UPDATE resource_index SET component_uuid='AWK_IuFOUH7itwdFn_vC', root_component_uuid='AWK_Itg9dZwPmUqtfRO8' WHERE id=793132574            |
|  95 | sonarqubedbuser | localhost:44846 | sonarqube | Sleep      | 1901 |                              | NULL                                                                                                                                      |
| 117 | sonarqubedbuser | localhost:44888 | sonarqube | Sleep      | 1047 |                              | NULL                                                                                                                                      |
| 118 | sonarqubedbuser | localhost:44890 | sonarqube | Sleep      | 1044 |                              | NULL                                                                                                                                      |
| 119 | sonarqubedbuser | localhost:44892 | sonarqube | Sleep      | 1044 |                              | NULL                                                                                                                                      |
| 120 | sonarqubedbuser | localhost:44894 | sonarqube | Sleep      |  987 |                              | NULL                                                                                                                                      |
| 121 | sonarqubedbuser | localhost:44896 | sonarqube | Sleep      |  984 |                              | NULL                                                                                                                                      |
| 122 | sonarqubedbuser | localhost:44898 | sonarqube | Sleep      |  984 |                              | NULL                                                                                                                                      |
| 123 | sonarqubedbuser | localhost:44900 | sonarqube | Sleep      |  927 |                              | NULL                                                                                                                                      |
| 124 | sonarqubedbuser | localhost:44902 | sonarqube | Sleep      |  924 |                              | NULL                                                                                                                                      |
| 125 | sonarqubedbuser | localhost:44904 | sonarqube | Sleep      |  924 |                              | NULL                                                                                                                                      |
| 126 | sonarqubedbuser | localhost:44906 | sonarqube | Sleep      |  867 |                              | NULL                                                                                                                                      |
| 127 | sonarqubedbuser | localhost:44908 | sonarqube | Sleep      |  864 |                              | NULL                                                                                                                                      |
| 128 | sonarqubedbuser | localhost:44910 | sonarqube | Sleep      |  864 |                              | NULL                                                                                                                                      |
| 129 | sonarqubedbuser | localhost:44912 | sonarqube | Sleep      |  807 |                              | NULL                                                                                                                                      |
| 130 | sonarqubedbuser | localhost:44914 | sonarqube | Sleep      |  804 |                              | NULL                                                                                                                                      |
| 131 | sonarqubedbuser | localhost:44916 | sonarqube | Sleep      |  804 |                              | NULL                                                                                                                                      |
| 132 | sonarqubedbuser | localhost:44918 | sonarqube | Sleep      |  747 |                              | NULL                                                                                                                                      |
| 133 | sonarqubedbuser | localhost:44920 | sonarqube | Sleep      |  744 |                              | NULL                                                                                                                                      |
| 134 | sonarqubedbuser | localhost:44922 | sonarqube | Sleep      |  744 |                              | NULL                                                                                                                                      |
| 135 | sonarqubedbuser | localhost:44924 | sonarqube | Sleep      |  687 |                              | NULL                                                                                                                                      |
| 136 | sonarqubedbuser | localhost:44926 | sonarqube | Sleep      |  684 |                              | NULL                                                                                                                                      |
| 137 | sonarqubedbuser | localhost:44928 | sonarqube | Sleep      |  684 |                              | NULL                                                                                                                                      |
| 138 | sonarqubedbuser | localhost:44930 | sonarqube | Sleep      |  627 |                              | NULL                                                                                                                                      |
| 139 | sonarqubedbuser | localhost:44932 | sonarqube | Sleep      |  624 |                              | NULL                                                                                                                                      |
| 140 | sonarqubedbuser | localhost:44934 | sonarqube | Sleep      |  624 |                              | NULL                                                                                                                                      |
| 141 | sonarqubedbuser | localhost:44936 | sonarqube | Sleep      |  567 |                              | NULL                                                                                                                                      |
| 142 | sonarqubedbuser | localhost:44938 | sonarqube | Sleep      |  564 |                              | NULL                                                                                                                                      |
| 143 | sonarqubedbuser | localhost:44940 | sonarqube | Sleep      |  564 |                              | NULL                                                                                                                                      |
| 144 | sonarqubedbuser | localhost:44942 | sonarqube | Sleep      |  507 |                              | NULL                                                                                                                                      |
| 151 | root            | localhost       | NULL      | Query      |  519 | Waiting for table flush      | FLUSH TABLES WITH READ LOCK                                                                                                               |
| 152 | sonarqubedbuser | localhost:44946 | sonarqube | Sleep      |  504 |                              | NULL                                                                                                                                      |
| 153 | sonarqubedbuser | localhost:44948 | sonarqube | Sleep      |  504 |                              | NULL                                                                                                                                      |
| 154 | sonarqubedbuser | localhost:44950 | sonarqube | Sleep      |  447 |                              | NULL                                                                                                                                      |
| 155 | sonarqubedbuser | localhost:44952 | sonarqube | Sleep      |  444 |                              | NULL                                                                                                                                      |
| 156 | sonarqubedbuser | localhost:44954 | sonarqube | Sleep      |  444 |                              | NULL                                                                                                                                      |
| 157 | sonarqubedbuser | localhost:44956 | sonarqube | Sleep      |  387 |                              | NULL                                                                                                                                      |
| 158 | root            | localhost       | sonarqube | Field List |  192 | Waiting for table flush      |                                                                                                                                           |
| 159 | sonarqubedbuser | localhost:44958 | sonarqube | Sleep      |  384 |                              | NULL                                                                                                                                      |
| 160 | sonarqubedbuser | localhost:44960 | sonarqube | Sleep      |  384 |                              | NULL                                                                                                                                      |
| 161 | sonarqubedbuser | localhost:44962 | sonarqube | Sleep      |  327 |                              | NULL                                                                                                                                      |
| 162 | sonarqubedbuser | localhost:44964 | sonarqube | Sleep      |  324 |                              | NULL                                                                                                                                      |
| 163 | sonarqubedbuser | localhost:44966 | sonarqube | Sleep      |  324 |                              | NULL                                                                                                                                      |
| 164 | sonarqubedbuser | localhost:44968 | sonarqube | Sleep      |  267 |                              | NULL                                                                                                                                      |
| 165 | sonarqubedbuser | localhost:44970 | sonarqube | Sleep      |  264 |                              | NULL                                                                                                                                      |
| 166 | sonarqubedbuser | localhost:44972 | sonarqube | Sleep      |  264 |                              | NULL                                                                                                                                      |
| 167 | sonarqubedbuser | localhost:44974 | sonarqube | Sleep      |  207 |                              | NULL                                                                                                                                      |
| 168 | sonarqubedbuser | localhost:44976 | sonarqube | Sleep      |  204 |                              | NULL                                                                                                                                      |
| 169 | sonarqubedbuser | localhost:44978 | sonarqube | Sleep      |  204 |                              | NULL                                                                                                                                      |
| 170 | sonarqubedbuser | localhost:44980 | sonarqube | Sleep      |  147 |                              | NULL                                                                                                                                      |
| 171 | sonarqubedbuser | localhost:44982 | sonarqube | Sleep      |  144 |                              | NULL                                                                                                                                      |
| 172 | sonarqubedbuser | localhost:44984 | sonarqube | Sleep      |  144 |                              | NULL                                                                                                                                      |
| 173 | sonarqubedbuser | localhost:44986 | sonarqube | Sleep      |   87 |                              | NULL                                                                                                                                      |
| 174 | sonarqubedbuser | localhost:44988 | sonarqube | Sleep      |   84 |                              | NULL                                                                                                                                      |
| 175 | sonarqubedbuser | localhost:44990 | sonarqube | Sleep      |   84 |                              | NULL                                                                                                                                      |
| 176 | sonarqubedbuser | localhost:44992 | sonarqube | Sleep      |   27 |                              | NULL                                                                                                                                      |
| 177 | sonarqubedbuser | localhost:44994 | sonarqube | Sleep      |   24 |                              | NULL                                                                                                                                      |
| 178 | sonarqubedbuser | localhost:44996 | sonarqube | Sleep      |   24 |                              | NULL                                                                                                                                      |
| 179 | sonarqubedbuser | localhost:44998 | sonarqube | Sleep      |    2 |                              | NULL                                                                                                                                      |
| 180 | root            | localhost       | NULL      | Query      |    0 | starting                     | show full processlist                                                                                                                     |
+-----+-----------------+-----------------+-----------+------------+------+------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
107 rows in set (0.01 sec)
4

Looks like Data is so huge and not able to complete select and finally update is getting timed out.

mysql> show variables like ‘%wait_%’;
±--------------------------------------------------±---------+
| Variable_name | Value |
±--------------------------------------------------±---------+
| innodb_lock_wait_timeout | 50 |
| innodb_spin_wait_delay | 6 |
| lock_wait_timeout | 31536000 |
| performance_schema_events_waits_history_long_size | 10000 |
| performance_schema_events_waits_history_size | 10 |
| wait_timeout | 31536000 |
±--------------------------------------------------±---------+
6 rows in set (0.01 sec)

mysql> show variables like ‘%net_%’;
±------------------±------+
| Variable_name | Value |
±------------------±------+
| net_buffer_length | 16384 |
| net_read_timeout | 5000 |
| net_retry_count | 10 |
| net_write_timeout | 5000 |
| skip_networking | OFF |
| slave_net_timeout | 60 |
±------------------±------+
6 rows in set (0.00 sec)

5

Sonar properties related to DB are as follows:

sonar.jdbc.url=jdbc:mysql://bamboo.in.alcatel-lucent.com:3309/sonarqube?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&autoReconnect=true
sonar.jdbc.maxActive=600
sonar.jdbc.maxIdle=50
sonar.jdbc.minIdle=20
sonar.jdbc.maxWait=50000
sonar.jdbc.minEvictableIdleTimeMillis=600000
sonar.jdbc.timeBetweenEvictionRunsMillis=30000
sonar.web.javaOpts=-Xmx10240m -Xms10240m -XX:+HeapDumpOnOutOfMemoryError
sonar.web.http.maxThreads=500

6

Hi,

The root error is in the MySQL stack, did you try the recommendation which MySQL mentions itself in the error message ?

See that message right there in the first error:

Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Application was streaming results when the connection failed. Consider raising value of ‘net_write_timeout’ on the server.

Note: 6.2 is a surprising choice as new SonarQube version. It’s old and no longer maintained. I strongly suggest picking latest versions offered on Download | SonarQube

7

I did update net_write_timeout to 5000, even then the same problem.

Also I tried to upgrade to 6.7, because of DB migration problems, thought we could get this problem away with 6.2 first and then upgrade from 6.2 to 6.7.