Hello @ganncamp,
I think I solved the problem, my server is now running with the new version 
But for the sake of information I would like to walk you throw what I have done and how it effect the service.
(1) First I simply restartet the service (without changing anything to recreate the scenario).
Then I got following error on web.log:
2021.07.01 09:04:46 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.NullPointerException: null
at org.sonar.server.es.metadata.MetadataIndexImpl.getMetadata(MetadataIndexImpl.java:98)
at org.sonar.server.es.metadata.MetadataIndexImpl.getInitialized(MetadataIndexImpl.java:62)
at org.sonar.server.es.IndexerStartupTask.lambda$getUninitializedTypes$0(IndexerStartupTask.java:99)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.sonar.server.es.IndexerStartupTask.getUninitializedTypes(IndexerStartupTask.java:100)
at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:62)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:53)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:88)
at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:84)
at org.sonar.server.platform.PlatformImpl.executeStartupTasks(PlatformImpl.java:198)
at org.sonar.server.platform.PlatformImpl.access$400(PlatformImpl.java:46)
at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$1(PlatformImpl.java:122)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:370)
at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:122)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:354)
at java.base/java.lang.Thread.run(Thread.java:834)
( but no error or warning on es.log)
â Service stopped, SonarQube Server was offline
(2) Then I deleted data/es7 and startet the service
After that I got the following error on web.log:
2021.07.01 09:18:35 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
org.sonar.server.es.ElasticsearchException: Fail to execute es requestES create index 'metadatas'
at org.sonar.server.es.EsClient.execute(EsClient.java:313)
at org.sonar.server.es.EsClient.create(EsClient.java:193)
at org.sonar.server.es.IndexCreator.createIndex(IndexCreator.java:149)
at org.sonar.server.es.IndexCreator.start(IndexCreator.java:89)
at org.sonar.core.platform.StartableCloseableSafeLifecyleStrategy.start(StartableCloseableSafeLifecyleStrategy.java:40)
at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84)
at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169)
at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132)
at org.picocontainer.behaviors.Stored.start(Stored.java:110)
at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1016)
at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1009)
at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:767)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.server.platform.platformlevel.PlatformLevel.start(PlatformLevel.java:90)
at org.sonar.server.platform.platformlevel.PlatformLevel4.start(PlatformLevel4.java:581)
at org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:213)
at org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:187)
at org.sonar.server.platform.PlatformImpl.access$500(PlatformImpl.java:46)
at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:120)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:370)
at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:120)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:354)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: 60.000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:865)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1639)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1606)
at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:134)
at org.sonar.server.es.EsClient.lambda$create$22(EsClient.java:193)
at org.sonar.server.es.EsClient.execute(EsClient.java:311)
... 22 common frames omitted
Caused by: java.net.SocketTimeoutException: 60.000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
... 1 common frames omitted
and in es.log
2021.07.01 09:17:23 WARN es[][o.e.c.InternalClusterInfoService] failed to retrieve stats for node [j6ip9-84QIettM9MaQBSkA]: [sonarqube][127.0.0.1:57435][cluster:monitor/nodes/stats[n]] request_id [11] timed out after [15229ms]
2021.07.01 09:17:23 WARN es[][o.e.t.TransportService] Received response for a request that has timed out, sent [15.2s/15229ms] ago, timed out [0s/0ms] ago, action [cluster:monitor/nodes/stats[n]], node [{sonarqube}{j6ip9-84QIettM9MaQBSkA}{KBc-Cso2Qg6Z8IsQkqObVg}{127.0.0.1}{127.0.0.1:57435}{dimr}{rack_id=sonarqube}], id [11]
2021.07.01 09:17:35 WARN es[][o.e.h.AbstractHttpServerTransport] handling request [null][HEAD][/metadatas?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false][Netty4HttpChannel{localAddress=/127.0.0.1:9001, remoteAddress=/127.0.0.1:57469}] took [10652ms] which is above the warn threshold of [5000ms]
2021.07.01 09:18:20 INFO es[][o.e.c.m.MetadataCreateIndexService] [metadatas] creating index, cause [api], templates [], shards [1]/[0]
2021.07.01 09:18:20 WARN es[][o.e.c.s.MasterService] took [24.9s], which is over [10s], to compute cluster state update for [create-index [metadatas], cause [api]]
â Service stopped, SonarQube Server was offline
(3) Simply Start service again
I get in web.log:
2021.07.01 09:30:25 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
org.sonar.server.es.ElasticsearchException: Fail to execute es requestES index request for key 'dbVendor' on index 'metadatas' on type 'metadata'
at org.sonar.server.es.EsClient.execute(EsClient.java:313)
at org.sonar.server.es.EsClient.index(EsClient.java:174)
at org.sonar.server.es.metadata.MetadataIndexImpl.setMetadata(MetadataIndexImpl.java:104)
at org.sonar.server.es.metadata.MetadataIndexImpl.setDbMetadata(MetadataIndexImpl.java:90)
at org.sonar.server.es.metadata.EsDbCompatibilityImpl.markAsCompatible(EsDbCompatibilityImpl.java:44)
at org.sonar.server.es.IndexCreator.checkDbCompatibility(IndexCreator.java:212)
at org.sonar.server.es.IndexCreator.start(IndexCreator.java:94)
at org.sonar.core.platform.StartableCloseableSafeLifecyleStrategy.start(StartableCloseableSafeLifecyleStrategy.java:40)
at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84)
at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169)
at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132)
at org.picocontainer.behaviors.Stored.start(Stored.java:110)
at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1016)
at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1009)
at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:767)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.server.platform.platformlevel.PlatformLevel.start(PlatformLevel.java:90)
at org.sonar.server.platform.platformlevel.PlatformLevel4.start(PlatformLevel4.java:581)
at org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:213)
at org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:187)
at org.sonar.server.platform.PlatformImpl.access$500(PlatformImpl.java:46)
at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:120)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:370)
at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:120)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:354)
at java.base/java.lang.Thread.run(Thread.java:834)
and in es.log
2021.07.01 09:28:10 WARN es[][o.e.c.InternalClusterInfoService] failed to retrieve shard stats from node [j6ip9-84QIettM9MaQBSkA]: [sonarqube][127.0.0.1:57515][indices:monitor/stats[n]] request_id [16] timed out after [29630ms]
2021.07.01 09:28:33 WARN es[][o.e.t.TransportService] Received response for a request that has timed out, sent [1.2m/77034ms] ago, timed out [47.4s/47404ms] ago, action [indices:monitor/stats[n]], node [{sonarqube}{j6ip9-84QIettM9MaQBSkA}{9YAOf7wcScCWipwn6e4aTA}{127.0.0.1}{127.0.0.1:57515}{dimr}{rack_id=sonarqube}], id [16]
2021.07.01 09:28:57 WARN es[][o.e.h.AbstractHttpServerTransport] handling request [null][HEAD][/metadatas?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false][Netty4HttpChannel{localAddress=/127.0.0.1:9001, remoteAddress=/127.0.0.1:57553}] took [8396ms] which is above the warn threshold of [5000ms]
2021.07.01 09:30:32 INFO es[][o.e.c.m.MetadataMappingService] [metadatas/4OP9xdhQR96fRZfcEAvjRg] create_mapping [metadata]
â Service stopped, SonarQube Server was offline
(4) Again deleting data/es7 and start service
Now, everything worked fine. Service started correctly and I see the logs I want to see
Summary:
So after restarting the service multiple times, with deleting the data/es7 folder twice, I got several different exceptions most likely in the same area (ES).
From the outside, it seems like some sort of hickup but now its working.
I hope, I didnât overwhelm you with log information, but maybe you got something out of it. You definitly got the deeper understanding here. And if you can comprehend what exaclty happened here, I would be happy to know about it.
P.S.: Great Webinar on the cognitive complexity theme yesterday. Iâm excited to show this to my colleagues.
Cheers
Claudius
). Iâve flagged this thread for more expert attention. It sounds like some kind of race condition that maybe the developers need to look into.