SonarQube 9.7 ES reboot loop

SonarQube Server / Community Build
1

I try to update to latest SonarQube 7.0.61563 and get all time Elastic Search reboot loop, es.log:

2022.10.20 21:57:05 INFO  es[][o.e.n.Node] version[7.17.5], pid[2232121], build[default/tar/8d61b4f7ddf931f219e3745f295ed2bbc50c8e84/2022-06-23T21:57:28.736740635Z], OS[Linux/5.15.0-50-generic/amd64], JVM[Ubuntu/OpenJDK 64-Bit Server VM/11.0.16/11.0.16+8-post-Ubuntu-0ubuntu122.04]
2022.10.20 21:57:05 INFO  es[][o.e.n.Node] JVM home [/usr/lib/jvm/java-11-openjdk-amd64]
2022.10.20 21:57:05 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseG1GC, -Djava.io.tmpdir=/opt/sonarqube/9.7.0.61563/temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=/opt/sonarqube/9.7.0.61563/temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Dcom.redhat.fips=false, -Des.enforce.bootstrap.checks=true, -Xmx512m, -Xms512m, -XX:MaxDirectMemorySize=256m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/9.7.0.61563/elasticsearch, -Des.path.conf=/opt/sonarqube/9.7.0.61563/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=false]
2022.10.20 21:57:05 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2022.10.20 21:57:05 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2022.10.20 21:57:05 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2022.10.20 21:57:05 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2022.10.20 21:57:05 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2022.10.20 21:57:05 INFO  es[][o.e.p.PluginsService] no plugins loaded
2022.10.20 21:57:05 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [27.2gb], net total_space [108.9gb], types [ext4]
2022.10.20 21:57:05 INFO  es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2022.10.20 21:57:05 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [Hm_eXsUqR9S7OWZgZGR3LA], cluster name [sonarqube], roles [data_frozen, master, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
2022.10.20 21:57:08 INFO  es[][o.e.t.NettyAllocator] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
2022.10.20 21:57:08 INFO  es[][o.e.i.r.RecoverySettings] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
2022.10.20 21:57:08 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and seed hosts providers [settings]
2022.10.20 21:57:08 INFO  es[][o.e.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2022.10.20 21:57:08 INFO  es[][o.e.n.Node] initialized
2022.10.20 21:57:08 INFO  es[][o.e.n.Node] starting ...
2022.10.20 21:57:08 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:38533}, bound_addresses {127.0.0.1:38533}
2022.10.20 21:57:09 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2022.10.20 21:57:09 INFO  es[][o.e.c.c.Coordinator] cluster UUID [pbP0sGR2SJuOnWwdgp1MbA]
2022.10.20 21:57:09 INFO  es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined)[{sonarqube}{Hm_eXsUqR9S7OWZgZGR3LA}{kuqGYDbcSZeCRojDCnnH1Q}{127.0.0.1}{127.0.0.1:38533}{cdfhimrsw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 31, version: 61, delta: master node changed {previous [], current [{sonarqube}{Hm_eXsUqR9S7OWZgZGR3LA}{kuqGYDbcSZeCRojDCnnH1Q}{127.0.0.1}{127.0.0.1:38533}{cdfhimrsw}]}
2022.10.20 21:57:09 INFO  es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{Hm_eXsUqR9S7OWZgZGR3LA}{kuqGYDbcSZeCRojDCnnH1Q}{127.0.0.1}{127.0.0.1:38533}{cdfhimrsw}]}, term: 31, version: 61, reason: Publication{term=31, version=61}
2022.10.20 21:57:09 INFO  es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2022.10.20 21:57:09 INFO  es[][o.e.n.Node] started
2022.10.20 21:57:09 INFO  es[][o.e.g.GatewayService] recovered [0] indices into cluster_state
2022.10.20 21:57:09 INFO  es[][o.e.n.Node] stopping ...
2022.10.20 21:57:09 INFO  es[][o.e.n.Node] stopped
2022.10.20 21:57:09 INFO  es[][o.e.n.Node] closing ...
2022.10.20 21:57:09 INFO  es[][o.e.n.Node] closed

I thing the most important line is:

2022.10.20 21:57:09 INFO  es[][o.e.g.GatewayService] recovered [0] indices into cluster_state

sonar.log

2022.10.20 21:57:17 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/opt/sonarqube/9.7.0.61563/elasticsearch]: /opt/sonarqube/9.7.0.61563/elasticsearch/bin/elasticsearch
2022.10.20 21:57:17 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2022.10.20 21:57:18 INFO  app[][o.s.a.SchedulerImpl] Stopping SonarQube
2022.10.20 21:57:18 INFO  app[][o.s.a.SchedulerImpl] Sonarqube has been requested to stop
2022.10.20 21:57:18 INFO  app[][o.s.a.SchedulerImpl] Stopping [Compute Engine] process...
2022.10.20 21:57:18 INFO  app[][o.s.a.SchedulerImpl] Stopping [Web Server] process...
2022.10.20 21:57:18 INFO  app[][o.s.a.SchedulerImpl] Stopping [ElasticSearch] process...
2022.10.20 21:57:18 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2022.10.20 21:57:18 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2022.10.20 21:57:18 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

Here the most important:

2022.10.20 21:57:18 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2

Hey there.

Can you try removing your data/es7 folder from your installation directory and see if that makes a difference?

3

Yes, with same results as previous.

4

Same situation when I try to upgrade to 9.7.1.62043 - restarting loop.
es.log:

2022.11.01 09:37:47 INFO  es[][o.e.n.Node] version[7.17.5], pid[1846170], build[default/tar/8d61b4f7ddf931f219e3745f295ed2bbc50c8e84/2022-06-23T21:57:28.736740635Z], OS[Linux/5.15.0-52-generic/amd64], JVM[Ubuntu/OpenJDK 64-Bit Server VM/11.0.16/11.0.16+8-post-Ubuntu-0ubuntu122.04]
2022.11.01 09:37:47 INFO  es[][o.e.n.Node] JVM home [/usr/lib/jvm/java-11-openjdk-amd64]
2022.11.01 09:37:47 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseG1GC, -Djava.io.tmpdir=/opt/sonarqube/9.7.1.62043/temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=/opt/sonarqube/9.7.1.62043/temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Dcom.redhat.fips=false, -Des.enforce.bootstrap.checks=true, -Xmx512m, -Xms512m, -XX:MaxDirectMemorySize=256m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/9.7.1.62043/elasticsearch, -Des.path.conf=/opt/sonarqube/9.7.1.62043/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=false]
2022.11.01 09:37:47 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2022.11.01 09:37:47 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2022.11.01 09:37:47 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2022.11.01 09:37:47 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2022.11.01 09:37:47 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2022.11.01 09:37:47 INFO  es[][o.e.p.PluginsService] no plugins loaded
2022.11.01 09:37:47 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [40.4gb], net total_space [108.9gb], types [ext4]
2022.11.01 09:37:47 INFO  es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2022.11.01 09:37:47 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [-Bc38fH5R52ul8QbjA5OJA], cluster name [sonarqube], roles [data_frozen, master, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
2022.11.01 09:37:50 INFO  es[][o.e.t.NettyAllocator] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
2022.11.01 09:37:50 INFO  es[][o.e.i.r.RecoverySettings] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
2022.11.01 09:37:50 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and seed hosts providers [settings]
2022.11.01 09:37:50 INFO  es[][o.e.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2022.11.01 09:37:50 INFO  es[][o.e.n.Node] initialized
2022.11.01 09:37:50 INFO  es[][o.e.n.Node] starting ...
2022.11.01 09:37:50 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:38623}, bound_addresses {127.0.0.1:38623}
2022.11.01 09:37:50 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2022.11.01 09:37:50 INFO  es[][o.e.c.c.Coordinator] setting initial configuration to VotingConfiguration{-Bc38fH5R52ul8QbjA5OJA}
2022.11.01 09:37:50 INFO  es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined)[{sonarqube}{-Bc38fH5R52ul8QbjA5OJA}{MiH9ABduS8-o7JtzVmYMyA}{127.0.0.1}{127.0.0.1:38623}{cdfhimrsw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{sonarqube}{-Bc38fH5R52ul8QbjA5OJA}{MiH9ABduS8-o7JtzVmYMyA}{127.0.0.1}{127.0.0.1:38623}{cdfhimrsw}]}
2022.11.01 09:37:50 INFO  es[][o.e.c.c.CoordinationState] cluster UUID set to [sr7NfXFURGWGb2-HWgHNWg]
2022.11.01 09:37:50 INFO  es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{-Bc38fH5R52ul8QbjA5OJA}{MiH9ABduS8-o7JtzVmYMyA}{127.0.0.1}{127.0.0.1:38623}{cdfhimrsw}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
2022.11.01 09:37:51 INFO  es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2022.11.01 09:37:51 INFO  es[][o.e.n.Node] started
2022.11.01 09:37:51 INFO  es[][o.e.g.GatewayService] recovered [0] indices into cluster_state
2022.11.01 09:38:00 INFO  es[][o.e.n.Node] stopping ...
2022.11.01 09:38:00 INFO  es[][o.e.n.Node] stopped
2022.11.01 09:38:00 INFO  es[][o.e.n.Node] closing ...
2022.11.01 09:38:00 INFO  es[][o.e.n.Node] closed

sonar.log:

2022.11.01 09:37:45 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/9.7.1.62043/temp
2022.11.01 09:37:45 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:38623]
2022.11.01 09:37:45 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/opt/sonarqube/9.7.1.62043/elasticsearch]: /opt/sonarqube/9.7.1.62043/elasticsearch/bin/elasticsearch
2022.11.01 09:37:45 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2022.11.01 09:37:51 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2022.11.01 09:37:51 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [/opt/sonarqube/9.7.1.62043]: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/9.7.1.62043/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -javaagent:./extensions/plugins/sonarqube-community-branch-plugin-1.12.0.jar=web -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.7.1.62043.jar:/opt/sonarqube/9.7.1.62043/lib/jdbc/postgresql/postgresql-42.4.1.jar org.sonar.server.app.WebServer /opt/sonarqube/9.7.1.62043/temp/sq-process12996528525982795996properties
2022.11.01 09:38:00 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [Web Server]: 134
2022.11.01 09:38:00 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2022.11.01 09:38:00 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2022.11.01 09:38:00 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2022.11.01 09:38:00 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
5

Your latest logs show that the web process is stopping first. What do you find in your web.log file?

6

No web.log file. I have only es.log and sonar.log in logs folder.