SonarQube - ES Not starting when upgrading from 8.9.1 to 9.1

SonarQube
#1

ES.log

dfhimrsw} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 63, version: 1275, delta: master node changed {previous , current [{sonarqube}{c_gSsmt3TKWJfbPYAywLOQ}{YhtHq6icQDqxplmqNd7THQ}{127.0.0.1}{127.0.0.1:33145}{cdfhimrsw}]}
2021.10.15 04:05:40 INFO es[o.e.c.s.ClusterApplierService] master node changed {previous , current [{sonarqube}{c_gSsmt3TKWJfbPYAywLOQ}{YhtHq6icQDqxplmqNd7THQ}{127.0.0.1}{127.0.0.1:33145}{cdfhimrsw}]}, term: 63, version: 1275, reason: Publication{term=63, version=1275}
2021.10.15 04:05:41 INFO es[o.e.g.GatewayService] recovered [7] indices into cluster_state
2021.10.15 04:05:41 INFO es[o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2021.10.15 04:05:41 INFO es[o.e.n.Node] started
2021.10.15 04:05:44 INFO es[o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[metadatas][0]]]).
2021.10.15 04:06:18 INFO es[o.e.n.Node] stopping …
2021.10.15 04:06:18 INFO es[o.e.n.Node] stopped
2021.10.15 04:06:18 INFO es[o.e.n.Node] closing …
2021.10.15 04:06:18 INFO es[o.e.n.Node] closed
2021.10.15 04:06:27 INFO es[o.e.n.Node] version[7.14.1], pid[31423], build[default/tar/66b55ebfa59c92c15db3f69a335d500018b3331e/2021-08-26T09:01:05.390870785Z], OS[Linux/3.10.0-1160.11.1.el7.x86_64/amd64], JVM[Red Hat, Inc./OpenJDK 64-Bit Server VM/11.0.9.1/11.0.9.1+1-LTS]
2021.10.15 04:06:27 INFO es[o.e.n.Node] JVM home [/usr/lib/jvm/java-11-openjdk-11.0.9.11-2.el7_9.x86_64]
2021.10.15 04:06:27 INFO es[o.e.n.Node] JVM arguments [-XX:+UseG1GC, -Djava.io.tmpdir=/usr/local/sonar/temp, -XX:ErrorFile=…/logs/es_hs_err_pid%p.log, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Des.enforce.bootstrap.checks=true, -Xmx2G, -Xms2G, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/local/sonar/elasticsearch, -Des.path.conf=/usr/local/sonar/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=false]
2021.10.15 04:06:28 INFO es[o.e.p.PluginsService] loaded module [analysis-common]
2021.10.15 04:06:28 INFO es[o.e.p.PluginsService] loaded module [lang-painless]
2021.10.15 04:06:28 INFO es[o.e.p.PluginsService] loaded module [parent-join]
2021.10.15 04:06:28 INFO es[o.e.p.PluginsService] loaded module [percolator]
2021.10.15 04:06:28 INFO es[o.e.p.PluginsService] loaded module [runtime-fields-common]
2021.10.15 04:06:28 INFO es[o.e.p.PluginsService] loaded module [transport-netty4]
2021.10.15 04:06:28 INFO es[o.e.p.PluginsService] no plugins loaded
2021.10.15 04:06:28 INFO es[o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [91.5gb], net total_space [99.9gb], types [xfs]
2021.10.15 04:06:28 INFO es[o.e.e.NodeEnvironment] heap size [2gb], compressed ordinary object pointers [true]
2021.10.15 04:06:28 INFO es[o.e.n.Node] node name [sonarqube], node ID [c_gSsmt3TKWJfbPYAywLOQ], cluster name [sonarqube], roles [data_frozen, master, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
2021.10.15 04:06:35 INFO es[o.e.t.NettyAllocator] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=256kb, suggested_max_allocation_size=256kb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=1mb}]
2021.10.15 04:06:35 INFO es[o.e.d.DiscoveryModule] using discovery type [zen] and seed hosts providers [settings]
2021.10.15 04:06:36 INFO es[o.e.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2021.10.15 04:06:36 INFO es[o.e.n.Node] initialized
2021.10.15 04:06:36 INFO es[o.e.n.Node] starting …
2021.10.15 04:06:36 INFO es[o.e.t.TransportService] publish_address {127.0.0.1:38315}, bound_addresses {127.0.0.1:38315}
2021.10.15 04:06:37 INFO es[o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2021.10.15 04:06:37 INFO es[o.e.c.c.Coordinator] cluster UUID [v5RMPjpFT_2VLODs51Uq8w]
2021.10.15 04:06:37 INFO es[o.e.c.s.MasterService] elected-as-master ([1] nodes joined)[{sonarqube}{c_gSsmt3TKWJfbPYAywLOQ}{diuOGG03Q9ixGqIHvNK9sA}{127.0.0.1}{127.0.0.1:38315}{cdfhimrsw} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 64, version: 1296, delta: master node changed {previous , current [{sonarqube}{c_gSsmt3TKWJfbPYAywLOQ}{diuOGG03Q9ixGqIHvNK9sA}{127.0.0.1}{127.0.0.1:38315}{cdfhimrsw}]}
2021.10.15 04:06:37 INFO es[o.e.c.s.ClusterApplierService] master node changed {previous , current [{sonarqube}{c_gSsmt3TKWJfbPYAywLOQ}{diuOGG03Q9ixGqIHvNK9sA}{127.0.0.1}{127.0.0.1:38315}{cdfhimrsw}]}, term: 64, version: 1296, reason: Publication{term=64, version=1296}
2021.10.15 04:06:38 INFO es[o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2021.10.15 04:06:38 INFO es[o.e.n.Node] started
2021.10.15 04:06:38 INFO es[o.e.g.GatewayService] recovered [7] indices into cluster_state
2021.10.15 04:06:40 INFO es[o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[components][0], [components][3], [metadatas][0]]]).

#2

Hi @Pappi_Mr ,

can you share some more information? from the logs you posted it seems like ES is starting up just fine

could the indicated stop result from another component?

#3

Sonarqube is NOT coming up online and I see this happening repeatedly in sonar.log and the above log in es.log

2021.10.15 13:48:29 INFO app[o.s.a.AppFileSystem] Cleaning or creating temp directory /usr/local/sonar/temp
2021.10.15 13:48:29 INFO app[o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:41352]
2021.10.15 13:48:30 INFO app[o.s.a.ProcessLauncherImpl] Launch process[[key=‘es’, ipcIndex=1, logFilenamePrefix=es]] from [/usr/local/sonar/elasticsearch]: /usr/local/sonar/elasticsearch/bin/elasticsearch
2021.10.15 13:48:30 INFO app[o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2021.10.15 13:48:48 INFO app[o.s.a.SchedulerImpl] Process[es] is up
2021.10.15 13:48:48 INFO app[o.s.a.ProcessLauncherImpl] Launch process[[key=‘web’, ipcIndex=2, logFilenamePrefix=web]] from [/usr/local/sonar]: /usr/lib/jvm/java-11-openjdk-11.0.9.11-2.el7_9.x86_64/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/usr/local/sonar/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Xmx1G -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.1.0.47736.jar:/usr/local/sonar/lib/jdbc/postgresql/postgresql-42.2.19.jar org.sonar.server.app.WebServer /usr/local/sonar/temp/sq-process8160704239073128173properties
2021.10.15 13:49:23 INFO app[o.s.a.SchedulerImpl] Process[web] is stopped
2021.10.15 13:49:23 WARN app[o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 143
2021.10.15 13:49:23 INFO app[o.s.a.SchedulerImpl] Process[es] is stopped
2021.10.15 13:49:23 INFO app[o.s.a.SchedulerImpl] SonarQube is stopped
← Wrapper Stopped
→ Wrapper Started as Daemon
Launching a JVM…
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.

2021.10.15 13:49:26 INFO app[o.s.a.AppFileSystem] Cleaning or creating temp directory /usr/local/sonar/temp
2021.10.15 13:49:26 INFO app[o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:38996]
2021.10.15 13:49:27 INFO app[o.s.a.ProcessLauncherImpl] Launch process[[key=‘es’, ipcIndex=1, logFilenamePrefix=es]] from [/usr/local/sonar/elasticsearch]: /usr/local/sonar/elasticsearch/bin/elasticsearch
2021.10.15 13:49:27 INFO app[o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2021.10.15 13:49:45 INFO app[o.s.a.SchedulerImpl] Process[es] is up
2021.10.15 13:49:45 INFO app[o.s.a.ProcessLauncherImpl] Launch process[[key=‘web’, ipcIndex=2, logFilenamePrefix=web]] from [/usr/local/sonar]: /usr/lib/jvm/java-11-openjdk-11.0.9.11-2.el7_9.x86_64/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/usr/local/sonar/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Xmx1G -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.1.0.47736.jar:/usr/local/sonar/lib/jdbc/postgresql/postgresql-42.2.19.jar org.sonar.server.app.WebServer /usr/local/sonar/temp/sq-process10416981152773834169properties

#4

this looks like something went wrong:

can you share all the logs in the logs directory?

#5

sonar.20211015.log (1.3 MB)
es.log (2.8 MB)

#6

web.zip (615.2 KB)

Please find all the logs attached.

#7

You have a lot of 3rd party plugins installed and one of them is causing your instance startup to fail. try to remove all 3rd party plugins for a clean start and then sort out the one that is causing this issue.

after that your best bet is to ask for help by one of the providers of this 3rd party plugin.

#8

Thanks.

#9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.