Crash loop when starting our SonarQube environment [Resolved]

Hello.
We are using sonarqube-10.4.1.88267 with developer edition on our own server.

The server is as next.
Linux build101 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

The sonarqube works for months, but in recent days, we encountered the issue that we can’t boot up our SonarQube anymore.

We investigated things, and it seems like elastic search is in crash loop.

Logs we found in es.log are as next.

2024.07.09 01:32:02 INFO  es[][o.e.n.Node] version[8.11.0], pid[1246068], build[tar/d9ec3fa628c7b0ba3d25692e277ba26814820b20/2023-11-04T10:04:57.184859352Z], OS[Linux/5.4.0-29-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/17.0.10/17.0.10+11-LTS-240]
2024.07.09 01:32:02 INFO  es[][o.e.n.Node] JVM home [/usr/lib/jvm/jdk-17-oracle-x64], using bundled JDK [false]
2024.07.09 01:32:02 INFO  es[][o.e.n.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, -XX:+UseG1GC, -Djava.io.tmpdir=/opt/sonarqube-10.4.1.88267/temp, -XX:ErrorFile=/opt/sonarqube-10.4.1.88267/logs/es_hs_err_pid%p.log, -Xlog:disable, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=/opt/sonarqube-10.4.1.88267/temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Dcom.redhat.fips=false, -Des.enforce.bootstrap.checks=true, -Xmx512m, -Xms512m, -XX:MaxDirectMemorySize=256m, -XX:+HeapDumpOnOutOfMemoryError, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=tar, --module-path=/opt/sonarqube-10.4.1.88267/elasticsearch/lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [x-pack-core]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [old-lucene-versions]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [rest-root]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [x-pack-redact]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [x-pack-security]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [aggregations]
2024.07.09 01:32:03 INFO  es[][o.e.p.PluginsService] loaded module [apm]
2024.07.09 01:32:04 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [467.8gb], net total_space [722gb], types [ext4]
2024.07.09 01:32:04 INFO  es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2024.07.09 01:32:04 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [kFEk9MuqQquqqaGgFOA0CA], cluster name [sonarqube], roles [data, data_cold, ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client]
2024.07.09 01:32:05 INFO  es[][o.e.x.s.Security] Security is disabled
2024.07.09 01:32:05 INFO  es[][o.e.t.n.NettyAllocator] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=512mb}]
2024.07.09 01:32:05 INFO  es[][o.e.i.r.RecoverySettings] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
2024.07.09 01:32:05 INFO  es[][o.e.d.DiscoveryModule] using discovery type [single-node] and seed hosts providers [settings]
2024.07.09 01:32:05 INFO  es[][o.e.n.Node] initialized
2024.07.09 01:32:05 INFO  es[][o.e.n.Node] starting ...
2024.07.09 01:32:05 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:45113}, bound_addresses {127.0.0.1:45113}
2024.07.09 01:32:05 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2024.07.09 01:32:05 INFO  es[][o.e.c.c.ClusterBootstrapService] this node is locked into cluster UUID [xjt9qQ7DQrW79V5gKIyq2A] and will not attempt further cluster bootstrapping
2024.07.09 01:32:05 INFO  es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined in term 36)[_FINISH_ELECTION_, {sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003} completing election], term: 36, version: 142, delta: master node changed {previous [], current [{sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}]}
2024.07.09 01:32:05 INFO  es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}]}, term: 36, version: 142, reason: Publication{term=36, version=142}
2024.07.09 01:32:05 INFO  es[][o.e.c.f.AbstractFileWatchingService] starting file watcher ...
2024.07.09 01:32:05 INFO  es[][o.e.c.f.AbstractFileWatchingService] file settings service up and running [tid=41]
2024.07.09 01:32:05 INFO  es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2024.07.09 01:32:05 INFO  es[][o.e.c.c.NodeJoinExecutor] node-join: [{sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}] with reason [completing election]
2024.07.09 01:32:05 INFO  es[][o.e.n.Node] started {sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}{rack_id=sonarqube, xpack.installed=true}
2024.07.09 01:32:05 INFO  es[][o.e.n.Node] stopping ...
2024.07.09 01:32:05 INFO  es[][o.e.c.f.AbstractFileWatchingService] shutting down watcher thread
2024.07.09 01:32:05 INFO  es[][o.e.c.f.AbstractFileWatchingService] watcher service stopped
2024.07.09 01:32:05 INFO  es[][o.e.n.Node] stopped
2024.07.09 01:32:05 INFO  es[][o.e.n.Node] closing ...
2024.07.09 01:32:05 INFO  es[][o.e.n.Node] closed

Could you please give us some of advice for this issue ?

Sorry for disturbing.
We found this is different topic than we expected.

There was an issue in postgres which we were using and it was actual reason of problem.

Let me resolve this issue.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.