Hello.
We are using sonarqube-10.4.1.88267 with developer edition on our own server.
The server is as next.
Linux build101 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
The sonarqube works for months, but in recent days, we encountered the issue that we can’t boot up our SonarQube anymore.
We investigated things, and it seems like elastic search is in crash loop.
Logs we found in es.log are as next.
2024.07.09 01:32:02 INFO es[][o.e.n.Node] version[8.11.0], pid[1246068], build[tar/d9ec3fa628c7b0ba3d25692e277ba26814820b20/2023-11-04T10:04:57.184859352Z], OS[Linux/5.4.0-29-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/17.0.10/17.0.10+11-LTS-240]
2024.07.09 01:32:02 INFO es[][o.e.n.Node] JVM home [/usr/lib/jvm/jdk-17-oracle-x64], using bundled JDK [false]
2024.07.09 01:32:02 INFO es[][o.e.n.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, -XX:+UseG1GC, -Djava.io.tmpdir=/opt/sonarqube-10.4.1.88267/temp, -XX:ErrorFile=/opt/sonarqube-10.4.1.88267/logs/es_hs_err_pid%p.log, -Xlog:disable, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=/opt/sonarqube-10.4.1.88267/temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Dcom.redhat.fips=false, -Des.enforce.bootstrap.checks=true, -Xmx512m, -Xms512m, -XX:MaxDirectMemorySize=256m, -XX:+HeapDumpOnOutOfMemoryError, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=tar, --module-path=/opt/sonarqube-10.4.1.88267/elasticsearch/lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [x-pack-core]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [old-lucene-versions]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [lang-painless]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [parent-join]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [rest-root]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [x-pack-redact]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [reindex]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [analysis-common]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [x-pack-security]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [transport-netty4]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [aggregations]
2024.07.09 01:32:03 INFO es[][o.e.p.PluginsService] loaded module [apm]
2024.07.09 01:32:04 INFO es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [467.8gb], net total_space [722gb], types [ext4]
2024.07.09 01:32:04 INFO es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2024.07.09 01:32:04 INFO es[][o.e.n.Node] node name [sonarqube], node ID [kFEk9MuqQquqqaGgFOA0CA], cluster name [sonarqube], roles [data, data_cold, ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client]
2024.07.09 01:32:05 INFO es[][o.e.x.s.Security] Security is disabled
2024.07.09 01:32:05 INFO es[][o.e.t.n.NettyAllocator] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=512mb}]
2024.07.09 01:32:05 INFO es[][o.e.i.r.RecoverySettings] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
2024.07.09 01:32:05 INFO es[][o.e.d.DiscoveryModule] using discovery type [single-node] and seed hosts providers [settings]
2024.07.09 01:32:05 INFO es[][o.e.n.Node] initialized
2024.07.09 01:32:05 INFO es[][o.e.n.Node] starting ...
2024.07.09 01:32:05 INFO es[][o.e.t.TransportService] publish_address {127.0.0.1:45113}, bound_addresses {127.0.0.1:45113}
2024.07.09 01:32:05 INFO es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2024.07.09 01:32:05 INFO es[][o.e.c.c.ClusterBootstrapService] this node is locked into cluster UUID [xjt9qQ7DQrW79V5gKIyq2A] and will not attempt further cluster bootstrapping
2024.07.09 01:32:05 INFO es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined in term 36)[_FINISH_ELECTION_, {sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003} completing election], term: 36, version: 142, delta: master node changed {previous [], current [{sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}]}
2024.07.09 01:32:05 INFO es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}]}, term: 36, version: 142, reason: Publication{term=36, version=142}
2024.07.09 01:32:05 INFO es[][o.e.c.f.AbstractFileWatchingService] starting file watcher ...
2024.07.09 01:32:05 INFO es[][o.e.c.f.AbstractFileWatchingService] file settings service up and running [tid=41]
2024.07.09 01:32:05 INFO es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2024.07.09 01:32:05 INFO es[][o.e.c.c.NodeJoinExecutor] node-join: [{sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}] with reason [completing election]
2024.07.09 01:32:05 INFO es[][o.e.n.Node] started {sonarqube}{kFEk9MuqQquqqaGgFOA0CA}{oinYuwdVRoOkaOKTfTwBjQ}{sonarqube}{127.0.0.1}{127.0.0.1:45113}{cdfhilmrstw}{8.11.0}{7000099-8500003}{rack_id=sonarqube, xpack.installed=true}
2024.07.09 01:32:05 INFO es[][o.e.n.Node] stopping ...
2024.07.09 01:32:05 INFO es[][o.e.c.f.AbstractFileWatchingService] shutting down watcher thread
2024.07.09 01:32:05 INFO es[][o.e.c.f.AbstractFileWatchingService] watcher service stopped
2024.07.09 01:32:05 INFO es[][o.e.n.Node] stopped
2024.07.09 01:32:05 INFO es[][o.e.n.Node] closing ...
2024.07.09 01:32:05 INFO es[][o.e.n.Node] closed
Could you please give us some of advice for this issue ?