At startup SonarQuube 9.3, Connection refused to elasticsearch

Hi,

After upgrade sonarqube 8.9.X (LTS version) to SonarQube 9.3.x, i have a problem at startup.
Connection refused to the elasticsearch database

logs in ce.log

--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

2022.02.03 11:01:56 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /home/sonar/temp
2022.02.03 11:01:56 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:44705]
2022.02.03 11:01:56 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/home/sonar/sonarqube-9.3.0.51899/elasticsearch]: /home/sonar/sonarqube-9.3.0.51899/elasticsearch/bin/elasticsearch
2022.02.03 11:01:56 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2022.02.03 11:01:57 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.ConnectException: Connexion refusée
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:97)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:82)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:67)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:220)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:285)
Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException: Connexion refusée
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.ConnectException: Connexion refusée
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:829)
2022.02.03 11:01:58 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.ConnectException: Connexion refusée
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:97)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:82)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:67)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:220)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:285)
Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException: Connexion refusée
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.ConnectException: Connexion refusée
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:829)
2022.02.03 11:01:58 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.ConnectException: Connexion refusée
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:97)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:82)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:67)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:220)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:285)
[...]

But sonarqube start and the index in elasticsearch is ok !
Logs in es.log

2022.02.03 11:01:14 INFO  es[][o.e.n.Node] closed
2022.02.03 11:01:58 INFO  es[][o.e.n.Node] version[7.16.3], pid[2352], build[default/tar/4e6e4eab2297e949ec994e688dad46290d018022/2022-01-06T23:43:02.825887787Z], OS[Linux/4.19.0-9-amd64/amd64], JVM[Debian/OpenJDK 64-Bit Server VM/11.0.13/11.0.13+8-post-Debian-1deb10u1]
2022.02.03 11:01:58 INFO  es[][o.e.n.Node] JVM home [/usr/lib/jvm/java-11-openjdk-amd64]
2022.02.03 11:01:58 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseG1GC, -Djava.io.tmpdir=/home/sonar/temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=/home/sonar/temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Dcom.redhat.fips=false, -Des.enforce.bootstrap.checks=true, -Xmx512m, -Xms512m, -XX:MaxDirectMemorySize=256m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/sonar/sonarqube-9.3.0.51899/elasticsearch, -Des.path.conf=/home/sonar/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=false]
2022.02.03 11:01:58 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2022.02.03 11:01:58 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2022.02.03 11:01:58 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2022.02.03 11:01:58 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2022.02.03 11:01:58 INFO  es[][o.e.p.PluginsService] no plugins loaded
2022.02.03 11:01:59 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/home (/dev/vdb1)]], net usable_space [36.8gb], net total_space [97.9gb], types [ext4]
2022.02.03 11:01:59 INFO  es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2022.02.03 11:01:59 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [lXsS9-zpTda0GrtBRS4-kw], cluster name [sonarqube], roles [data_frozen, master, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
2022.02.03 11:02:02 INFO  es[][o.e.t.NettyAllocator] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
2022.02.03 11:02:02 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and seed hosts providers [settings]
2022.02.03 11:02:03 INFO  es[][o.e.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2022.02.03 11:02:03 INFO  es[][o.e.n.Node] initialized
2022.02.03 11:02:03 INFO  es[][o.e.n.Node] starting ...
2022.02.03 11:02:03 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:44705}, bound_addresses {127.0.0.1:44705}
2022.02.03 11:02:04 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2022.02.03 11:02:04 INFO  es[][o.e.c.c.Coordinator] cluster UUID [gUl1cvLWRFGz9GY32hZalw]
2022.02.03 11:02:04 INFO  es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined)[{sonarqube}{lXsS9-zpTda0GrtBRS4-kw}{-4BtmVK3QSKjyQUQL1GGNQ}{127.0.0.1}{127.0.0.1:44705}{cdfhimrsw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 65, delta: master node changed {previous [], current [{sonarqube}{lXsS9-zpTda0GrtBRS4-kw}{-4BtmVK3QSKjyQUQL1GGNQ}{127.0.0.1}{127.0.0.1:44705}{cdfhimrsw}]}
2022.02.03 11:02:04 INFO  es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{lXsS9-zpTda0GrtBRS4-kw}{-4BtmVK3QSKjyQUQL1GGNQ}{127.0.0.1}{127.0.0.1:44705}{cdfhimrsw}]}, term: 3, version: 65, reason: Publication{term=3, version=65}
2022.02.03 11:02:04 INFO  es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2022.02.03 11:02:04 INFO  es[][o.e.n.Node] started
2022.02.03 11:02:04 INFO  es[][o.e.g.GatewayService] recovered [7] indices into cluster_state
2022.02.03 11:02:06 INFO  es[][o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[components][2], [components][0], [metadatas][0]]]).

How to correct this error please ?

Thanks for your help,

Hi,

Sometimes these ce.log errors appear transiently as the processes wait for each other to start up. What do you see in your sonar.log? Is the instance running now?

 
Ann