SonarQube 9.9LTS prod stopped working this night

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    9.9LTS Enterprise Edition
  • how is SonarQube deployed: zip, Docker, Helm
    Azure App Service
  • what are you trying to achieve
    Trying to run the service again.
  • what have you tried so far to achieve this
    Restarted the app service, tried to run StartSona.bat. Nothing has been changed on our side. It just stopped to work this night.

last web.log

2023.10.02 03:06:09 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2023.10.02 03:06:09 INFO  web[][o.s.s.e.CoreExtensionStopper] Stopping Governance
2023.10.02 03:06:09 INFO  web[][o.s.s.e.CoreExtensionStopper] Stopping Governance (done) | time=0ms
2023.10.02 03:06:09 INFO  web[][o.s.s.n.NotificationDaemon] Notification service stopped
2023.10.02 03:06:10 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2023.10.02 03:06:10 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [pool-17-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-12] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-14] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-16] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@17.0.4.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)\n java.base@17.0.4.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)\n java.base@17.0.4.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@17.0.4.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)
2023.10.02 03:06:10 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [OkHttp TaskRunner] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.4.1/java.lang.Object.wait(Native Method)\n java.base@17.0.4.1/java.lang.Object.wait(Object.java:472)\n app//okhttp3.internal.concurrent.TaskRunner$RealBackend.coordinatorWait(TaskRunner.kt:294)\n app//okhttp3.internal.concurrent.TaskRunner.awaitTaskToRun(TaskRunner.kt:218)\n app//okhttp3.internal.concurrent.TaskRunner$runnable$1.run(TaskRunner.kt:59)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)

last ce.log

2023.10.02 03:06:09 INFO  ce[][o.s.p.ProcessEntryPoint] Hard stopping process
2023.10.02 03:06:09 INFO  ce[][o.s.ce.app.CeServer] Compute Engine is stopping...
2023.10.02 03:06:09 INFO  ce[][o.s.c.t.CeProcessingSchedulerImpl] Hard stopping workers...
2023.10.02 03:06:09 INFO  ce[][o.s.s.e.CoreExtensionStopper] Stopping Governance
2023.10.02 03:06:09 INFO  ce[][o.s.s.e.CoreExtensionStopper] Stopping Governance (done) | time=0ms
2023.10.02 03:06:09 INFO  ce[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2023.10.02 03:06:09 INFO  ce[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.
2023.10.02 03:06:09 INFO  ce[][o.s.ce.app.CeServer] Compute Engine is stopped

tried to startsonar.bat

.\StartSonar.bat
Starting SonarQube...
2023.10.02 06:44:02 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory C:\home\site\wwwroot\sonarqube-9.9.0.65466\temp
2023.10.02 06:44:02 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:52087]
2023.10.02 06:44:02 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [C:\home\site\wwwroot\sonarqube-9.9.0.65466\elasticsearch]: C:\Program Files\Java\microsoft-jdk-17.0.4.1\bin\java -XX:+UseG1GC -Djava.io.tmpdir=C:\home\site\wwwroot\sonarqube-9.9.0.65466\temp -XX:ErrorFile=C:\home\site\wwwroot\sonarqube-9.9.0.65466\logs\es_hs_err_pid%p.log -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.tmpdir=C:\home\site\wwwroot\sonarqube-9.9.0.65466\temp -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=COMPAT -Dcom.redhat.fips=false -Des.enforce.bootstrap.checks=true -Xmx2G -Xms2G -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=C:\home\site\wwwroot\sonarqube-9.9.0.65466\elasticsearch -Des.path.conf=C:\home\site\wwwroot\sonarqube-9.9.0.65466\temp\conf\es -cp lib/* org.elasticsearch.bootstrap.Elasticsearch
2023.10.02 06:44:02 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\home\site\wwwroot\sonarqube-9.9.0.65466\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:328)
	at org.elasticsearch.node.Node.<init>(Node.java:429)
	at org.elasticsearch.node.Node.<init>(Node.java:309)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
	at org.elasticsearch.cli.Command.main(Command.java:77)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at C:\home\site\wwwroot\sonarqube-9.9.0.65466\logs\sonarqube.log
2023.10.02 06:44:07 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 1
2023.10.02 06:44:07 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.10.02 06:44:07 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
PS C:\home\site\wwwroot\sonarqube-9.9.0.65466\bin\windows-x86-64>

sonar.log

2023.10.02 06:50:04 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.02 06:50:05 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.02 06:50:06 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.02 06:50:06 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.02 06:50:07 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.02 06:50:07 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 1
2023.10.02 06:50:07 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.10.02 06:50:07 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

Trying to start SonarService

PS C:\home\site\wwwroot\sonarqube-9.9.0.65466\bin\windows-x86-64> .\SonarService.bat start
.\SonarService.bat start
2023-10-02 08:05:22,870 FATAL - Service SonarQube was not found on computer '.'.

Also, getting these

500 - The request timed out.

The web server failed to respond within the specified time.

Hi,

I’m not surprised you’re getting 500 errors back from the server; it seems to be in a bad state.

In your logs, I’m seeing this:

And a lot of this:

So it looks like SonarQube doesn’t have permission to run. It needs to be able to open sockets, and it needs file permissions on its directory. From the errors you’ve reported, it doesn’t look like it has perms for either of those things.

 
HTH,
Ann

1 Like

Hi Ann. Thanks for the reply. So, far managed to scale upp the app service to P3V3 in Azure. But, now es.log files are 20 GB huge. Also, we are getting out of mem exceptions. Trying to get these files locally to see what is causing the issue.

Also, the indexing is done, but there are failures

Error logs we are getting in background tasks

Error Details
java.lang.IllegalStateException: Unrecoverable indexation failures: 15 errors among 15 requests. Check Elasticsearch logs for further details.
	at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:42)
	at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:131)
	at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:261)
	at org.sonar.server.issue.index.IssueIndexer.indexOnAnalysis(IssueIndexer.java:115)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.lambda$execute$1(IndexIssuesStep.java:52)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.execute(IndexIssuesStep.java:48)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IssueSyncTaskProcessor.process(IssueSyncTaskProcessor.java:60)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Here are the latest logs.

2023.10.04 07:24:42 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.04 07:24:42 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.04 07:24:43 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.04 07:24:43 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.04 07:24:44 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)

And these are the sonar.log

Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.10.04 08:49:03 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)

@ganncamp what does the es.log deletion does in practice? this means we delete the indexed data? or? these files are now both from today and yesterday at 20 GB.

Hello,

The indexed data are in the data/es8 folder. You can safely delete the es.log file, which “only” contains logs about ElasticSearch activity.

Once you made sure you have enough disk storage & the proper permissions applied to the file system, you might want to consider forcing an ElasticSearch reindex (as it seems many project data indexing failed).

2 Likes

Thanks @pierreguillot. Will try this out and let you know about the results.

Can you suggest the documentation for this or any other documentation about the permissions? Thanks.

I did stop the app service, deleted es.log both files and cleared the es7 directory. Reindexing now.

Have been able to recover. Stopped the app service, deleted es.log, cleared es7 contents and did run the app service. So, thanks :slight_smile:

1 Like

Great, thanks for the update!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.