We are experiencing recurring issues with SonarQube where an ElasticsearchException is thrown, specifically when executing ES refresh requests on indices ‘components’. Below is the detailed error log:
org.sonar.server.es.ElasticsearchException: Fail to execute es requestES refresh request on indices 'components'
at org.sonar.server.es.EsClient.execute(EsClient.java:313)
at org.sonar.server.es.EsClient.refresh(EsClient.java:155)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:128)
at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:164)
at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:106)
at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:69)
at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:54)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:940)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
at org.elasticsearch.client.IndicesClient.refresh(IndicesClient.java:900)
at org.sonar.server.es.EsClient.lambda$refresh$9(EsClient.java:155)
at org.sonar.server.es.EsClient.execute(EsClient.java:311)
... 27 common frames omitted
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
... 35 common frames omitted
- We are using Sonar version - Developer EditionVersion 9.9 (build 65466)
- Sonar scanner version - SonarScanner 4.8.0.2856
- SonarQube deployed using zip.
- We tried the following work arounds,
Initially, we restarted the SonarQube server, which resolved the issue temporarily.
Upon encountering the issue again, we followed the steps in the SonarQube community thread `https://community.sonarsource.com/t/fail-to-execute-es-requestes-refresh-request-on-indices-components/68817`:
Stopped the SonarQube server.
Cleared the $SONAR_HOME/data/es7 path.
Restarted the SonarQube server.
This resolved the issue temporarily once again.
After performing the restart, we noticed several .hprof files had been created, though it is unclear if they were generated as a result of our actions.
We need assistance in permanently resolving this issue. Please provide guidance or potential solutions to prevent this error from recurring.