Fail to execute es requestES refresh request on indices 'components'

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    SonarQube 9.5 latest, Scanner 4.7
  • what are you trying to achieve
    Regular scan
  • what have you tried so far to achieve this
Error Details
org.sonar.server.es.ElasticsearchException: Fail to execute es requestES refresh request on indices 'components'
	at org.sonar.server.es.EsClient.execute(EsClient.java:313)
	at org.sonar.server.es.EsClient.refresh(EsClient.java:155)
	at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:128)
	at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:170)
	at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
	at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:107)
	at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:70)
	at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
	at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:54)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:80)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:71)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:58)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:926)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:299)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:287)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.IndicesClient.refresh(IndicesClient.java:900)
	at org.sonar.server.es.EsClient.lambda$refresh$9(EsClient.java:155)
	at org.sonar.server.es.EsClient.execute(EsClient.java:311)
	... 27 more
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
	at org.apache.http.util.Asserts.check(Asserts.java:46)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:295)
	... 35 more

Please let me know how to fix this

Thanks
Shiva

It seems like something has gone wrong with your Elasticsearch index.

I would recommend checking logs/es.log file of your SonarQube installation directory for additional information.

Do we need to check anything specific? in it
Need to share the log in private

2022.07.27 09:26:29 DEBUG es[i.n.u.i.l.InternalLoggerFactory] Using Log4J2 as the default logging framework
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent0] -Dio.netty.noUnsafe: true
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent0] sun.misc.Unsafe: unavailable (io.netty.noUnsafe)
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent0] Java version: 11
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent0] java.nio.DirectByteBuffer.(long, int): unavailable
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent] maxDirectMemory: 536870912 bytes (maybe)
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent] -Dio.netty.tmpdir: /opt/Sonar/sonarqube-9.5.0.56709/temp (java.io.tmpdir)
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent] -Dio.netty.bitMode: 64 (sun.arch.data.model)
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent] -Dio.netty.maxDirectMemory: -1 bytes
2022.07.27 09:26:29 DEBUG es[i.n.u.i.PlatformDependent] -Dio.netty.uninitializedArrayAllocationThreshold: -1
2022.07.27 09:26:29 DEBUG es[i.n.u.i.CleanerJava9] java.nio.ByteBuffer.cleaner(): unavailable
java.lang.UnsupportedOperationException: sun.misc.Unsafe unavailable
at io.netty.util.internal.CleanerJava9.(CleanerJava9.java:68) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.PlatformDependent.(PlatformDependent.java:193) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.ConstantPool.(ConstantPool.java:34) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.AttributeKey$1.(AttributeKey.java:27) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.AttributeKey.(AttributeKey.java:27) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.(Netty4HttpServerTransport.java:294) [transport-netty4-client-7.17.1.jar:7.17.1]
at org.elasticsearch.transport.Netty4Plugin.getSettings(Netty4Plugin.java:45) [transport-netty4-client-7.17.1.jar:7.17.1]
at org.elasticsearch.plugins.PluginsService.lambda$getPluginSettings$0(PluginsService.java:84) [elasticsearch-7.17.1.jar:7.17.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) [?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) [?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) [?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) [?:?]

These are some logs from es.log

Can anyone help me on this please let me know.
I got the same error today. I tried to modify “sonar.es.javaOpts=-Xmx4g -Xms4g -XX:+HeapDumpOnOutOfMemoryError”
It did not work.

Not sure how to fix this.

[spoiler]
[details="Summary"]
Status	Task	ID	Submitter		Submitted	Started	Finished	Duration
FAILED	V12_SECURITY/SYS 	AYJEI9N8FqzeVV45UZTd	admin	28-Jul-22	5:29:35 AM	7:43:06 AM	7:43:37 AM	31s
FAILED	V12_SECURITY/VERSION_SCRIPTS 	AYJEI3MmFqzeVV45UZTc	admin		5:29:10 AM	7:41:51 AM	7:43:05 AM	1min 14s
FAILED	V12_SECURITY/PM 	AYJEIw3xFqzeVV45UZTb	admin		5:28:44 AM	7:41:21 AM	7:41:51 AM	30s
FAILED	V12_SECURITY/CMIC 	AYJEIqyPFqzeVV45UZTa	admin		5:28:19 AM	7:40:50 AM	7:41:20 AM	30s
FAILED	V12_SECURITY/BI 	AYJEIk8PFqzeVV45UZTZ	admin		5:27:55 AM	7:40:22 AM	7:40:50 AM	27s
FAILED	V12_SECURITY/PY 	AYJEIfDpFqzeVV45UZTY	admin		5:27:31 AM	7:39:55 AM	7:40:22 AM	26s
FAILED	V12_SECURITY/SD 	AYJEIZVFFqzeVV45UZTX	admin		5:27:08 AM	7:39:26 AM	7:39:55 AM	29s
FAILED	V12_SECURITY/FORMS 	AYJEITORFqzeVV45UZTW	admin		5:26:42 AM	7:39:00 AM	7:39:25 AM	25s
FAILED	V12_SECURITY/SC 	AYJEINkxFqzeVV45UZTV	admin		5:26:19 AM	7:38:33 AM	7:38:59 AM	26s
FAILED	V12_SECURITY/MISCELLANEOUS 	AYJEIH2bFqzeVV45UZTU	admin		5:25:56 AM	7:37:53 AM	7:38:33 AM	39s
FAILED	V12_SECURITY/TM 	AYJEIB3yFqzeVV45UZTT	admin		5:25:31 AM	7:37:26 AM	7:37:53 AM	27s
FAILED	V12/JC 	AYJEH6t4FqzeVV45UZTS	admin		5:25:07 AM	7:32:39 AM	7:37:26 AM	4min 46s
FAILED	V12/JB 	AYJEHvj4FqzeVV45UZTR	admin		5:24:19 AM	7:30:42 AM	7:32:39 AM	1min 56s
FAILED	V12/BI 	AYJEHl1EFqzeVV45UZTQ	admin		5:23:43 AM	7:28:58 AM	7:30:42 AM	1min 44s
FAILED	V12/WKF 	AYJEHZslFqzeVV45UZTP	admin		5:22:48 AM	7:27:32 AM	7:28:57 AM	1min 25s
FAILED	V12/FORMS 	AYJEHSgwFqzeVV45UZTO	admin		5:22:17 AM	7:27:07 AM	7:27:32 AM	25s
FAILED	V12/FA 	AYJEHMqvFqzeVV45UZTN	admin		5:21:54 AM	7:26:23 AM	7:27:06 AM	43s
FAILED	V12/WO 	AYJEHGbKFqzeVV45UZTM	admin		5:21:28 AM	7:25:57 AM	7:26:23 AM	25s
FAILED	V12/BD 	AYJEHAmzFqzeVV45UZTL	admin		5:21:04 AM	7:25:10 AM	7:25:56 AM	45s
FAILED	V12/MS 	AYJEG6IsFqzeVV45UZTK	admin		5:20:39 AM	7:23:48 AM	7:25:10 AM	1min 21s
FAILED	V12/IMG 	AYJEGypdFqzeVV45UZTJ	admin		5:20:08 AM	7:22:09 AM	7:23:48 AM	1min 39s
FAILED	V12/OM 	AYJEGrPNFqzeVV45UZTI	admin		5:19:37 AM	7:20:45 AM	7:22:08 AM	1min 23s
FAILED	V12/GT 	AYJEGkKkFqzeVV45UZTH	admin		5:19:08 AM	7:19:53 AM	7:20:44 AM	51s
FAILED	V12/UE 	AYJEGdpIFqzeVV45UZTG	admin		5:18:41 AM	7:19:07 AM	7:19:53 AM	45s
FAILED	V12/SD 	AYJEGUsJFqzeVV45UZTF	admin		5:18:15 AM	7:16:20 AM	7:19:07 AM	2min 47s

[/details]
[/spoiler]

I would suggest you try clearing your Elasticsearch index and letting it rebuild.

  • Stop your SonarQube instance
  • Delete the data/es7 folder from your SonarQube installation directory
  • Start your SonarQube instance

I tried it already
But the issue is not resolved yet

Still projects are failing to publish on SonarQube

Java heapspace issue
I tried with modifying my to -Xmx=4g

No luck with that
Thinking of going with 8g now.
Please let me know if it solve my issue

Thanks

sonar.es.javaOpts=-Xmx8g -Xms8g -XX:+HeapDumpOnOutOfMemoryError
sonar.search.javaOpts=-Xmx8g -Xms8g -XX:MaxDirectMemorySize=2g -XX:+HeapDumpOnOutOfMemoryError

even this one did not work

It looks like the OutOfMemory is coming from the ComputeEngine, so I would suggest adjusting sonar.ce.javaOpts – be careful that you aren’t allocating more memory than is actually available on your system.

1 Like