Sudden error on background tasks

Hi,

What do your server logs say?

 
Ann

es.log file show below error

2022.03.11 17:53:47 WARN  es[][stderr] java.lang.reflect.InvocationTargetException
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
2022.03.11 17:53:47 WARN  es[][stderr] Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.org.objectweb.asm")
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/java.security.AccessController.checkPermission(AccessController.java:897)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1238)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:174)
2022.03.11 17:53:47 WARN  es[][stderr]  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
2022.03.11 17:53:47 WARN  es[][stderr]  at Log4jHotPatch.asmVersion(Log4jHotPatch.java:71)
2022.03.11 17:53:47 WARN  es[][stderr]  at Log4jHotPatch.agentmain(Log4jHotPatch.java:93)

Hi @Rajesh_Mundhe,

It looks like you’re using a non-standard build of SonarQube. I suggest you try upgrading to 8.9.7 LTS or 9.3, the current version.

 
Ann

Thanks, @ganncamp for your reply.

I am already using * Version 9.3 (build 51899)

More information about this error

org.sonar.server.es.ElasticsearchException: Fail to execute es requestES refresh request on indices 'components'
   at org.sonar.server.es.EsClient.execute(EsClient.java:313)
   at org.sonar.server.es.EsClient.refresh(EsClient.java:155)
   at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:128)
   at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:170)
   at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
   at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:107)
   at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:70)
   at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
   at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:54)
   at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
   at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
   at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
   at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
   at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
   at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
   at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
   at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
   at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
   at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
   at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
   at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
   at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
   at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:904)
   at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
   at org.elasticsearch.client.RestClient.performRequest(RestClient.java:276)
   at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
   at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
   at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
   at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
   at org.elasticsearch.client.IndicesClient.refresh(IndicesClient.java:900)
   at org.sonar.server.es.EsClient.lambda$refresh$9(EsClient.java:155)
   at org.sonar.server.es.EsClient.execute(EsClient.java:311)
   ... 27 common frames omitted
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
   at org.apache.http.util.Asserts.check(Asserts.java:46)
   at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
   at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
   at org.elasticsearch.client.RestClient.performRequest(RestClient.java:284)
   ... 35 common frames omitted

Hi,

You may be using some variation on 9.3, but you’re not using the version we issued. The standard 9.3 distribution does not include

 
Ann

Thanks, @ganncamp, I am using the SonarQube developer edition. I will redeploy and cross-check again. I used https://binaries.sonarsource.com/CommercialDistribution/sonarqube-developer/sonarqube-developer-9.3.0.51899.zip link to download. I this correct version?

@ganncamp Issue with elasticsearch 7.16.3 version. This version has the Log4jHotPatch. This issue is throwing due to that. I think sonarQube should include the new elasticsearch version. I set the es Log level to TRACE in that logs it shows all details.

Hi,

Elasticsearch 7.16.3 upgraded to Log4j 2.17.1. No need for a HotPatch.

 
Ann

@ganncamp thanks for your help, it is resolved now. The issue was that the old version of Hot patch dependency was cached and considered.

@Rajesh_Mundhe and @ganncamp Could you help me on this?
I am using 9.2.3 version of Sonarqube and facing issue. Could you elaborate the resolution for the issue?

@prasad-clouduser the resolution is to upgrade to the official distribution of the latest version.

 
Ann

@prasad-clouduser Upgrade to either 9.3/9.4 version. What exact issue are you facing?

9.2.3 Community ain’t the official release?

Below are the complete error details:

org.sonar.server.es.ElasticsearchException: Fail to execute es requestES refresh request on indices 'components'
	at org.sonar.server.es.EsClient.execute(EsClient.java:313)
	at org.sonar.server.es.EsClient.refresh(EsClient.java:155)
	at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:128)
	at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:170)
	at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
	at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:107)
	at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:70)
	at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
	at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:54)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:904)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:276)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.IndicesClient.refresh(IndicesClient.java:900)
	at org.sonar.server.es.EsClient.lambda$refresh$9(EsClient.java:155)
	at org.sonar.server.es.EsClient.execute(EsClient.java:311)
	... 27 more
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
	at org.apache.http.util.Asserts.check(Asserts.java:46)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:284)
	... 35 more

Here is the dockerhub image: should be official one.

Hi @prasad-clouduser,

You seem to be getting a different error than the OP. Please create a new thread and provide all relevant details.

 
Ann

@ganncamp we are same issues on community edition 8.9.6 (Request cannot be executed; I/O reactor status: STOPPED). All our background tasks are failing due to I/O error. can you please guide us how to upgrade (hotpatch) elastick search version to 7.16.3. I don’t see proper instruction.

Hello @navaneeth.ramakrishn,

No, I cannot tell you how to hotpatch SonarQube. I can only tell you to upgrade to the latest point version, which already has all the patches you need.

 
HTH,
Ann

@ganncamp thanks for your response. we always used to upgrade LTS and Noticed on 8.9.7/8.9.8 LTS don’t have Elasticsearch 7.16.3 libs version. Attached screen shot. Is that something you recommend to upgrade 9.3 or above versions?

Hi,

This topic has been treated exhaustively here:

 
Ann