Elastic search indexing error

SonarQube Server / Community Build
1

Version 10.0 (build 68432)

Hi

We’re getting this exception on background analysis

java.lang.IllegalStateException: Unrecoverable indexation failures: 11941 errors among 11941 requests. Check Elasticsearch logs for further details.
	at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:42)
	at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:131)
	at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:169)
	at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
	at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:105)
	at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:68)
	at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
	at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:54)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

I reviewed the elastic es log file however there are no errors or warnings just the standard informational events. Reviewing the computational engine ce log we see

2023.04.19 05:43:09 ERROR ce[][o.s.s.es.BulkIndexer] index [components], type [_doc], id [AYdORETKN6CjUwWJ0PHI], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [components] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
2023.04.19 05:43:09 ERROR ce[][o.s.s.es.BulkIndexer] index [components], type [_doc], id [AYdORETUN6CjUwWJ0Phc], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [components] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
2023.04.19 05:43:09 INFO  ce[AYeY4gvePfaKdl7TpRfC][o.s.c.t.s.ComputationStepExecutor] Purge db | status=FAILED | time=5868ms

Appreciate your help

2

Hi,

This is the problem:

Elasticsearch requires a minimum 30 percent free disk space, and without that, it locks its indices. Unfortunately, that means that even if you have 30Gb free, if that’s less than 30% of the disk size, ES locks the indices.

So free up some disk space and restart (which will unlock the indices).

 
HTH,
Ann