ES indexation failure again and again

Greetings!
On SonarQube 7.7 Developer Editon C++ build wrapper version 6.3 (build-wrapper-linux-x86-64):
I get every now and then (after max. 10 runs) the following exception when running the scanner:

java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)…

After that every consecutive scanner run fails.
I saw that also others have reported this issue here, but my understanding is that they got rid of it after deleting the ES data once, like this:

:black_medium_small_square:/opt/sonarqube/bin/linux-x86-64/sonar.sh stop

:black_medium_small_square:rm -rf /opt/sonarqube/data/es*

:black_medium_small_square:/opt/sonarqube/bin/linux-x86-64/sonar.sh start

It also works for me temporarily, but the problem is coming back after a couple of runs as described above.
Any hint what to check in ElasticSearch config or elsewhere?

best regards
Matthias

Hello ! did you check the remaining disk space left on your device ? Elasticsearch indexes switch to read-only when you reach 95%

Thanks, Pierre, good to know, but unfortunately it’s not the root cause on my machine:
I have currently the error and not yet deleted the ES data and disk space is at 74%.
KR
Matthias

Can you provide more logs please ? sonar.log, es.log, web.log

Sorry for the delay, we had some server problem.
But indeed, your hint is very valuable - in the es.log there is low disk mentioned:

“INFO es[o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded … free: 43.8gb[14.9%], replicas will not be assigned to this node”

But I do not really understand how this can be, since free on that disk partition is more than 75gb (i.e. only 75% used)!
And secondly you mentioned the critical border for ES is 95%, but in the log it’s 85% - maybe ES is configured to a lower threshold on my machine? Do you know where this is set?

KR
Matthias

Pierre, pls. find attached the requested logs:
ce.log.txt (15.2 KB) web.log.txt (273.0 KB) es.2019-07-20.log.txt (4.6 KB) es.2019-07-21.log.txt (875 Bytes)

Your web.log show that indexes are read-only :

2019.07.19 00:05:34 ERROR web[o.s.s.es.BulkIndexer] index [projectmeasures], type [auth], id [AWtVBF5hrsX19e1yZHce], message [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]

And es.log show that you hit the low watermark :

low disk watermark [85%] exceeded on [UeRVjxMESfesnoT-4wrPZw][sonarqube][/opt/sonarqube/data/es6/nodes/0] free: 43.1gb[14.7%], replicas will not be assigned to this node

Free some space an restart SQ.

About the threshold, you can find more information here : https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

85% is the low watermark meaning that new shards cannot be created on this node, but you can still write on existing ones. The 95% high watermark lock all writings. In your case, a new shard is created, so you are stuck at the low watermark.

Hope this help :slight_smile:

Thanks, Pierre.
I still do not see were those 10gb are missing, but at least it’s clear what to do to fix the issue.
KR
Matthias