Greetings!
On SonarQube 7.7 Developer Editon C++ build wrapper version 6.3 (build-wrapper-linux-x86-64):
I get every now and then (after max. 10 runs) the following exception when running the scanner:
java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)…
After that every consecutive scanner run fails.
I saw that also others have reported this issue here, but my understanding is that they got rid of it after deleting the ES data once, like this:
/opt/sonarqube/bin/linux-x86-64/sonar.sh stop
rm -rf /opt/sonarqube/data/es*
/opt/sonarqube/bin/linux-x86-64/sonar.sh start
It also works for me temporarily, but the problem is coming back after a couple of runs as described above.
Any hint what to check in ElasticSearch config or elsewhere?
Thanks, Pierre, good to know, but unfortunately it’s not the root cause on my machine:
I have currently the error and not yet deleted the ES data and disk space is at 74%.
KR
Matthias
Sorry for the delay, we had some server problem.
But indeed, your hint is very valuable - in the es.log there is low disk mentioned:
“INFO es[o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded … free: 43.8gb[14.9%], replicas will not be assigned to this node”
But I do not really understand how this can be, since free on that disk partition is more than 75gb (i.e. only 75% used)!
And secondly you mentioned the critical border for ES is 95%, but in the log it’s 85% - maybe ES is configured to a lower threshold on my machine? Do you know where this is set?
2019.07.19 00:05:34 ERROR web[o.s.s.es.BulkIndexer] index [projectmeasures], type [auth], id [AWtVBF5hrsX19e1yZHce], message [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
And es.log show that you hit the low watermark :
low disk watermark [85%] exceeded on [UeRVjxMESfesnoT-4wrPZw][sonarqube][/opt/sonarqube/data/es6/nodes/0] free: 43.1gb[14.7%], replicas will not be assigned to this node
85% is the low watermark meaning that new shards cannot be created on this node, but you can still write on existing ones. The 95% high watermark lock all writings. In your case, a new shard is created, so you are stuck at the low watermark.