Initial setup integration failing

SonarQube Server / Community Build
12

Dear all,

We are running the official SonarQube 7.9.1 from the Docker Hub in production and also had the same error from Jenkins. Removing whole embedded ES database is actually a very bad idea. So I digged further.

[2020-01-27T18:41:21.991Z] INFO: Detected Jenkins
[2020-01-27T18:41:21.991Z] INFO: Load active rules
[2020-01-27T18:41:26.113Z] INFO: ------------------------------------------------------------------------
[2020-01-27T18:41:26.113Z] INFO: EXECUTION FAILURE
[2020-01-27T18:41:26.113Z] INFO: ------------------------------------------------------------------------
[2020-01-27T18:41:26.113Z] INFO: Total time: 12.599s
[2020-01-27T18:41:26.113Z] INFO: Final Memory: 11M/114M
[2020-01-27T18:41:26.113Z] INFO: ------------------------------------------------------------------------
[2020-01-27T18:41:26.113Z] ERROR: Error during SonarQube Scanner execution
[2020-01-27T18:41:26.113Z] ERROR: Unable to load component class org.sonar.scanner.report.ActiveRulesPublisher
[2020-01-27T18:41:26.113Z] ERROR: Caused by: Unable to load component interface org.sonar.api.batch.rule.ActiveRules
[2020-01-27T18:41:26.113Z] ERROR: 
[2020-01-27T18:41:26.113Z] ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.

So what I did

Manual analysed the logfiles under /opt/sonarqube/logs;

  • SonarQube web* logfiles contained suspicous entries about the index forbidden read-only: web.2020-01-27.log:2020.01.27 23:58:28 ERROR web[][o.s.s.es.BulkIndexer] index [rules], type [rule], id [ar_3875], message [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
  • Elastic search es* logfiles contained suspicous entries: es.2020-01-27.log:2020.01.27 22:10:15 INFO es[][o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded on [lUZ2LV4YSPO-dui6kr83Xg][sonarqube][/opt/sonarqube/data/es6/nodes/0] free: 7.6gb[12.2%], replicas will not be assigned to this node
  • Docker host disk usage was checked; df -h and was indeed 88% full
  • Docker prune was performed to free some disk space; docker system prune
  • SonarQube with Embedded Elasticsearch Container was manual stopped; docker stop 05be7e73592f
  • SonarQube was manual started; docker start 05be7e73592f

Hope this helps anybody in the future, it seems that ES graceful disallows API call writes to the indexer and puts it into read-only mode when the low disk watermark is reached as expected. Only this error propagating so far into the CI/CD pipeline is very weird and I hate Java for its unreadable stacktraces which are not very helpfull at all. That said here is the watermark doc from ES https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html