None of the configured nodes are available

SonarQube 6.7.3.38370 Community Edition
org.sonarsource.scanner.maven:sonar-maven-plugin:3.0.2:sonar

In Jenkins this error occures since last weekend from time to time (not reproducable):

**13:49:56** [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.0.2:sonar (default-cli) on project Build_Product.4.0: Unable to load component class org.sonar.scanner.report.ActiveRulesPublisher: Unable to load component interface org.sonar.api.batch.rule.ActiveRules: Error 500 on http://s-hqdev5:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=jsp-findbugs-security-jsp-83706&p=1&ps=500 : {"errors":[{"msg":"An error has occurred. Please contact your administrator"}]} -> [Help 1]

the web.log contained the following error:

2018.08.15 13:49:56 ERROR web[AWU8YL29jf7yhBzVABnE][o.s.s.w.WebServiceEngine] Fail to process request http://s-hqdev5:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=jsp-findbugs-security-jsp-83706&p=1&ps=500
java.lang.IllegalStateException: Fail to execute ES search request 'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[rules], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[rule], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, source={
  "from" : 0,
  "size" : 500,
  "query" : {
    "bool" : {
      "must" : [
        {
          "match_all" : {
            "boost" : 1.0
          }
        }
      ],
      "filter" : [
        {
          "bool" : {
            "must" : [
              {
                "terms" : {
                  "lang" : [
                    "jsp"
                  ],
                  "boost" : 1.0
                }
              },
              {
                "has_child" : {
                  "query" : {
                    "bool" : {
                      "must" : [
                        {
                          "term" : {
                            "ruleProfile" : {
                              "value" : "jsp-findbugs-security-jsp-83706",
                              "boost" : 1.0
                            }
                          }
                        }
                      ],
                      "disable_coord" : false,
                      "adjust_pure_negative" : true,
                      "boost" : 1.0
                    }
                  },
                  "type" : "activeRule",
                  "score_mode" : "none",
                  "min_children" : 0,
                  "max_children" : 2147483647,
                  "ignore_unmapped" : false,
                  "boost" : 1.0
                }
              },
              {
                "bool" : {
                  "must_not" : [
                    {
                      "term" : {
                        "status" : {
                          "value" : "REMOVED",
                          "boost" : 1.0
                        }
                      }
                    }
                  ],
                  "disable_coord" : false,
                  "adjust_pure_negative" : true,
                  "boost" : 1.0
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "sort" : [
    {
      "updatedAt" : {
        "order" : "desc"
      }
    },
    {
      "key.sortable_analyzer" : {
        "order" : "asc"
      }
    }
  ]
}}' on indices '[rules]' on types '[rule]'
	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:44)
	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:32)
	at org.sonar.server.rule.index.RuleIndex.search(RuleIndex.java:164)
	at org.sonar.server.rule.ws.SearchAction.doSearch(SearchAction.java:351)
	at org.sonar.server.rule.ws.SearchAction.handle(SearchAction.java:162)
	at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:103)
	at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:86)
	at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:126)
	at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:95)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:87)
	at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:71)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:72)
	at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:48)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:61)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.sonar.server.platform.web.requestid.RequestIdFilter.doFilter(RequestIdFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{REK0FSfuQ2K_mep9_-xxaw}{127.0.0.1}{127.0.0.1:9001}]
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:42)
	... 46 common frames omitted

steps to reproduce:

  • cannot be reproduced, happens 5-10% of all builds

tried workaround:

I’ve tried to delete the data/es5 directory (shut the server down, deleted the directory, started the server again), but the error occurs afterwards too. The compute engine crashed with heap errors, so I have increased the xmx from 0.5g to 1g, but the error occurs afterwards too.

Hi,

I’ve often seen this relating to problems with computing resources. Whether it’s CPU/RAM/IO , it’s important in such case do some monitoring and understand the precise state of resources when hitting the error (there’s some introductory documentation about that here). I understand you’ve increased Xmx already, but maybe more is needed, and maybe more is needed on other processes too. Only close monitoring of the JVMs will guide you in understanding what’s going on, and whether config/provisioning should be fine-tuned accordingly.