Hi,
I’m currently running SonarQube 8.1.0.31237 (just updated from 7.8 to 7.9 TLS to 8.1) and I’m running into a strange problem. I can’t seem to open an issues page on ANY project I have currently. The dashboard still provides me with numbers (3 bugs, 11 code smells, …) but when I click on the number and navigate to https://xxxxxxxxxx.xxx/project/issues?branch=story%2F173&id=&resolved=false&types=BUG I get a “An error has occurred. Please contact your administrator” message.
I started investigating the logs files and can only find something interesting in the web.log file:
2020.01.08 15:54:50 ERROR web[AW+F1OIQTMvvNS/xAACP][o.s.s.w.WebServiceEngine] Fail to process request xxxxxxxxx.xxx is available for purchase - Sedo.com&s=FILE_LINE&resolved=false&ps=100&organization=default-organization&facets=severities%2Ctypes&additionalFields=_all
java.lang.IllegalStateException: Fail to execute ES search request ‘SearchRequest{searchType=QUERY_THEN_FETCH, indices=[issues], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[auth], routing=‘null’, preference=‘null’, requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={“from”:0,“size”:100,“query”:{“bool”:{“must”:[{“match_all”:{“boost”:1.0}}],“filter”:[{“bool”:{“must”:[{“term”:{“isMainBranch”:{“value”:“false”,“boost”:1.0}}},{“bool”:{“must_not”:[{“exists”:{“field”:“resolution”,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},{“terms”:{“project”:[“AW-FgzicuAz_Iuw3piPC”],“boost”:1.0}},{“term”:{“branch”:{“value”:“AW-FqcsmuAz_Iuw3piPJ”,“boost”:1.0}}},{“term”:{“indexType”:{“value”:“issue”,“boost”:1.0}}},{“bool”:{“adjust_pure_negative”:true,“boost”:1.0}},{“term”:{“organization”:{“value”:“AW3PvsaoqyfACwE80caQ”,“boost”:1.0}}},{“has_parent”:{“query”:{“bool”:{“filter”:[{“bool”:{“should”:[{“term”:{“auth_allowAnyone”:{“value”:true,“boost”:1.0}}},{“term”:{“auth_userIds”:{“value”:1,“boost”:1.0}}},{“term”:{“auth_groupIds”:{“value”:1,“boost”:1.0}}},{“term”:{“auth_groupIds”:{“value”:2,“boost”:1.0}}}],“adjust_pure_negative”:true,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},“parent_type”:“auth”,“score”:false,“ignore_unmapped”:false,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},“_source”:false,“sort”:[{“project”:{“order”:“asc”,“missing”:“_first”}},{“filePath”:{“order”:“asc”,“missing”:“_first”}},{“line”:{“order”:“asc”,“missing”:“_first”}},{“severityValue”:{“order”:“desc”,“missing”:“_first”}},{“key”:{“order”:“asc”,“missing”:“_first”}}],“aggregations”:{“types”:{“global”:{},“aggregations”:{“types_filter”:{“filter”:{“bool”:{“must”:[{“match_all”:{“boost”:1.0}},{“term”:{“isMainBranch”:{“value”:“false”,“boost”:1.0}}},{“bool”:{“must_not”:[{“exists”:{“field”:“resolution”,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},{“terms”:{“project”:[“AW-FgzicuAz_Iuw3piPC”],“boost”:1.0}},{“term”:{“branch”:{“value”:“AW-FqcsmuAz_Iuw3piPJ”,“boost”:1.0}}},{“term”:{“indexType”:{“value”:“issue”,“boost”:1.0}}},{“bool”:{“adjust_pure_negative”:true,“boost”:1.0}},{“term”:{“organization”:{“value”:“AW3PvsaoqyfACwE80caQ”,“boost”:1.0}}},{“has_parent”:{“query”:{“bool”:{“filter”:[{“bool”:{“should”:[{“term”:{“auth_allowAnyone”:{“value”:true,“boost”:1.0}}},{“term”:{“auth_userIds”:{“value”:1,“boost”:1.0}}},{“term”:{“auth_groupIds”:{“value”:1,“boost”:1.0}}},{“term”:{“auth_groupIds”:{“value”:2,“boost”:1.0}}}],“adjust_pure_negative”:true,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},“parent_type”:“auth”,“score”:false,“ignore_unmapped”:false,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},“aggregations”:{“types”:{“terms”:{“field”:“type”,“size”:4,“min_doc_count”:1,“shard_min_doc_count”:0,“show_term_doc_count_error”:false,“order”:[{“_count”:“desc”},{“_key”:“asc”}]}}}}}},“severities”:{“global”:{},“aggregations”:{“severities_filter”:{“filter”:{“bool”:{“must”:[{“match_all”:{“boost”:1.0}},{“term”:{“isMainBranch”:{“value”:“false”,“boost”:1.0}}},{“bool”:{“must_not”:[{“exists”:{“field”:“resolution”,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},{“terms”:{“project”:[“AW-FgzicuAz_Iuw3piPC”],“boost”:1.0}},{“term”:{“branch”:{“value”:“AW-FqcsmuAz_Iuw3piPJ”,“boost”:1.0}}},{“term”:{“indexType”:{“value”:“issue”,“boost”:1.0}}},{“bool”:{“adjust_pure_negative”:true,“boost”:1.0}},{“term”:{“organization”:{“value”:“AW3PvsaoqyfACwE80caQ”,“boost”:1.0}}},{“has_parent”:{“query”:{“bool”:{“filter”:[{“bool”:{“should”:[{“term”:{“auth_allowAnyone”:{“value”:true,“boost”:1.0}}},{“term”:{“auth_userIds”:{“value”:1,“boost”:1.0}}},{“term”:{“auth_groupIds”:{“value”:1,“boost”:1.0}}},{“term”:{“auth_groupIds”:{“value”:2,“boost”:1.0}}}],“adjust_pure_negative”:true,“boost”:1.0}}],“adjust_pure_negative”:true,“boost”:1.0}},“parent_type”:“auth”,“score”:false,“ignore_unmapped”:false,“boost”:1.0}}],“must_not”:[{“term”:{“type”:{“value”:“SECURITY_HOTSPOT”,“boost”:1.0}}}],“adjust_pure_negative”:true,“boost”:1.0}},“aggregations”:{“severities”:{“terms”:{“field”:“severity”,“size”:5,“min_doc_count”:1,“shard_min_doc_count”:0,“show_term_doc_count_error”:false,“order”:[{“_count”:“desc”},{“_key”:“asc”}]}}}}}},“effort”:{“sum”:{“field”:“effort”}}}}}’ on indices ‘[issues]’ on types ‘[auth]’
at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:44)
at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:32)
at org.sonar.server.issue.index.IssueIndex.search(IssueIndex.java:337)
at org.sonar.server.issue.ws.SearchAction.doHandle(SearchAction.java:410)
at org.sonar.server.issue.ws.SearchAction.handle(SearchAction.java:391)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:110)
at org.sonar.server.platform.web.WebServiceFilter.doFilter(WebServiceFilter.java:88)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:139)
at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:88)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:72)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:76)
at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RequestIdFilter.doFilter(RequestIdFilter.java:66)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296)
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133)
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259)
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100)
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48)
at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:220)
at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:174)
at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48)
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:220)
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:463)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1114)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1226)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1200)
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60)
at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56)
at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:367)
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:361)
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:355)
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1107)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
Caused by: org.elasticsearch.index.query.QueryShardException: [has_parent] no join field has been configured
at org.elasticsearch.join.query.HasParentQueryBuilder.joinFieldDoToQuery(HasParentQueryBuilder.java:189)
at org.elasticsearch.join.query.HasParentQueryBuilder.doToQuery(HasParentQueryBuilder.java:177)
at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:105)
at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:415)
at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:383)
at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:105)
at org.elasticsearch.index.query.AbstractQueryBuilder.toFilter(AbstractQueryBuilder.java:127)
at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:419)
at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:386)
at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:105)
at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$2(QueryShardContext.java:319)
at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:331)
at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:318)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:791)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:637)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:596)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:387)
at org.elasticsearch.search.SearchService.access$100(SearchService.java:126)
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:359)
… 9 common frames omitted
Seems like ES has some problems with the data. I already tried to delete and create a new project but that doesn’t solve the issue. I also checked and I still have more then 80Gb of free disk space so that can’t be the problem either.
Any idea what may cause this problem?
If needed, I can supply more information or other logs files.
EDIT:
Forgot to mention that I can access the measures page. But when I drill down for example into bugs and then open a file it’s just empty page and I see the same exceptions in the web.log file.
Best regards,
mregni