Sonarqube v8.9 background task failure

Hi Team,

We are frequently facing the analysis failure issue, when we restart and delete es7 folder it is working fine for some time and all the jobs failing with the below error.

org.sonar.ce.task.projectanalysis.component.VisitException: Visit of Component {key=com.**.csu_was_inter:csusvc:source/java/csuRest/src/com/hmhs/csu/rest/cs/register/resource/TemparoryRegisterResource.java,type=FILE} failed
	at org.sonar.ce.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:74)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
	at org.sonar.ce.task.projectanalysis.step.ExecuteVisitorsStep.execute(ExecuteVisitorsStep.java:51)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Fail to process issues of component 'com.**.csu_was_inter:csusvc:source/java/csuRest/src/com/hmhs/csu/rest/cs/register/resource/TemparoryRegisterResource.java'
	at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:79)
	at org.sonar.ce.task.projectanalysis.component.TypeAwareVisitorWrapper.visitAny(TypeAwareVisitorWrapper.java:77)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitNode(VisitorsCrawler.java:117)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:100)
	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
	... 35 more
Caused by: org.sonar.server.es.ElasticsearchException: Fail to execute es requestES search request 'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[users], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[user], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":3,"query":{"bool":{"must":[{"match_all":{"boost":1.0}}],"filter":[{"bool":{"must":[{"term":{"active":{"value":true,"boost":1.0}}}],"should":[{"term":{"login":{"value":"guruswamy.bobburi@thryvedigital.com","boost":1.0}}},{"match":{"email.sortable_analyzer":{"query":"guruswamy.bobburi@thryvedigital.com","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"scmAccounts.sortable_analyzer":{"query":"guruswamy.bobburi@thryvedigital.com","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}' on indices '[users]' on types '[user]'
	at org.sonar.server.es.EsClient.execute(EsClient.java:313)
	at org.sonar.server.es.EsClient.search(EsClient.java:135)
	at org.sonar.server.user.index.UserIndex.getAtMostThreeActiveUsersForScmAccount(UserIndex.java:73)
	at org.sonar.ce.task.projectanalysis.issue.ScmAccountToUserLoader.load(ScmAccountToUserLoader.java:49)
	at org.sonar.ce.task.projectanalysis.issue.ScmAccountToUserLoader.load(ScmAccountToUserLoader.java:37)
	at org.sonar.ce.task.projectanalysis.util.cache.MemoryCache.getNullable(MemoryCache.java:46)
	at java.base/java.util.Optional.map(Optional.java:265)
	at org.sonar.ce.task.projectanalysis.issue.IssueAssigner.onIssue(IssueAssigner.java:85)
	at org.sonar.ce.task.projectanalysis.issue.IssueVisitors.onIssue(IssueVisitors.java:41)
	at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.process(IntegrateIssuesVisitor.java:129)
	at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.fillNewOpenIssues(IntegrateIssuesVisitor.java:96)
	at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:73)
	... 39 more
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:888)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1110)
	at org.sonar.server.es.EsClient.lambda$search$1(EsClient.java:135)
	at org.sonar.server.es.EsClient.execute(EsClient.java:311)
	... 50 more
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
	at org.apache.http.util.Asserts.check(Asserts.java:46)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:279)
	... 57 more
-----------------------

Hi,

Is there anything in es.log?

 
Ann

es_log.txt (4.8 KB)
Es log attached.

Hi,

This log starts with a shutdown. Was that shutdown automatic because of a failure or initiated by an admin?

Can we have a log that covers the time period of the problem?

 
Ann

Hi Ann,

I’m from admin team, for the workaround we are restarting Sonarqube frequently. This problem is started happening from 26/Sep/2022, but we have logs only from 30/sep/2022. all other logs deleted due to cleanup in the server. Is there any way to capture that log? Many error saying java heap space related issue.
Ce.log below

2022.10.06 10:45:31 ERROR ce[AYOtwfRY7HIUhpsyz7tM][o.s.c.t.CeWorkerImpl] Failed to execute task AYOtwfRY7HIUhpsyz7tM
org.sonar.ce.task.projectanalysis.component.VisitException: Visit of Component {key=com.*.srs_was_intra:srsweb:source/java/SRSWEB/src/com/*/srs/dao/HierTrnDAO.java,type=FILE} failed
        at org.sonar.ce.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:74)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        at org.sonar.ce.task.projectanalysis.step.ExecuteVisitorsStep.execute(ExecuteVisitorsStep.java:51)
        at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
        at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
        at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
        at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Fail to process issues of component 'com.*.srs_was_intra:srsweb:source/java/SRSWEB/src/com/*/srs/dao/HierTrnDAO.java'
        at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:79)
        at org.sonar.ce.task.projectanalysis.component.TypeAwareVisitorWrapper.visitAny(TypeAwareVisitorWrapper.java:77)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitNode(VisitorsCrawler.java:117)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:100)
        at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
        ... 41 common frames omitted
Caused by: org.sonar.server.es.ElasticsearchException: Fail to execute es requestES search request 'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[users], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[user], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":3,"query":{"bool":{"must":[{"match_all":{"boost":1.0}}],"filter":[{"bool":{"must":[{"term":{"active":{"value":true,"boost":1.0}}}],"should":[{"term":{"login":{"value":"travis.myers@hmhs.com","boost":1.0}}},{"match":{"email.sortable_analyzer":{"query":"travis.myers@hmhs.com","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"scmAccounts.sortable_analyzer":{"query":"travis.myers@hmhs.com","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}' on indices '[users]' on types '[user]'
        at org.sonar.server.es.EsClient.execute(EsClient.java:313)
        at org.sonar.server.es.EsClient.search(EsClient.java:135)
        at org.sonar.server.user.index.UserIndex.getAtMostThreeActiveUsersForScmAccount(UserIndex.java:73)
        at org.sonar.ce.task.projectanalysis.issue.ScmAccountToUserLoader.load(ScmAccountToUserLoader.java:49)
        at org.sonar.ce.task.projectanalysis.issue.ScmAccountToUserLoader.load(ScmAccountToUserLoader.java:37)
        at org.sonar.ce.task.projectanalysis.util.cache.MemoryCache.getNullable(MemoryCache.java:46)
        at java.base/java.util.Optional.map(Optional.java:265)
        at org.sonar.ce.task.projectanalysis.issue.IssueAssigner.onIssue(IssueAssigner.java:85)
        at org.sonar.ce.task.projectanalysis.issue.IssueVisitors.onIssue(IssueVisitors.java:41)
        at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.process(IntegrateIssuesVisitor.java:129)
        at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.fillNewOpenIssues(IntegrateIssuesVisitor.java:96)
        at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:73)
        ... 45 common frames omitted
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:888)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1110)
        at org.sonar.server.es.EsClient.lambda$search$1(EsClient.java:135)
        at org.sonar.server.es.EsClient.execute(EsClient.java:311)
        ... 56 common frames omitted
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
        at org.apache.http.util.Asserts.check(Asserts.java:46)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:279)
        ... 63 common frames omitted

Hi,

If you’re getting heap space errors then you should adjust the javaOpts for that process. Look in $SONARQUBE-HOME/conf/sonar.properties for: sonar.ce.javaOpts and sonar.search.javaOpts. Also of possible interest: sonar.web.javaOpts.

 
HTH,
Ann

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.