Hello, we have just upgraded from 7.9 to 8.9.9.56886.
At the end of the setup we have 17 projects in FAILED status out of 122.
What is the origin of the problem and how to solve it, please?
Error Details: II(ASSURETUDES / INDEX) [Project Data Reload]
Error Details
java.lang.IllegalStateException: Unrecoverable indexation failures: 13685 errors among 320169 requests
at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:131)
at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:261)
at org.sonar.server.issue.index.IssueIndexer.indexOnAnalysis(IssueIndexer.java:115)
at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.lambda$execute$1(IndexIssuesStep.java:52)
at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep$$Lambda$534/0x0000000000000000.accept(Unknown Source)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.execute(IndexIssuesStep.java:48)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
at org.sonar.ce.task.projectanalysis.taskprocessor.IssueSyncTaskProcessor.process(IssueSyncTaskProcessor.java:59)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$$Lambda$1225/0x0000000000000000.apply(Unknown Source)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:866)
2022.09.22 12:01:15 INFO ce[][o.s.c.t.CeWorkerImpl] Execute task | project=II | type=ISSUE_SYNC | branch=master | branchType=BRANCH | id=AYNkeefg4ZoFK5fV2uBz
2022.09.22 12:01:15 INFO ce[AYNkeefg4ZoFK5fV2uBz][o.s.c.t.s.ComputationStepExecutor] Ignore orphan component | status=SUCCESS | time=0ms
2022.09.22 12:01:15 INFO ce[AYNkeefg4ZoFK5fV2uBz][o.s.c.t.p.t.IndexIssuesStep] indexing issues of branch AXnLdnDjMrf5C34ese-U
2022.09.22 12:01:22 ERROR ce[][o.s.s.es.BulkIndexer] Fail to execute bulk index request: org.elasticsearch.action.bulk.BulkRequest/unset
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [518977000/494.9mb], which is larger than the limit of [510027366/486.3mb], real usage: [516721752/492.7mb], new bytes reserved: [2255248/2.1mb], usages [request=0/0b, fielddata=832454/812.9kb, in_flight_requests=2255248/2.1mb, accounting=1017392/993.5kb]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178)
at org.elasticsearch.client.RestHighLevelClient$$Lambda$752/0x0000000000000000.apply(Unknown Source)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461)
at org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:2372)
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:657)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:393)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:866)
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:9001], URI [/_bulk?timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [518977000/494.9mb], which is larger than the limit of [510027366/486.3mb], real usage: [516721752/492.7mb], new bytes reserved: [2255248/2.1mb], usages [request=0/0b, fielddata=832454/812.9kb, in_flight_requests=2255248/2.1mb, accounting=1017392/993.5kb]","bytes_wanted":518977000,"bytes_limit":510027366,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [518977000/494.9mb], which is larger than the limit of [510027366/486.3mb], real usage: [516721752/492.7mb], new bytes reserved: [2255248/2.1mb], usages [request=0/0b, fielddata=832454/812.9kb, in_flight_requests=2255248/2.1mb, accounting=1017392/993.5kb]","bytes_wanted":518977000,"bytes_limit":510027366,"durability":"TRANSIENT"},"status":429}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:331)
at org.elasticsearch.client.RestClient.access$1800(RestClient.java:106)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381)
... 16 common frames omitted
Elasticsearch is doing its best here to prevent an Out Of Memory exception – adjusting sonar.es.javaOpts to a larger value (-Xmx2g -Xms2g) would prevent this error in the future.
I recommend you:
Adjust this setting in your conf/sonar.properties file
Stop your SonarQube instance
Delete the data/es7 folder of your SonarQube installation
Thank Colin for your answer.
The options of “sonar.es.javaOpts” were already at (Xmx10g Xms512m).
I tried with (Xmx10g Xms10g) and (Xmx12g Xms12g), but with no more success.
Hello, after a “VAACUM FULL”, I tested the loading with several values ​​of Xmx, higher and higher.
I always have FAILED projects, not always the same number, not always the same ones.
