[SonarQube 8.9] Some projects status FAILED after installing the new version

Hello, we have just upgraded from 7.9 to 8.9.9.56886.
At the end of the setup we have 17 projects in FAILED status out of 122.

What is the origin of the problem and how to solve it, please?

Error Details: II(ASSURETUDES / INDEX) [Project Data Reload]
Error Details
java.lang.IllegalStateException: Unrecoverable indexation failures: 13685 errors among 320169 requests
	at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
	at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:131)
	at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:261)
	at org.sonar.server.issue.index.IssueIndexer.indexOnAnalysis(IssueIndexer.java:115)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.lambda$execute$1(IndexIssuesStep.java:52)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep$$Lambda$534/0x0000000000000000.accept(Unknown Source)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IndexIssuesStep.execute(IndexIssuesStep.java:48)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
	at org.sonar.ce.task.projectanalysis.taskprocessor.IssueSyncTaskProcessor.process(IssueSyncTaskProcessor.java:59)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$$Lambda$1225/0x0000000000000000.apply(Unknown Source)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:866)

In the ce.log we have these errors :

2022.09.22 12:01:15 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=II | type=ISSUE_SYNC | branch=master | branchType=BRANCH | id=AYNkeefg4ZoFK5fV2uBz
2022.09.22 12:01:15 INFO  ce[AYNkeefg4ZoFK5fV2uBz][o.s.c.t.s.ComputationStepExecutor] Ignore orphan component | status=SUCCESS | time=0ms
2022.09.22 12:01:15 INFO  ce[AYNkeefg4ZoFK5fV2uBz][o.s.c.t.p.t.IndexIssuesStep] indexing issues of branch AXnLdnDjMrf5C34ese-U
2022.09.22 12:01:22 ERROR ce[][o.s.s.es.BulkIndexer] Fail to execute bulk index request: org.elasticsearch.action.bulk.BulkRequest/unset
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [518977000/494.9mb], which is larger than the limit of [510027366/486.3mb], real usage: [516721752/492.7mb], new bytes reserved: [2255248/2.1mb], usages [request=0/0b, fielddata=832454/812.9kb, in_flight_requests=2255248/2.1mb, accounting=1017392/993.5kb]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178)
	at org.elasticsearch.client.RestHighLevelClient$$Lambda$752/0x0000000000000000.apply(Unknown Source)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461)
	at org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:2372)
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:657)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:393)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:866)
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:9001], URI [/_bulk?timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [518977000/494.9mb], which is larger than the limit of [510027366/486.3mb], real usage: [516721752/492.7mb], new bytes reserved: [2255248/2.1mb], usages [request=0/0b, fielddata=832454/812.9kb, in_flight_requests=2255248/2.1mb, accounting=1017392/993.5kb]","bytes_wanted":518977000,"bytes_limit":510027366,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [518977000/494.9mb], which is larger than the limit of [510027366/486.3mb], real usage: [516721752/492.7mb], new bytes reserved: [2255248/2.1mb], usages [request=0/0b, fielddata=832454/812.9kb, in_flight_requests=2255248/2.1mb, accounting=1017392/993.5kb]","bytes_wanted":518977000,"bytes_limit":510027366,"durability":"TRANSIENT"},"status":429}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:331)
		at org.elasticsearch.client.RestClient.access$1800(RestClient.java:106)
		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:381)
		... 16 common frames omitted

Hey there.

Elasticsearch is doing its best here to prevent an Out Of Memory exception – adjusting sonar.es.javaOpts to a larger value (-Xmx2g -Xms2g) would prevent this error in the future.

I recommend you:

  1. Adjust this setting in your conf/sonar.properties file
  2. Stop your SonarQube instance
  3. Delete the data/es7 folder of your SonarQube installation
  4. Start your SonarQube instance
1 Like

Thank Colin for your answer.
The options of “sonar.es.javaOpts” were already at (Xmx10g Xms512m).
I tried with (Xmx10g Xms10g) and (Xmx12g Xms12g), but with no more success.

I’m suspicious that those heap sizes are actually being taken into account. Are they reflected in your global Administration > System?

1 Like

Not sure to understand.
We have 24 GB, on a windows serveur 2016

and openJDK version 11

We have not yet carried out the “VACUUM FULL” on our postgreSQL database.

Throughout the reloading of projects, we are at a maximum of 51% memory usage.

Hello, after a “VAACUM FULL”, I tested the loading with several values ​​of Xmx, higher and higher.
I always have FAILED projects, not always the same number, not always the same ones.

I’m referring to this section of the global Administration. Are the values you’re setting for sonar.search.javaOpts reflecting here?

ok, i think i am using the default configuration for “sonar.search.javaOpts”, the problem seems to be on the Compute Engine.

sonar.web.javaOpts=-Xmx10g -Xms128m -XX:+HeapDumpOnOutOfMemoryError
sonar.ce.javaOpts=-Xmx16g -Xms16g -XX:+HeapDumpOnOutOfMemoryError
#sonar.search.javaOpts=-Xmx512m -Xms512m -XX:MaxDirectMemorySize=256m -XX:+HeapDumpOnOutOfMemoryError

image
image

All is OK with this :

Thanks.

1 Like