SonarQube LTS 8.9.0
(we have since upgraded 8.9.6, during which we identified the previous failures)
When reviewing the SonarQube background Tasks, identified a series of project analysis failures with the same error condition. Over 900 scans were done that day and somewhere in the middle, these failed.
| UUID | STATUS | TIMESTAMP | DATE |
|---|---|---|---|
| AX4TYckDOP_YqP5z1O15 | SUCCESS | 1641002682650 | 2022-01-01 2:04:43 |
| AX4TYdXMOP_YqP5z1O16 | FAILED | 1641002687816 | 2022-01-01 2:04:48 |
| AX4TYleCOP_YqP5z1O17 | FAILED | 1641002719379 | 2022-01-01 2:05:19 |
| AX4TYxymOP_YqP5z1O18 | FAILED | 1641002769636 | 2022-01-01 2:06:10 |
| AX4TY1glOP_YqP5z1O19 | FAILED | 1641002784840 | 2022-01-01 2:06:25 |
| AX4TY2DoOP_YqP5z1O1- | FAILED | 1641002787115 | 2022-01-01 2:06:27 |
| AX4TY8B1OP_YqP5z1O1_ | FAILED | 1641002811529 | 2022-01-01 2:06:52 |
| AX4TY86MOP_YqP5z1O2A | FAILED | 1641002815197 | 2022-01-01 2:06:55 |
| AX4TZKEQOP_YqP5z1O2B | SUCCESS | 1641002869064 | 2022-01-01 2:07:49 |
The first project error (see below) reported as “issues”, the remainder “components” . Then everything else completed as expected.
The first failed project one is one of our larger projects, with these metrics:
| Lines of Code | 666,043 |
|---|---|
| Lines | 1,050,233 |
| Statements | 370,265 |
| Functions | 62,233 |
| Classes | 2,602 |
| Files | 2,329 |
| Comment Lines | 139,514 |
It has previously and since scanned successfully, as have the ones that followed in the minutes after; those are very small projects.
Seems to be related to Elastic Search data limitation:
Caused by: ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [2042481616/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2042481616/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=493847/482.2kb, in_flight_requests=0/0b, accounting=1042908/1018.4kb]]]`
Also not sure if related Q:
sonar.properties:sonar.ce.javaOpts=-Xmx2g -Xms128m -XX:+HeapDumpOnOutOfMemoryError
Elastic search System Info:
| Disk Available | 180 GB |
|---|---|
| Store Size | 476 MB |
| Translog Size | 1 KB |
| Open File Descriptors | 251 |
| Max File Descriptors | 65536 |
| JVM Heap Usage | 19.00% |
| JVM Heap Used | 397 MB |
| JVM Heap Max | 2 GB |
| JVM Non Heap Used | 159 MB |
| JVM Threads | 55 |
| Field Data Memory | 1 MB |
| Field Data Circuit Breaker Limit | 819 MB |
| Field Data Circuit Breaker Estimation | 1 MB |
| Request Circuit Breaker Limit | 1 GB |
| Request Circuit Breaker Estimation | 0 bytes |
| Query Cache Memory | 424 KB |
| Request Cache Memory | 21 KB |
| Search Indexes | |
| Index components - Docs | 2236 |
| Index components - Shards | 5 |
| Index components - Store Size | 4 MB |
| Index projectmeasures - Docs | 19609 |
| Index projectmeasures - Shards | 5 |
| Index projectmeasures - Store Size | 3 MB |
| Index metadatas - Docs | 17 |
| Index metadatas - Shards | 1 |
| Index metadatas - Store Size | 33 KB |
| Index rules - Docs | 19823 |
| Index rules - Shards | 2 |
| Index rules - Store Size | 13 MB |
| Index issues - Docs | 2747925 |
| Index issues - Shards | 5 |
| Index issues - Store Size | 453 MB |
| Index users - Docs | 397 |
| Index users - Shards | 1 |
| Index users - Store Size | 835 KB |
| Index views - Docs | 1 |
| Index views - Shards | 5 |
| Index views - Store Size | 4 KB |
Error messages
Fail to execute es requestES refresh request on indices 'issues'
"org.sonar.server.es.ElasticsearchException: Fail to execute es requestES refresh request on indices 'issues'
at org.sonar.server.es.EsClient.execute(EsClient.java:313)
at org.sonar.server.es.EsClient.refresh(EsClient.java:156)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:128)
at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:261)
at org.sonar.server.issue.index.IssueIndexer.indexOnAnalysis(IssueIndexer.java:115)
at org.sonar.ce.task.projectanalysis.step.IndexAnalysisStep.execute(IndexAnalysisStep.java:45)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$$Lambda$1222/0x0000000000000000.apply(Unknown Source)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:836)
Caused by: ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [2042481616/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2042481616/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=493847/482.2kb, in_flight_requests=0/0b, accounting=1042908/1018.4kb]]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
at org.elasticsearch.client.RestHighLevelClient$$Lambda$2654/0x0000000000000000.apply(Unknown Source)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1933)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1910)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1667)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)
at org.elasticsearch.client.IndicesClient.refresh(IndicesClient.java:689)
at org.sonar.server.es.EsClient.lambda$refresh$9(EsClient.java:156)
at org.sonar.server.es.EsClient$$Lambda$1342/0x0000000000000000.execute(Unknown Source)
at org.sonar.server.es.EsClient.execute(EsClient.java:311)
... 25 more
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:9001], URI [/issues/_refresh?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [2042481616/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2042481616/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=493847/482.2kb, in_flight_requests=0/0b, accounting=1042908/1018.4kb]","bytes_wanted":2042481616,"bytes_limit":2040109465,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [2042481616/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2042481616/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=493847/482.2kb, in_flight_requests=0/0b, accounting=1042908/1018.4kb]","bytes_wanted":2042481616,"bytes_limit":2040109465,"durability":"PERMANENT"},"status":429}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
... 31 more
"
Fail to execute es requestES refresh request on indices 'components'
"org.sonar.server.es.ElasticsearchException: Fail to execute es requestES refresh request on indices 'components'
at org.sonar.server.es.EsClient.execute(EsClient.java:313)
at org.sonar.server.es.EsClient.refresh(EsClient.java:156)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:128)
at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:170)
at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:101)
at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:64)
at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:54)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$$Lambda$1222/0x0000000000000000.apply(Unknown Source)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:836)
Caused by: ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [2044390368/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2044390368/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=493847/482.2kb, in_flight_requests=0/0b, accounting=1042908/1018.4kb]]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
at org.elasticsearch.client.RestHighLevelClient$$Lambda$2654/0x0000000000000000.apply(Unknown Source)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1933)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1910)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1667)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)
at org.elasticsearch.client.IndicesClient.refresh(IndicesClient.java:689)
at org.sonar.server.es.EsClient.lambda$refresh$9(EsClient.java:156)
at org.sonar.server.es.EsClient$$Lambda$1342/0x0000000000000000.execute(Unknown Source)
at org.sonar.server.es.EsClient.execute(EsClient.java:311)
... 28 more
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:9001], URI [/components/_refresh?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [2044390368/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2044390368/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=493847/482.2kb, in_flight_requests=0/0b, accounting=1042908/1018.4kb]","bytes_wanted":2044390368,"bytes_limit":2040109465,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [2044390368/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2044390368/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=493847/482.2kb, in_flight_requests=0/0b, accounting=1042908/1018.4kb]","bytes_wanted":2044390368,"bytes_limit":2040109465,"durability":"PERMANENT"},"status":429}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
... 34 more
"