BackendService consumes GBs of Heap (cause OOMs) saving content of all changed files

SonarQube for IDE IntelliJ Platform
,
1

Report we get with OOM:

[ 266K/ 13%/1.15GB] 1.22GB          1   ROOT: Java Frame: com.intellij.openapi.application.impl.ApplicationImpl.runUnlockingIntendedWrite(ApplicationImpl.java:842)
[ 266K/ 13%/1.15GB] 1.22GB          1   (root): com.intellij.openapi.application.impl.ApplicationImpl
[ 138K/  7%/1.13GB] 1.18GB          1   +-serviceParentDisposable: com.intellij.openapi.util.Disposer$2
[ 127K/  6%/1.13GB] 1.17GB          1   | (disposer-tree): org.sonarlint.intellij.core.BackendService
[ 127K/  6%/1.13GB] 1.17GB          1   | backendFuture: java.util.concurrent.CompletableFuture
[ 127K/  6%/1.13GB] 1.17GB          1   | result: jdk.proxy7.$Proxy142
[ 127K/  6%/1.13GB] 1.17GB          1   | h: org.eclipse.lsp4j.jsonrpc.services.EndpointProxy
[ 127K/  6%/1.13GB] 1.17GB          1   | delegate: org.eclipse.lsp4j.jsonrpc.RemoteEndpoint
[ 126K/  6%/1.13GB] 1.17GB          1   | out: org.sonarsource.sonarlint.core.rpc.protocol.SingleThreadedMessageConsumer
[ 126K/  6%/1.13GB] 1.17GB          1   | queue: java.util.concurrent.LinkedBlockingQueue
[ 126K/  6%/1.13GB] 1.17GB          1   | head: java.util.concurrent.LinkedBlockingQueue$Node
[ 126K/  6%/1.13GB] 1.17GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 124K/  6%/1.13GB] 1.17GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 118K/  6%/1.11GB] 1.15GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 118K/  6%/1.11GB] 1.15GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 118K/  6%/1.11GB] 1.15GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 117K/  6%/1.11GB] 1.15GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 117K/  6%/1.11GB] 1.15GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 117K/  6%/1.11GB] 1.14GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 117K/  5%/1.11GB] 1.14GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 117K/  5%/1.11GB] 1.14GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 117K/  5%/1.11GB] 1.14GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 114K/  5%/ 1.1GB] 1.14GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 108K/  5%/1.09GB] 1.13GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 108K/  5%/1.09GB] 1.13GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 104K/  5%/1.08GB] 1.12GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 104K/  5%/1.08GB] 1.12GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 104K/  5%/1.08GB] 1.12GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[ 102K/  5%/1.06GB]  1.1GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[97.0K/  4%/1.05GB] 1.08GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[96.9K/  4%/1.04GB] 1.08GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[96.7K/  4%/1.04GB] 1.08GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[96.7K/  4%/1.04GB] 1.08GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.3K/  4%/1.04GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.2K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.2K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.2K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.2K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.2K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.1K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.1K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.1K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.1K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.1K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.0K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[95.0K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[94.9K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[94.9K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[94.8K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[93.8K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[93.4K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[93.1K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[92.3K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[92.3K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[91.9K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[91.9K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[91.9K/  4%/1.03GB] 1.07GB          1   | next: java.util.concurrent.LinkedBlockingQueue$Node
[89.5K/  4%/ 692MB]  729MB          1   | +-next: java.util.concurrent.LinkedBlockingQueue$Node
[67.1K/  3%/ 622MB]  626MB          1   | | +-item: org.eclipse.lsp4j.jsonrpc.messages.NotificationMessage
[67.1K/  3%/ 622MB]  626MB          1   | | | params: org.sonarsource.sonarlint.core.rpc.protocol.backend.file.DidUpdateFileSystemParams
[67.1K/  3%/ 622MB]  626MB          1   | | | addedOrChangedFiles: java.util.ArrayList
[67.1K/  3%/ 622MB]  626MB          1   | | | elementData: java.lang.Object[]
[67.1K/  3%/ 622MB]  626MB      11240   | | | []: org.sonarsource.sonarlint.core.rpc.protocol.common.ClientFileDto
[10.9K/  0%/ 613MB]  613MB      10960   | | | content: java.lang.String
[10.9K/  0%/ 613MB]  613MB      10960 * | | | value: byte[]

It looks like when a lot of files change, ClientFileDto is created and stored in the queue for each file. As a result, all content of these files takes up heap space.

Maybe instead, “links” to files should be saved, and only during the sending should the actual content be fetched into memory.

We had about 100 similar OOMs in 242 version and already 11 reports for 243 version.

4

Hey @Maxim_Kolmakov, we are unfortunately aware of this issue, which was reported once by another user. We are currently working on a fix for it.

This should usually not happen. I believe this is because we are indexing many files that are being generated. We typically have heuristics that do not index these generated files, but it seems it’s not working in some rare cases.

Do you have more information about the files that are being sent? The last time it happened, the user was using Apache Ignite. Is it happening on startup or after a while?

5

Unfortunately, we don’t have any more information, since we receive those OOMs in the compacted form (HPROF are too large to store for thousands of customers).

But this pattern is very common, as I wrote, it’s the largest object in many OOMs and they still keep coming.

image
image2808Ă—1766 458 KB

They also happen for different products so it’s probably not project related. Maybe some massive branch switch (just guessing) where a lot of files are changed.

6

Maybe some massive branch switch (just guessing) where many files are changed.

That could very be true given the current implementation. I’ll include an improvement in the next version. File content should be sent only when a file is actually modified (typically when it’s opened in the editor); our design is not meant to send the file content of every file.

1 Like
7

Hey, this is just an update that we have released a new version of SonarQube for IDE, which contains the fix. Please do not hesitate to come back to us if you still face an issue.

And if you enjoy SonarQube for IDE, do not hesitate to leave a review on our marketplace page, it’s always appreciated!

1 Like