File leak with Gradle Plugin?

SonarQube Server / Community Build
,
1

Hello,
I’m trying to resolve a file leak with SonarQube 8.9.6 and the SonarQube Gradle Plugin 3.3
Once the analysis is done some temporary files are still used by the Gradle daemon and cannot be deleted in:
C:\Users\gtoison\.sonar\cache\269ee1005f2bd9f90d39b16616376948\sonar-findbugs-plugin.jar_unzip\META-INF\lib

I looked at some memory dumps of the Gradle process and it looks like OkHttp is preventing the cleanup because it is apparently waiting for some pooled connections to timeout.

The top element is a RandomAccessFile that cannot be deleted:

image
image1748×899 106 KB

From what I understand there were some lengthy discussion on ways to cleanup OkHttp resources in container-like applications and in the end it was added to OkHttp 4.3: shutdown of OkHttp resources.
Apparently sonar-scanner-api is using OkHttp 3.14.2
If I understand correctly the scanner should shutdown to release these cleanable resources held in the Cleaner queue

I might have completely misunderstood the problem but it would be great if anyone could advise here.
Thanks in advance!

1 Like
4

Hi!
Thanks for the work you put into investigating it.
One thing that I’m having a hard time understanding is why would OkHttp have a handle on a folder that contains the FindBugs plugin files.
Is FindBugs packaging OkHttp and running it itself?

5

The FindBugs plugin does not use or package OkHttp and the class name from the memory dump is org.sonarsource.scanner.api.internal.shaded.okhtttp.internal.connection.RealConnectionPool so I’m confident that this instance is created by the scanner.

What I think is happening is:

  • The Gradle daemon starts the scanner from its own JVM
  • The scanner opens a connection (pool) with the SonarQube server
  • That connection is registered as a java.net.SocketCleanable in the Cleaner linked list of cleanables
  • The scanner unzips the FindBugs plugin jar to add its content to a classloader
  • The unzipped contents of the jar are registered as java.io.FileCleanable in the Cleaner
  • Sockets and files are registered with the same singleton Cleaner from jdk.internal.ref.CleanerFactory
  • Once the analysis is complete OkHttp is still holding to the socket in the stack of AsyncTimeout.awaitTimeout()
  • Since it is a GC root, the socket cannot be GC’ed and the SocketCleanable does not execute
  • The next elements of the Cleaner linked list are held open because the 1st element is not cleaned
  • I suppose that the scanner attempts to delete the files but this fails because they are still open

I went down this rabbit hole investigating https://github.com/spotbugs/sonar-findbugs/issues/128
After disabling the part where SpotBugs plugins (SB-Contrib and FindSecBugs) are loaded dynamically, I still saw that temporary files where locked.
I have also seen in some cases that triggering a GC run in the Gradle daemon released these files, but not reliably. So maybe it works once the OkHttp socket has timed out, is cleaned and then the other cleanables are released.

I don’t think this would matter but I’m testing using Orchestrator, not a full SonarQube installation.

1 Like
6

Thanks again for the in depth investigation.

I did some profiling and got a snapshot at the time the analysis actually starts (scanner API finished its part), and I can only see one instance of http body/headers in memory, and that’s a static member ‘EMPTY_RESPONSE’ in okhttp.internal.util. The profiler did run a full GC.
My understanding based on the links you provided is that the okhttp resources don’t need to manually shutdown in most use cases. In our case, I think the scanner only uses 2 separate clients.

I also went through all connections established in the scanner API. They look good. There was a [leak](https://jira.sonarsource.com/browse/SCANNERAPI-189) fixed recently and it should be included in the scanner for gradle 3.3.

About the opened files, I see a lot of instances of RandomAccessFile that are created by classloaders to read the jar files, and are left open. That would be the case for C:\Users\gtoison\.sonar\cache\269ee1005f2bd9f90d39b16616376948\sonar-findbugs-plugin.jar_unzip\META-INF\lib, since files in that folder would be read by a classloader created to load findbugs. However, all the plugin classloaders are closed at the end of the analysis. I step-debugged this and it’s working.

These problems are very hard to debug and I feel like I reached a dead end.

7

Thank you very much for looking into it, I agree that this is hard to debug :slightly_smiling_face:

I will try to isolate the problem starting from a smaller case because it’s not easy to debug a Gradle plugin using a SonarQube plugin using a SpotBugs plugin …

1 Like