Memory leaks in sonar gradle plugin

SonarQube
, ,
1

I have a multi project build in gradle(27 small subprojects) and I use a sonar gradle plugin for code analysis. The issue is that it use too much Metaspace memory to run.
Gradle is running in docker with heap space limits in 2Gb and unlimited metaspace. During the build metaspace eats all available memory and build is crashing with Docker daemon disapeared error

Must-share information (formatted with Markdown):

  • Sonarqube version Enterprise Edition Version 9.9.1 (build 69595), gragle 7.6.3, gradle sonar plugin 2.7,java 17, build is running in Github with Github actions
  • how is SonarQube deployed: self hosted
  • what are you trying to achieve - build a project with 4Gb of memory with gradle jacocoTestReport sonar command
  • what have you tried so far to achieve this:
    ** I’ve upgrade gradle to 8.4 with latest sonar plugin 4.4.1.3373 - same result
    ** I’ve rewritten build from docker to use gradle/gradle-build-action@v2 action - same result
    ** I’ve tried to limit the Metaspace to 1,5Gb - build is failing with OOM for metaspace
    ** I’ve tried to split gradle command to 2 separate runs gradle jacocoTestReport and gradle sonar. The first one completes successfully with 1250Mb of memory, the second one fails because 4Gb of memory is not enough to extend the Metaspace.

I’ve added logs after each run of sonar task for each of 27 subprojects.
I’ve tried to build the project on bigger runner and here is the memory footprint just for sonar task:
after the first project

Free memory=657.86106109619140625Mb.Total memory=900Mb.Max memory=1820.5Mb
	name: CodeHeap 'non-nmethods' usage init = 2555904(2496K) used = 2539776(2480K) committed = 2621440(2560K) max = 5828608(5692K)
	name: Metaspace usage init = 0(0K) used = 165047928(161179K) committed = 167051264(163136K) max = -1(-1K)
	name: CodeHeap 'profiled nmethods' usage init = 2555904(2496K) used = 27331840(26691K) committed = 27394048(26752K) max = 122912768(120032K)
	name: PS Old Gen usage init = 358088704(349696K) used = 131755408(128667K) committed = 358088704(349696K) max = 1431830528(1398272K)
	name: Compressed Class Space usage init = 0(0K) used = 22420416(21894K) committed = 23265280(22720K) max = 1073741824(1048576K)
	name: PS Survivor Space usage init = 22020096(21504K) used = 45594544(44525K) committed = 45613056(44544K) max = 45613056(44544K)
	name: PS Eden Space usage init = 134742016(131584K) used = 76551128(74756K) committed = 540016640(527360K) max = 611319808(596992K)
	name: CodeHeap 'non-profiled nmethods' usage init = 2555904(2496K) used = 7574144(7396K) committed = 7602176(7424K) max = 122916864(120036K)

after the 9th project

Free memory=425.83075714111328125Mb.Total memory=1120Mb.Max memory=1820.5Mb
	name: CodeHeap 'non-nmethods' usage init = 2555904(2496K) used = 2542080(2482K) committed = 2686976(2624K) max = 5828608(5692K)
	name: Metaspace usage init = 0(0K) used = 898942448(877873K) committed = 906166272(884928K) max = -1(-1K)
	name: CodeHeap 'profiled nmethods' usage init = 2555904(2496K) used = 119832960(117024K) committed = 119865344(117056K) max = 122912768(120032K)
	name: PS Old Gen usage init = 358088704(349696K) used = 521827632(509597K) committed = 633339904(618496K) max = 1431830528(1398272K)
	name: Compressed Class Space usage init = 0(0K) used = 120719072(117889K) committed = 124059648(121152K) max = 1073741824(1048576K)
	name: PS Survivor Space usage init = 22020096(21504K) used = 50133080(48958K) committed = 181403648(177152K) max = 181403648(177152K)
	name: PS Eden Space usage init = 134742016(131584K) used = 155928496(152273K) committed = 359661568(351232K) max = 359661568(351232K)
	name: CodeHeap 'non-profiled nmethods' usage init = 2555904(2496K) used = 28860032(28183K) committed = 28901376(28224K) max = 122916864(120036K)

after the 18th project

Free memory=275.74322509765625Mb.Total memory=1488Mb.Max memory=1820.5Mb
	name: CodeHeap 'non-nmethods' usage init = 2555904(2496K) used = 2548992(2489K) committed = 2686976(2624K) max = 5828608(5692K)
	name: Metaspace usage init = 0(0K) used = 1708462192(1668420K) committed = 1721630720(1681280K) max = -1(-1K)
	name: CodeHeap 'profiled nmethods' usage init = 2555904(2496K) used = 121813376(118958K) committed = 122912768(120032K) max = 122912768(120032K)
	name: PS Old Gen usage init = 358088704(349696K) used = 948743504(926507K) committed = 1012400128(988672K) max = 1431830528(1398272K)
	name: Compressed Class Space usage init = 0(0K) used = 230743768(225335K) committed = 236978176(231424K) max = 1073741824(1048576K)
	name: PS Survivor Space usage init = 22020096(21504K) used = 98029368(95731K) committed = 168296448(164352K) max = 168296448(164352K)
	name: PS Eden Space usage init = 134742016(131584K) used = 224370488(219111K) committed = 379584512(370688K) max = 384303104(375296K)
	name: CodeHeap 'non-profiled nmethods' usage init = 2555904(2496K) used = 88928384(86844K) committed = 88997888(86912K) max = 122916864(120036K)

after the last project

Free memory=461.138458251953125Mb.Total memory=1845.5Mb.Max memory=1845.5Mb
	name: CodeHeap 'non-nmethods' usage init = 2555904(2496K) used = 2551296(2491K) committed = 2686976(2624K) max = 5828608(5692K)
	name: Metaspace usage init = 0(0K) used = 2517064208(2458070K) committed = 2535129088(2475712K) max = -1(-1K)
	name: CodeHeap 'profiled nmethods' usage init = 2555904(2496K) used = 120125440(117310K) committed = 122912768(120032K) max = 122912768(120032K)
	name: PS Old Gen usage init = 358088704(349696K) used = 1430323912(1396800K) committed = 1431830528(1398272K) max = 1431830528(1398272K)
	name: Compressed Class Space usage init = 0(0K) used = 342094920(334077K) committed = 351076352(342848K) max = 1073741824(1048576K)
	name: PS Survivor Space usage init = 22020096(21504K) used = 0(0K) committed = 212336640(207360K) max = 212336640(207360K)
	name: PS Eden Space usage init = 134742016(131584K) used = 21284376(20785K) committed = 290979840(284160K) max = 290979840(284160K)
	name: CodeHeap 'non-profiled nmethods' usage init = 2555904(2496K) used = 115696000(112984K) committed = 115736576(113024K) max = 122916864(120036K)

as we can see the Metaspace is continuously growing.

Just to summarize - to build a project with all tests and jacoco reports we need 1250Mb and just to run sonarqube scan only - 4705Mb

2 Likes
4

Same issue here. As part of updating our build from Gradle 4.8 to 8.4, I needed to update the Sonar Gradle plugin to version 4.0.0.2929. It used to run the “sonarqube” (now “sonar”) task fine with 1024m of heap space, but now it runs out of memory even with 2g. This really feels like a memory leak to me.

6

Thanks for the report. Indeed, there seems to be an issue here. I’ve started investigating and can see some strange behavior but have not yet figured out what exactly is going on.

Are you able to share a heap dump when this occurs? Even better, also include one where you use a plugin version that does not have this issue.

If you can share a reproducer project that could help speed up the investigation significantly.

Until this issue is resolved, you may want to consider downgrading the Sonar plugin for Gradle version if you face issues with not enough memory being available otherwise.

7

I experienced the similar issues while using version 4.3.1.3277 up till 4.4.1.3373. It took about 51mins to run sonar scan coupled with excessive memory usage leading to CI server crashing.

When I downgraded to version 4.0.0.2929, sonar scan completes within 6minutes and no more memory issues.

8

Hi Johann,

Unfortunately I don’t know previous version where this issue didn’t exist. We’ve noticed this issue because we’ve reduced amount of memory for runner.
To be honest I don’t have a time now to work on a heap dump for it. Maybe later. We analyze more than 2k files divided to 27 projects, maybe you are collecting file names or analyze results for all files and never clear them when run scan for multi project build? it could be a reason for memory overflow

As a solution for us we’ve switched builds from sequential to parallel approach on several runners at the same time which is building faster now and within a memory limits