SonarQube service stops due to java.lang.OutOfMemoryError: Java heap space

After an upgrade the Sonarqube server does not come back up due to memory issues.

  • which versions are you using: Sonarqube Community Edition 25.1

  • how is SonarQube deployed: zip

  • what are you trying to achieve: Working Sonarqube instance
    Upgrade Sonarqube from 10.4 → 25.1 (should be possible going by upgrade path)
    Didn’t have any memory issues with version 10.4
    After the upgrade Sonarqube shutsdown due to memory issues.
    Sonarqube exited with code: 143
    The largest project in this instance is 1-2M LoC

  • what have you tried so far to achieve this:
    validate settings are according to requirements
    increased memory settings by 2-3x
    clearing es cache
    maintenance on database
    elevated access on database for sonarqube user

  • logs

2025.01.24 10:52:00 INFO  web[][o.a.c.h.Http11Processor] The host [_] is not valid\n Note: further occurrences of request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: null
	at org.apache.tomcat.util.http.parser.Host.parse(Host.java:74)
	at org.apache.tomcat.util.http.parser.Host.parse(Host.java:43)
	at org.apache.coyote.AbstractProcessor.parseHost(AbstractProcessor.java:298)
	at org.apache.coyote.http11.Http11Processor.prepareRequest(Http11Processor.java:790)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:833)
2025.01.24 10:52:37 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.IllegalStateException: Fail to execute request to select measures of project AWDWil5FhqZ8At2Q6wmj
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectMeasures(ProjectMeasuresIndexerIterator.java:247)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.doNext(ProjectMeasuresIndexerIterator.java:229)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.doNext(ProjectMeasuresIndexerIterator.java:52)
	at org.sonar.core.util.CloseableIterator.bufferNext(CloseableIterator.java:96)
	at org.sonar.core.util.CloseableIterator.hasNext(CloseableIterator.java:87)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.doIndex(ProjectMeasuresIndexer.java:189)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.indexOnStartup(ProjectMeasuresIndexer.java:81)
	at org.sonar.server.es.IndexerStartupTask.synchronousIndexing(IndexerStartupTask.java:83)
	at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:53)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:135)
	at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:46)
	at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:131)
	at org.sonar.server.platform.PlatformImpl.executeStartupTasks(PlatformImpl.java:204)
	at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:365)
	at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:119)
	at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:349)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalStateException: Fail to execute request to select the project biggest branch
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectProjectBranchForNcloc(ProjectMeasuresIndexerIterator.java:315)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.lambda$selectMeasures$0(ProjectMeasuresIndexerIterator.java:239)
	at java.base/java.util.Optional.flatMap(Optional.java:289)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectMeasures(ProjectMeasuresIndexerIterator.java:239)
	... 20 common frames omitted
Caused by: org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2391)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:517)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:434)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectProjectBranchForNcloc(ProjectMeasuresIndexerIterator.java:311)
	... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.postgresql.core.PGStream.receiveTupleV3(PGStream.java:619)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2387)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:517)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:434)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectProjectBranchForNcloc(ProjectMeasuresIndexerIterator.java:311)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.lambda$selectMeasures$0(ProjectMeasuresIndexerIterator.java:239)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator$$Lambda$1905/0x0000000801d326b0.apply(Unknown Source)
	at java.base/java.util.Optional.flatMap(Optional.java:289)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectMeasures(ProjectMeasuresIndexerIterator.java:239)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.doNext(ProjectMeasuresIndexerIterator.java:229)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.doNext(ProjectMeasuresIndexerIterator.java:52)
	at org.sonar.core.util.CloseableIterator.bufferNext(CloseableIterator.java:96)
	at org.sonar.core.util.CloseableIterator.hasNext(CloseableIterator.java:87)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.doIndex(ProjectMeasuresIndexer.java:189)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.indexOnStartup(ProjectMeasuresIndexer.java:81)
	at org.sonar.server.es.IndexerStartupTask.synchronousIndexing(IndexerStartupTask.java:83)
	at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68)
	at org.sonar.server.es.IndexerStartupTask$$Lambda$1822/0x0000000801d10000.accept(Unknown Source)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:53)
	at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1$$Lambda$1821/0x0000000801d0c400.accept(Unknown Source)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:135)
	at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:46)
	at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:131)
	at org.sonar.server.platform.PlatformImpl.executeStartupTasks(PlatformImpl.java:204)
2025.01.24 10:52:37 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2025.01.24 10:52:37 INFO  web[][o.s.s.n.NotificationDaemon] Notification service stopped
2025.01.24 10:52:37 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2025.01.24 10:52:37 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.
2025.01.24 10:52:37 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [Progress[BulkIndexer[projectmeasures]]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@17.0.7/java.lang.Object.wait(Native Method)\n java.base@17.0.7/java.util.TimerThread.mainLoop(Timer.java:563)\n java.base@17.0.7/java.util.TimerThread.run(Timer.java:516)

Hello @k0mand1r,

Thank you for the report. We had similar issues with measures in other parts of the application but missed this one. I created a ticket to fix it: SONAR-24216.
In the meantime, I suggest rolling back to the previous version you used, or you can try increasing the memory even more.

Can you confirm you are using the dependency-check plugin?

Thank you for your reply.
We are not using the DependencyCheck plugin for about 2 years now and I’ve also tried removing all current other plugins (verified they are supported) and just get a clean upgrade going.
I’m in the process of recovering the database, as there was a schema upgrade.
Increasing the memory is a last option if it’s confirmed just the memory is the issue.

Good to know it wasn’t me in this case, and looking forward to a working 25.1.X release :slight_smile:

It can still be related to this plugin if you have some old branches that were analyzed while this plugin was active and they haven’t been analyzed since.
To confirm, you can run this query on the database after you’ve restored it:

select * from live_measures lm
inner join metrics m on lm.metric_uuid = m.uuid
where m.name = 'report';

If you get very large records in the result (column measure_data), it confirms that the issue comes from the plugin. Then you can delete those records and the upgrade should be successful.

Thank you for the guidance. I’ll give it a go and get back at this.
If it’s the case there are some old records that are causing this issue. Can I just removed those records or are they directly from the database or should I delete the branches/projects through the webUI?

You can remove those records directly in the database. If you are no longer using the plugin, they are not used for anything. But always work with backups just in case :slight_smile:
Deleting the branches/projects would be more tricky as you would have to identify them (and you might not want to delete those branches).

The query doesn’t seem to work. It fails on identifying live_measures is missing.
Ran it from the context of the sonarqube database.

Did you restore your 10.4 database? Are you able to query some other tables, issues for example?

Yes, the table seems empty?
We’ve cleared a lot of stale projects a while back after stepping away from the DependencyCheck plugin. So it would be weird if there would still be projects with this type of data.

Thank you for checking.
I’d be also interested in the results of this query:

select m.name, lm.measure_data, lm.project_uuid from live_measures lm
left join metrics m on m.uuid = lm.metric_uuid
where octet_length(lm.measure_data) > 1000000;

Could you check if you get any results?

The result is the same. Empty screen with the header.

Just to be sure, you can do this query to verify the table is not empty:

select * from live_measures;

Will have to try it outside office hours. The application was responding extremely slow running this query and it apparently takes a while.

Query ran for 1 hr without finishing, with high CPU usage. Can’t confirm the table isn’t empty, but I would expect it to have finished a lot sooner if it was.

Yes, it’s probably not empty. I’m running out of ideas about why it failed with an OOM error :frowning:
In any case, the fix to reduce memory usage will be included in the February release of Community Build (25.2). You can try to upgrade again when it’s released.

Thanks for the help Eric. I’ll wait for the new release and see if that fixes things.
The documentation doesn’t provide any guidance on the required resources the database server should have. I might try upping the resources as a final resort.

1 Like