How to properly configure memory to JVM on SonarQube

SonarQube Server / Community Build
, , , ,
1

  • Versions SonarQube 10.2.0 , Sonarlint 7.6.0 , Sonarqube Extension for Azure DevOps v5.17.2)
    *SonarQube is deployed with zip,

  • Description :
    We are are using SonarQube Enterprise Edition, version : 10.2.0.77647 and
    we are facing issues with instability due to Memory saturation of the JVM. In our situation we have 4,778 M LOC with 36 projects onboarded and we are allocating the following values:

# JVM options of Elasticsearch process
sonar.search.javaOpts=-Xmx4G -Xms4G -XX:MaxDirectMemorySize=1G -XX:+HeapDumpOnOutOfMemoryError
# WEB SERVER
sonar.web.javaOpts=-Xmx6G -Xms6G -XX:+HeapDumpOnOutOfMemoryError
# COMPUTE ENGINE
sonar.ce.javaOpts=-Xmx6G -Xms6G -XX:+HeapDumpOnOutOfMemoryError
  • The largest project has 1.2 M LOC
  • Store Size is 201.2 MB
    How can we calculate the proper value to each one ?

*LOG extraction

2024.01.24 17:37:08 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.8/sun.nio.ch.SocketDispatcher.read0(Native Method)
 java.base@17.0.8/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
 java.base@17.0.8/sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:261)
 java.base@17.0.8/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:312)
 java.base@17.0.8/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
 java.base@17.0.8/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
 java.base@17.0.8/java.net.Socket$SocketInputStream.read(Socket.java:966)
 java.base@17.0.8/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
 java.base@17.0.8/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
 java.base@17.0.8/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
 java.naming@17.0.8/com.sun.jndi.ldap.Connection.run(Connection.java:856)
 java.base@17.0.8/java.lang.Thread.run(Thread.java:833)

I look forward to hearing from you!
Thank you in advance.

2

Hi,

Welcome to the community!

The log snippet you’ve provided seems unrelated to this. Can you give more details on this instability?

 
Thx,
Ann

3

Hello Ann,

Thank you for your response, would you please check this log then :
2024.01.26 09:46:56 INFO es[o.e.n.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=D:\APP\SonarQube\current\temp, -XX:ErrorFile=D:\APP\SonarQube\current\logs\es_hs_err_pid%p.log, -Xlog:disable, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=D:\APP\SonarQube\current\temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Dcom.redhat.fips=false, -Des.enforce.bootstrap.checks=true, -Xmx7G, -Xms7G, -XX:MaxDirectMemorySize=1G, -XX:+HeapDumpOnOutOfMemoryError, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=tar, --module-path=D:\APP\SonarQube\current\elasticsearch\lib,
–add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]

Regards,
Amine

4

Hi Amine,

Again, what you’ve provided seems unrelated.

What are the symptoms of “instability” that you’re experiencing?

 
Ann

6

Hi Amine,

The 502 and the SOCKETTIMEOUT in your screenshot both reflect a failure to reach SonarQube. But unless you’re seeing errors in your server logs, I’m going to have to refer you elsewhere for the cause.

More than likely, you have something “helpful” on your network, such as a proxy, that’s interfering. You should talk to your network folks.

 
HTH,
Ann

7

Hello Ann,

I’m reaching out as a member of the same team as Amine.
We have checked with our network team and they confirmed that our SonarQube instance is not using the proxy. However, we noticed that when SonarQube becomes unreachable, attempting to access localhost:9000 yields no response. Also upon restarting the SonarQube service, we’ve noticed that the JVM for Elasticsearch quickly reaches the heap limit as soon as the service is up. This behavior seems unusual to us.
Could you please provide guidance on troubleshooting steps to diagnose the issue?

Thank you for your help.

Regards,
Marwa Hamdi

8

Hi Marwa,

Welcome to the community!

In my previous response I referenced checking the server logs. Have you done that?

 
Ann

10

Hi,

I’m not seeing any errors in the log snippet you’ve shared. Your 500-class response code indicates an error. Since you’re not finding any errors in your SonarQube log, this indicates the problem is elsewhere.

And specifically,

The 502 Bad Gateway error is an HTTP status code that occurs when a server acting as a gateway or proxy receives an invalid or faulty response from another server in the communication chain .

So again, you need to look outside SonarQube.

 
HTH,
Ann

11

Hello Ann,
We detected this error message after enabling the debug.
You can check also the log file
[Log sonarqube 18_03.txt redacted]

Request http://sonarqube.domain.com:9000/api/developers/search_events?projects=Clients-Aggregator&from=2024-03-14T16%3A50%3A37%2B0100 has been aborted by client
java.lang.IllegalStateException: Error while writing protobuf message

12

Hi,

Here’s that error in more detail:

Request http://sonarqube.domain.com:9000/api/developers/search_events?projects=Clients-Aggregator&from=2024-03-14T16%3A50%3A37%2B0100 has been aborted by client
java.lang.IllegalStateException: Error while writing protobuf message
...
Caused by: org.sonar.api.utils.text.WriterException: Fail to write JSON
...
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
...
Caused by: java.io.IOException: Connection reset by peer

aborted by client and Connection reset by peer indicate that the error logged here was caused externally to SonarQube. SonarQube was sending and the receiver just closed the connection.

I’m sorry, but this doesn’t change my answer: the problem is external to SonarQube.

Scrolling back in the history, I see a mention that you’re using SonarLint. Is that where you’re seeing the problem manifest? If so, I suggest making sure you’re on the latest SonarLint version.

 
Ann