Elasticsearch exited unexpectedly - SQ 2025.1.4

[rahul.phatkare (Rahul R)]

We have upgraded our Sonarqube version from 9.9.9 to Sonarqube 2025.1.4 and it is up and running .

but when we tried to scan the project , it is getting scan successfully, but later SonarQube server is getting shutdown and after some time again starting.

In logs we have observed below error logs which is related to elastic search. Could you please assist what’s wrong here.PFA error log file for reference.

JFYI we are using m5.large AWS EC2 instance type to host our sonarqube , with below config in sonar.properties :

sonar.ce.javaOpts=-Xmx6144m -Xms256m -XX:+HeapDumpOnOutOfMemoryError

sonar.ce.workerCount=2

*sonar.search.javaAdditionalOpts=-Dlog4j2.formatMsgNoLookups=true
*
below are the logs :

2025.11.25 06:24:50 INFO ce[o.s.c.t.CeWorkerImpl] Execute task | project=no.dnb.tools:tools_sonar_build_test | type=REPORT | branch=feature/dev_setup | branchType=BRANCH | id=a969ca1f-2bf8-4275-b201-89446abddb4c | submitter=tools-test
2025-11-25T11:54:51.769+05:30
2025.11.25 06:24:51 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=37ms
2025-11-25T11:54:51.783+05:30
2025.11.25 06:24:51 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=14ms
2025-11-25T11:54:51.852+05:30
2025.11.25 06:24:51 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=61ms
2025-11-25T11:54:51.855+05:30
2025.11.25 06:24:51 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2025-11-25T11:54:52.135+05:30
2025.11.25 06:24:52 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=264ms
2025-11-25T11:54:52.193+05:30
2025.11.25 06:24:52 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=37ms
2025-11-25T11:54:52.249+05:30
2025.11.25 06:24:52 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=20 | status=SUCCESS | time=54ms
2025-11-25T11:54:52.255+05:30
2025.11.25 06:24:52 INFO ce[a969ca1f-2bf8-4275-b201-89446abddb4c][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=5ms
2025-11-25T11:54:55.048+05:30
ERROR: Elasticsearch exited unexpectedly, with exit code 137
2025-11-25T11:54:55.200+05:30
2025.11.25 06:24:55 WARN app[o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 137
2025-11-25T11:54:55.201+05:30
2025.11.25 06:24:55 INFO app[o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2025-11-25T11:54:55.472+05:30
2025.11.25 06:24:55 INFO ce[o.s.p.ProcessEntryPoint] Hard stopping process
2025-11-25T11:54:55.493+05:30
2025.11.25 06:24:55 INFO ce[o.s.ce.app.CeServer] Compute Engine is stopping…

Sq2025.1.4-error-while scanningproject.txt (16.8 KB)

[Colin]

Hey there.

An m5.large instance has 8 GB of RAM. It looks like you’re letting the CE take up to 6 GB. That’s not leaving a lot of room for the other Java processes.

I would suggest you knock the CE back down to the defaults (2 GB, or 2048m), or increase the amount of memory available by changing instance type.

[rahul.phatkare (Rahul R)]

Hi @Colin , as per suggestion we changed instance type to m5.xlarge which almost double in capacity than what we were using earlier m5.large.

with m5.large at least service was up and running and only falling when sonar analysis used to happen but after changing instance type sonarqube service is not coming up ,it is shutting down again and again.our sonar.properties is below for your reference which was working for Sonarqube 9.9.9 hosted on m5.large ,the same we are using now also with m5.xlarge:

sonar.ce.javaOpts=-Xmx6144m -Xms256m -XX:+HeapDumpOnOutOfMemoryError

sonar.ce.workerCount=2

sonar.search.javaAdditionalOpts=-Dlog4j2.formatMsgNoLookups=true

Could you please assist.

PFA logs file

Sq2025.1.4-error1-afterm5xlarge.txt (94.2 KB)

Sq2025.1.4-error-afterm5xlarge.txt (130.3 KB)

[Colin]

The logs seem to say:

Caused by: java.io.IOException: Cannot allocate memory

Which would imply there’s not enough free memory on the host system.

What does free -h show on that system?

[rahul.phatkare (Rahul R)]

@Colin

free -h

Below are the results of free -h when I tried to restart it

when it is trying to come it shows this and after that again restarting :

[Colin]

Hi @rahul.phatkare,

Thanks for sharing your screenshot—it clearly shows there’s no swap space at all, which is a great clue!

Looking at the logs, the key issue is this error: Caused by: java.io.IOException: Cannot allocate memory. Unlike the usual Java OutOfMemoryError (which points to your heap being full), this is actually a native Linux OS error (Error 12). It means the system is being very conservative and refusing to grant the memory your Java process needs—even if you still have free RAM—because it doesn’t have swap space as a backup.

This situation is pretty common on most standard EC2 images because they come with swap disabled by default. Whenever swap is set to 0 Bytes (as your screenshot shows), the kernel errs on the side of caution and blocks large memory allocations. That’s why your Java process (SonarQube) fails to start, even though you’re running on an m5.xlarge with 16GB RAM.

Here are a couple of ways you can solve this. I recommend Option 1 for most cases:

Option 1: Add Some Swap (Recommended)
Creating a small swap file (2GB - 4GB) gives the kernel a safety net and loosens up its memory rules. Here’s how you can do it:

# 1. Create a 4GB swap file
sudo dd if=/dev/zero of=/swapfile bs=128M count=32

# 2. Secure the swap file permissions
sudo chmod 600 /swapfile

# 3. Format it for swap
sudo mkswap /swapfile

# 4. Enable the swap file
sudo swapon /swapfile

# 5. Check that swap is active
free -h

Don’t forget: To keep this after a reboot, add it to /etc/fstab.

Option 2: Change Kernel Overcommit Policy (Alternative)
If swap can’t be enabled, you can relax the kernel’s memory rules:

sudo sysctl -w vm.overcommit_memory=1

Option 3: Reduce the amount of memory you’re asking for

Comment out your increased memory settings for sonar.ce.javaOpts, or set to a more conservative value (-Xmx3G)

After making either change, just restart SonarQube. Your startup issue should be resolved!

Let us know how it goes, and thanks for the helpful details and screenshot!

[rahul.phatkare (Rahul R)]

Sure , will let you know once we deploy . For now we have done changes as per 3rd option ,so our sonar.properties is as below .Could you please let us know will that be fine ?

# Web Service

sonar.web.javaOpts=-Xms1g -Xmx1g

# Compute Engine

sonar.ce.javaOpts=-Xms4g -Xmx4g -XX:+HeapDumpOnOutOfMemoryError

sonar.ce.workerCount=2

# Elastic search

sonar.search.javaOpts=-Xms2g -Xmx2g

#Disables message lookups in Log4j2.

sonar.search.javaAdditionalOpts=-Dlog4j2.formatMsgNoLookups=true

[ganncamp (G Ann Campbell)]

Hi,

Your updated settings are at or well above the default values. For normal usage you should be fine.

 
HTH,
Ann

[rahul.phatkare (Rahul R)]

we have done memory tuning as shown earlier also we increased at container level CPU(4096 MB = 4 vCPUs) and memory(14336MB = 14 GB) as well which was the main cause for the OOM . earlier container level memory was less(CPU = 2048 MB and memory=4096 MB) which was casing oom error as shown above.

I hope container level memory[CPU(4096 MB = 4 vCPUs) and memory(14336MB = 14 GB)] is fine now. Let me know if in case anything missed .

but we have observed below issues in :

1. we are seeing this by anonymous under background tasks

   

2. we have observed below logs , and seems they are fine but still need your confirmation on it:

   

3. found this in logs ,and seems its safe to use this api atleast in this release ,if not please let us know
   2025.12.02 12:35:18 WARN web[7fda8d05-55d2-44a7-b7d6-386735b8a906] /api/users/search Web service is deprecated since 10.4 and will be removed in a future version.

if any further issues found will update here but as of now found these above issue .Could you please assist.

[ganncamp (G Ann Campbell)]

Hi,

We try to keep it to one topic per thread. Otherwise it can get messy, fast. You’ve posed 3 new questions here. I’ll make a quick pass, but if you have any (further) followups, I’ll need you to create new threads for them.

1. This is related to the collection of SCM metadata during analysis and to whether the people who last committed on the issue lines have accounts in SonarQube.

2. Yes, you can ignore these logs.

3. I mean… the API is still there. But as the message says, don’t rely on it long-term.

 
HTH,
Ann

[rahul.phatkare (Rahul R)]

Thanks so much !

[rahul.phatkare (Rahul R)]

Hi @Colin ,

after memory tuning for EC2 (m5.xlarge) and also increased CPU and memory at container level .now free -h on instance looks like below , hope that’s looks good !

[ganncamp (G Ann Campbell)]

Hi,

Your memory settings should be good. Feel free to create a new thread if you experience subsequent problems.

 
Ann