SonarQube upgrade 6.7 > 7.6 - Too Many open files

karl_murray · May 23, 2019, 1:15pm

During upgrade of SonarQube 6.7 -> 7.6, running into a few problems:

First problem was a snapshot too old error which came from DB - fixed this by dropping all unused columns for the tables (had to manually recreate project_measures table due to the amount of data and length of time drop command was taking)

Now db migration moves onto the next step:
delete person and file measures

It seems to be running a lot slower than previous step, processing less rows per second, obviously they’re doing different operations so understandable.

Below is the a section of web.log:

2019.05.23 08:15:34 INFO  web[][o.s.s.p.d.m.s.MassUpdate] 119165 snapshots processed (8 items/sec)
2019.05.23 08:15:59 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:100)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.<init>(DefaultProcessCommands.java:34)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.secondary(DefaultProcessCommands.java:52)
        at org.sonar.server.app.WebServer.isOperational(WebServer.java:68)
        at org.sonar.server.app.WebServer.getStatus(WebServer.java:60)
        at org.sonar.process.ProcessEntryPoint.waitForOperational(ProcessEntryPoint.java:145)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:120)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:100)
        at org.sonar.server.app.WebServer.main(WebServer.java:91)
Caused by: java.io.FileNotFoundException: /opt/sonarqube/sonarqube-7.6/temp/sharedmemory (Too many open files in system)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:97)
        ... 8 common frames omitted

sonar.log



2019.05.23 05:20:57 WARN  app[][startup]
################################################################################
      Database must be upgraded. Please backup database and browse /setup
################################################################################
2019.05.23 08:16:06 INFO  app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.05.23 08:16:07 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.05.23 08:16:07 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
2019.05.23 08:16:08 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
<-- Wrapper Stopped
(END)

I seen on requirements page: https://docs.sonarqube.org/latest/requirements/requirements/
it needs to be the below values or higher:

sysctl -w vm.max_map_count=262144
sysctl -w fs.file-max=65536
ulimit -n 65536
ulimit -u 2048

Here are the below values when I log in as the user starting sonarqube:

bash-4.1$ sysctl vm.max_map_count
vm.max_map_count = 262144
bash-4.1$ sysctl fs.file-max
fs.file-max = 175536
bash-4.1$ ulimit -u
3048
bash-4.1$ ulimit -n
125536

We’re using an oracle database 11g.

ganncamp · May 24, 2019, 1:41pm

Hi,

First, you should never manually update the database unless you’ve been advised to with SQL provided by a SonarQube staffer. The fact that you have made manual updates may lead to other problems in the future. I advise you to restore the database backup you made before you started the upgrade (you made one, right?) and find another way to address the resource problem on your server.

Ann

karl_murray · May 24, 2019, 7:36pm

Hi
Was advised to do so by a DBA, due to taking at least 50 hours. Recreated the table and indices just fine and dropped the other, after that I was actually able to get past the live_measures section
This is all just in our uat environment using a production clone db

Do you know if I should increase any other values? Or increase the values I’ve altered even further such as fs.file-max?

sns-seb · June 14, 2019, 8:32am

Hi Karl,

How to you start SonarQube?

System settings may be overridden in some conditions. Eg. gnome terminal, systemD started service, …

karl_murray · July 10, 2019, 9:16am

Hi
I start sonarqube with below:
/opt/sonarqube/sonarqube-6.7/bin/linux-x86-64/sonar.sh start

what settings would be overwritten?
we set a few in sonar.properties such as javaOpts, and a few others in sonar.properties

sns-seb · July 10, 2019, 9:39am

Hi @karl_murray,

I was referring to system settings, not SonarQube’s. Sorry this wasn’t clear.

Where do you run the command /opt/sonarqube/sonarqube-6.7/bin/linux-x86-64/sonar.sh start from? TTY? Some graphical terminal? If so, which one?

In some terminal, the system settings vm.max_map_count can be overwritten.
Eg.: in Gnome shell terminal, it is the case and the maxFileCount must be changed by updating /etc/systemd/system.conf (as root) and changing/setting the value of DefaultLimitNOFILE: DefaultLimitNOFILE=65536.
(here is a Stackoverflow post for reference)

Cheers,

karl_murray · July 10, 2019, 10:01am

oh okay
Yeah so in production, we have /etc/sysctl.d/99-sonarqube.conf with below settings:
vm.max_map_count=262144
fs.file-max=65536

and can see that /etc/security/limits.d/99-sonarqube.conf has:
- nofile 65536

in my original post the numbers I stated where in my UAT environment during the upgrade and still ran into those issues. Is there any problem with the limits/values in the original post that would still cause the issue?

p928s1984 · June 18, 2020, 1:10am

were you ever able to get this resolved? I am having very similar issues and have been working on trying to get this resolved for weeks.

I also was able to upgrade in lower env no problem.

I know this is an older thread, hopefully someone can assist.

guru_sathish · September 5, 2020, 4:59am

I am also facing the same issue? Let me know how to resolve this.

Tobias_Trabelsi · September 7, 2020, 6:25am

Hi @guru_sathish and @p928s1984 ,

you can increse the values as follows:

sysctl -w vm.max_map_count=524288
sysctl -w fs.file-max=131072
ulimit -n 131072
ulimit -u 8192

depending on your system startup type and installation these need to be adapted to the specific systemd service, sysctl- or limits configuration. at least in our database migration tests we never hit the limit of fs.file-max=131072, but if there is more than sonarqube running on your system you can increase this value further

hope that helps

guru_sathish · September 7, 2020, 10:37am

This resolved the issue.

system · September 14, 2020, 12:27pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.