SonarQube upgrade 6.7 > 7.6 - Too Many open files

During upgrade of SonarQube 6.7 -> 7.6, running into a few problems:

First problem was a snapshot too old error which came from DB - fixed this by dropping all unused columns for the tables (had to manually recreate project_measures table due to the amount of data and length of time drop command was taking)

Now db migration moves onto the next step:
delete person and file measures

It seems to be running a lot slower than previous step, processing less rows per second, obviously they’re doing different operations so understandable.

Below is the a section of web.log:

2019.05.23 08:15:34 INFO  web[][o.s.s.p.d.m.s.MassUpdate] 119165 snapshots processed (8 items/sec)
2019.05.23 08:15:59 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:100)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.<init>(DefaultProcessCommands.java:34)
        at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.secondary(DefaultProcessCommands.java:52)
        at org.sonar.server.app.WebServer.isOperational(WebServer.java:68)
        at org.sonar.server.app.WebServer.getStatus(WebServer.java:60)
        at org.sonar.process.ProcessEntryPoint.waitForOperational(ProcessEntryPoint.java:145)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:120)
        at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:100)
        at org.sonar.server.app.WebServer.main(WebServer.java:91)
Caused by: java.io.FileNotFoundException: /opt/sonarqube/sonarqube-7.6/temp/sharedmemory (Too many open files in system)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:97)
        ... 8 common frames omitted

sonar.log



2019.05.23 05:20:57 WARN  app[][startup]
################################################################################
      Database must be upgraded. Please backup database and browse /setup
################################################################################
2019.05.23 08:16:06 INFO  app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.05.23 08:16:07 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.05.23 08:16:07 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
2019.05.23 08:16:08 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
<-- Wrapper Stopped
(END)

I seen on requirements page: https://docs.sonarqube.org/latest/requirements/requirements/
it needs to be the below values or higher:

sysctl -w vm.max_map_count=262144
sysctl -w fs.file-max=65536
ulimit -n 65536
ulimit -u 2048

Here are the below values when I log in as the user starting sonarqube:

bash-4.1$ sysctl vm.max_map_count
vm.max_map_count = 262144
bash-4.1$ sysctl fs.file-max
fs.file-max = 175536
bash-4.1$ ulimit -u
3048
bash-4.1$ ulimit -n
125536

We’re using an oracle database 11g.

Hi,

First, you should never manually update the database unless you’ve been advised to with SQL provided by a SonarQube staffer. The fact that you have made manual updates may lead to other problems in the future. I advise you to restore the database backup you made before you started the upgrade (you made one, right?) and find another way to address the resource problem on your server.

 
Ann

Hi
Was advised to do so by a DBA, due to taking at least 50 hours. Recreated the table and indices just fine and dropped the other, after that I was actually able to get past the live_measures section
This is all just in our uat environment using a production clone db

Do you know if I should increase any other values? Or increase the values I’ve altered even further such as fs.file-max?

Hi Karl,

How to you start SonarQube?

System settings may be overridden in some conditions. Eg. gnome terminal, systemD started service, …

Hi
I start sonarqube with below:
/opt/sonarqube/sonarqube-6.7/bin/linux-x86-64/sonar.sh start

what settings would be overwritten?
we set a few in sonar.properties such as javaOpts, and a few others in sonar.properties

Hi @karl_murray,

I was referring to system settings, not SonarQube’s. Sorry this wasn’t clear.

Where do you run the command /opt/sonarqube/sonarqube-6.7/bin/linux-x86-64/sonar.sh start from? TTY? Some graphical terminal? If so, which one?

In some terminal, the system settings vm.max_map_count can be overwritten.
Eg.: in Gnome shell terminal, it is the case and the maxFileCount must be changed by updating /etc/systemd/system.conf (as root) and changing/setting the value of DefaultLimitNOFILE: DefaultLimitNOFILE=65536.
(here is a Stackoverflow post for reference)

Cheers,

oh okay
Yeah so in production, we have /etc/sysctl.d/99-sonarqube.conf with below settings:
vm.max_map_count=262144
fs.file-max=65536

and can see that /etc/security/limits.d/99-sonarqube.conf has:
- nofile 65536

in my original post the numbers I stated where in my UAT environment during the upgrade and still ran into those issues. Is there any problem with the limits/values in the original post that would still cause the issue?