Sonarqube 7.9.4 CE: Too many open files

SonarQube
#1

Hello,

My Sonarqube 7.9.4 CE does not start anymore.

I got these kind of errors:

2022.03.10 00:56:27 WARN es[][o.e.c.r.a.AllocationService] failing shard [failed shard, shard [metadatas][0], node[eGKx9oIOStOGeJ6V0ePiCw], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=Ql6jr5dGSSujR0UyjHjEKw], unassigned_info[[reason=ALLOCATION_FAILED], at[2022-03-10T00:56:26.234Z], failed_attempts[4], delayed=false, details[failed shard on node [eGKx9oIOStOGeJ6V0ePiCw]: failed recovery, failure RecoveryFailedException[[metadatas][0]: Recovery failed on {sonarqube}{eGKx9oIOStOGeJ6V0ePiCw}{-YtQ5M7tSoKru0HneQ0BUA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/opt/sonarqube/data/es6/nodes/0/indices/FdEFB_zgSjmvFu5P7wxYZA/0/translog/translog-3245.ckp: Too many open files]; ], allocation_status[no_valid_shard_copy]], message [failed recovery], failure [RecoveryFailedException[[metadatas][0]: Recovery failed on {sonarqube}{eGKx9oIOStOGeJ6V0ePiCw}{-YtQ5M7tSoKru0HneQ0BUA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/opt/sonarqube/data/es6/nodes/0/indices/FdEFB_zgSjmvFu5P7wxYZA/0/translog/translog-3245.ckp: Too many open files]; ], markAsStale [true]]

My instance is running in a container.

I have seen that it can be caused by the system limitations. But I think the values on the host are high enough (according to Prerequisites and Overview | SonarQube Docs):

# sysctl vm.max_map_count
vm.max_map_count = 524288
# sysctl fs.file-max
fs.file-max = 131072
# ulimit -n
131072
# ulimit -u
8192
#2

Hi @cosades and welcome to the community :wave:

Your version is past EOL. You should upgrade to either the latest version or the current LTS at your earliest convenience. Your upgrade path is:

7.9.4 → 8.9.7 → 9.3 (last step optional)

You may find the Upgrade Guide and the LTS-to-LTS Upgrade Notes helpful. If you have questions about upgrading, feel free to open a new thread for that here.

We updated our recommendations regarding open file descriptors with the 8.4 release as we noticed that the previous once were hit from time to time. feel free to increase them further as stated in the LTS to LTS Upgrade notes to get rid of the error at hand.

Hope that helps :slight_smile: