SonarQube Server Stopped Responding - Elasticsearch Shard Failure and Read Past EOF Exception

SonarQube Server / Community Build
, ,
1

Hello SonarQube Support Team,

We are encountering a critical issue with our Production SonarQube setup, where the server has stopped responding. This seems to be related to an Elasticsearch issue that surfaced after our recent activities.

Here are the relevant details:

  • SonarQube Version: 9.9.5 (Docker setup)
  • Elasticsearch Version: 7.17.15
  • System Specs: Container with 4 cores and 16 GB of RAM
  • OS: Linux/5.15.164.1-1.cm2/amd64 (containerized)

Problem Description:

Our SonarQube instance was functioning correctly, but it stopped responding, and we started seeing the following errors in the logs. The issue seems to revolve around an Elasticsearch shard failure during a refresh operation, leading to a series of exceptions such as EOFException and FileAlreadyExistsException.

Logs show repeated instances of:

  • java.io.IOException: read past EOF
  • java.nio.file.FileAlreadyExistsException

Here is a snippet of the log:

java.io.EOFException: read past EOF: NIOFSIndexInput(path="/opt/sonarqube/data/es7/nodes/0/indices/7AU_OguyRCSKeuWakFRsmQ/1/index/_kix_7_Lucene80_0.dvd") [slice=docs]

This resulted in our SonarQube server becoming unresponsive. Additionally, our cluster health status changed from GREEN to RED due to shard failures, followed by a lucene commit failed message, and the Elasticsearch engine failing with a FlushFailedEngineException.

Error Summary:

  • Error 1: java.io.EOFException: read past EOF
  • Error 2: java.nio.file.FileAlreadyExistsException
  • Error 3: FlushFailedEngineException: Flush failed

I have deployed SonarQube on Azure Container App and configured an Azure Fileshare volume mount to store Elasticsearch indices data.

Current sonar.properties parameters are:

# Compute Engine configuration
sonar.ce.javaOpts=-Xms4096m -Xmx4096m -XX:+HeapDumpOnOutOfMemoryError

# JVM options of Elasticsearch process
sonar.search.javaOpts=-Xms4096m -Xmx4096m -XX:+HeapDumpOnOutOfMemoryError

# Same as previous property, but allows to not repeat all other settings like -Xmx
sonar.search.javaAdditionalOpts=-Dnode.store.allow_mmap=false

These errors appear to be related to index refresh operations and file handling in Elasticsearch, causing the SonarQube service to halt.

We would appreciate any guidance or suggestions on resolving this issue and bringing the server back to a stable state.

Thank you!
sonarqube_es.log (71.8 KB)

2

Hi,

Welcome to the community!

What were these activities?

Why do you feel the need to provide the Elasticsearch version? Surely you’re running the version embedded in SonarQube?

Have you tried forcing an ElasticSearch reindex?

 
Ann

3

Ann

I’m currently using the Docker image “sonarqube:lts-enterprise” with the embedded Elasticsearch (no additional installations). To speed up the server startup, I’ve mounted a volume on Azure File Share. However, this setup has become a performance bottleneck.

Could you please advise on best practices to maintain high server performance while handling approximately 800K lines of code across 1000+ projects, with a database size of around 70 GB?

I’d greatly appreciate any insights or reference materials you could share.

Thanks in advance!

4

Hi,

The docs should help.

 
Ann