6.7.5 LTS - Unresponsive ES - Always need to reindex ES

elasticsearch

(Jeannie Chan) #1

Hello,

We are upgrading from SonarQube 5.6 LTS version (Community) to SonarQube 6.7.5 LTS version (build 38563) with Enterprise License.
We follow instructions from here https://docs.sonarqube.org/display/SONARQUBE67/SonarSource+Editions and have upraded our VMs according to the new spec here: https://docs.sonarqube.org/latest/requirements/requirements/. At the large part, it works. However, as usage increase, SonarQube performance went down to irresponsive. The dashboard no longer loads any projects and report uploads came to a halt. We have look into the following area:

  • LOC: We are way below from the LOC of our license
  • Application server Stat: RAM, CPU, Hard disk all at normal state
  • Database (MSSQL 2016): Performance is fine, though CPU usage is higher than previous version. No error in DB logs
  • All logs, include CE, Web, ES, Access, turned on DEBUG level, there is no Error/Warning in any of these logs.
  • IIS logs: no errors
  • We have also ensure that our MSSQL DB has set READ COMMITTED SNAPSHOT as per documentation.

We have tried:

  • Restart the application
  • Restart the server
    And nothing improve the performance of it

We have re-index ES (by renaming the /data/es5) and the performance return to be very snappy. However, after couple hours, the performance went back to irresponsive. We have exhausted our wits to diagnose any possible logs. Any pointer will be greatly appreciated. Thanks!


(Alexandre Frigout) #3

Greetings Jeannie,

Could you please share your server logs so we can have a look at it ?
Also, did you have the opportunity to have a look at our monitoring page ? This could give you pointers to look at to debug this slowness.
Finally, could you attach your system info file so we can perform a sanity check of your setup ?

Alex.


(Jeannie Chan) #4

Hello Alexandre,

We have finally resolve this issue last Friday (was trying to confirm stability over the weekend to ensure that it didn’t recur). The problem is at the Database server where MS SQL 2016 require much more memory usage when running SonarQube compare to the previous version (MS SQL 2012). As I have stated previously, none of the SonarQube logs has any error or hinting anything with this aspect where it stated everything is up and successful. What we discovered is that SoanrQube database queries (or checks) on system resources, and if resources are not high enough, it just repeat the same query and awaits forever. The endless query spinning causes the CPU of the database server become extremely busy. Once we upgrade the RAM and CPU on the database server, the problem is gone and SonarQube returns to normal operation.

If possible, I would suggest there is a recommendation on minimum CPU/RAM for Database server for SonarQube as well (It wasn’t listed here: https://docs.sonarqube.org/latest/requirements/hardware-recommendations/ and we did separate out DB and application server as recommended)

Once again, thank you.