SQ Community Edition is unexpectedly slow on AWS ECS

SonarQube
, , , ,
1

Hi all! I’ve just rolled out a service with SQ Community Edition on AWS ECS and can’t get over it’s slow processing. There’re few things bothering me now:

  1. The UI is noticeably slow, takes up 4-5 seconds to load the list of projects, opening a project for at least 6-7 seconds, but more importantly it takes more than 80s to load any Project → Measures section, which is too slow for a request to fulfil in the web-server timeout range and it simply 504-s back to the request (I could adjust it with the env var, but it’s just unbearably slow).
  2. Compute engine (or background tasks) seem to be quite slow as well — a relatively small project of 26k LOC takes 25 min to analyse. A bit larger one with 120k LOC takes around 50min. That’s a lot (right?).

My configuration:

  • SonarQube Community edition (10.3.0.82913) from docker hub;
  • AWS ECS Service running a task with SQ container on 2vCPU and 6GB RAM;
  • AWS EFS Volumes (NAT) used for the /data, /logs and /extensions
  • Postgres 12 on AWS RDS is used as the DB

I monitor utilization and it was on a very moderate level in every component, including under-utilized the ECS resources, so I went for bumping the heap settings to the following:

{
  name: "SONAR_CE_JAVAOPTS",
  value: "-Xms512m -Xmx2048m"
},            {
  name: "SONAR_WEB_JAVAOPTS",
  value: "-Xms512m -Xmx2048m"
},
{
  name: "SONAR_SEARCH_JAVAOPTS",
  value: "-Xms2048m -Xmx2048m"
},
{
  name: "SONAR_SEARCH_JAVAADDITIONALOPTS",
  value: "-Dnode.store.allow_mmap=false"
}

The logs are quite clear, there’s nothing extraordinary that could be related to the mentioned performance issues — no hints on why it’s that slow, no unexpected exceptions.

So, I’m wondering:

  1. First of all, is the described performance normal and expected, or something is off with my service?
  2. If something is off, how do I troubleshoot it? What’s the best way to debug such issues?
  3. Is the heap settings the only thing I can do to “boost” performance (though honestly I haven’t noticed any change since they were applied — it performs on a level similar to default). Is there a way to turn-off particular language analyzers as I don’t use them if that would help to optimize utilization?

Here are screenshots of the System page:

image_2024-01-04_113137615
image_2024-01-04_1131376151744×922 28.2 KB

image_2024-01-04_113226574
image_2024-01-04_1132265741618×858 27.4 KB

image
image1668×1456 156 KB

2

Most performance on SonarQube is related to database performance, including:

  • DB Resources (CPU, memory, disk space)
  • DB latency (how far away is the database network-wise)
  • DB efficiency (is the database regularly having its indices and statistics refreshed)?

I would suggest you start your search there. And – be careful with your memory settings. If you’re setting a max heap of 2 GB for all 3 processes, that doesn’t leave much (any) room for the OS and other processes if the heap max is met.

1 Like
3

Hi Colin! Thanks for a swift reply! The DB is currently located in a different region, so it might influence. I’m going to relocate it now and will circle back with the results.

1 Like
4

You were right! It’s now blazing fast! :smiley: The UI is very responsive and swift! Thanks a lot! Appreciate your advice! :pray:

5

One more question: based on your experience, what heap distribution is optimal for an average setup? Like 40% for the compute, 30% for the search and 30% for the web server?

8

Hey @lancedikson

A very late reply – but the vast majority of instances don’t need to deviate from the default. There’s no optimum “distribution”.

For commercial editions of SonarQube, we bump up the defaults

sonar.web.javaOpts=-Xmx1G -Xms128m -XX:+HeapDumpOnOutOfMemoryError
sonar.ce.javaOpts=-Xmx2G -Xms128m -XX:+HeapDumpOnOutOfMemoryError
sonar.search.javaOpts=-Xmx2G -Xms2G -XX:MaxDirectMemorySize=1G -XX:+HeapDumpOnOutOfMemoryError

Normally these only have to be adjusted if:

  • Very large projects are being submitted (millions of lines of code)
  • Multiple Compute Engine workers have been enabled (not relevant for Community Edition)