ElasticSearch reindex VERY slow: 5 minutes per project

We are using SonarQube Community Edition Version 8.4.1 (build 35646)

We are having issues where projects are not showing up in Search and the Issues tab is empty even though the Overview shows we have bugs/code smells.

We are trying to fix this by deleting our $SQ_HOME/data/es6 directory to trigger a reindex.

However, when SonarQube comes back online the reindex is unacceptably slow, where it takes 5 minutes to complete one project. What’s strange is the ComputationStepExecutor log lines show processing in the milliseconds, yet it waits 5 minutes before picking up the next project:
sonar_ce.log (15.5 KB)

We can not recreate the slow reindexing in our other environments, even after a full restore. We appreciate any input on why our search and bug reports are broken, and why our reindex is taking so long to run. Thank you!

I guess a good way to start investigating this would be to find the differences between your production environment and your other environments. You may want to look for how the database was restored, DB version, disk IO performance, network topology between SQ and the DB, etc.

Thank you for replying Pierre. DB and networking are from the same cloudformation template, and we suspected disk IO as well but the performance was the same across environments. Right now our only hunch is something with the EBS volume which stores $SQ_HOME.
Do the symptoms of “some projects not appearing in search results” and “blank Issues/Security Hotspots tab” align with a corrupt elasticsearch index?

Definitely. You need to rebuild your ES index from scratch. Also, double-check that your servers are set with the right date and time. The compute engine is supposed to pause for 5 seconds maximum between tasks, not 5 minutes.