Unable to scan after upgrading postgresSql engine version only

[jeverakes (Justin Everakes)]

• SonarQube version: 9.9.1.69595
• SonarScanner version: 4.8.0.2856
• Deployed using Helm

We upgraded our aurora postgresSql database engine from version 11.17 to 12.13 and lost the ability to scan anything with SonarScanner.

Some details of what I’m seeing:

• SonarQube starts up fine, however, it will always do a full index whenever its restarted. There are no failing project reloads either.

• We are unable to scan any projects using the exact same set up and it always fails the same way (posted below)

DEBUG: GET 200 https://sonarqube.example.com/api/settings/values.protobuf | time=376ms
INFO: Load active rules (done) | time=8452ms
INFO: Load analysis cache
DEBUG: GET 404 https://sonarqube.example.com/api/analysis_cache/get?project=test_sonar_AYicDWSALMhkzVM5eMkz | time=109ms
INFO: Load analysis cache (404) | time=171ms
INFO: Load project repositories
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 1:14.246s
INFO: Final Memory: 22M/100M
INFO: ------------------------------------------------------------------------
ERROR: Error during SonarScanner execution
java.lang.IllegalStateException: Unable to load component class org.sonar.scanner.scan.filesystem.ProjectFileIndexer

Then it will exit with a stack trace which comes down to it timing out, which make sense, because it can’t find any projects.

I didn’t post the entire log because even with debug, everything comes back as 200, except for when it gets to loading the analysis cache.

We have a dev instance running the same versions of both and everything works fine. They are deployed exactly the same way and analysis testing is also done exactly the same.

If we were to restore a database snapshot, back down to the lower version, then it will start working again. Any help is greatly appreciated!

UPDATE

After restoring the old database, i can successfully scan still, but i still get a 404 on the cache, however it is able to load the projects.

16:43:45.477 INFO: Load active rules (done) | time=14655ms
16:43:45.481 INFO: Load analysis cache
16:43:45.685 DEBUG: GET 404 https://sonarqube.example.com/api/analysis_cache/get?project=aledade_test_sonar_AYicDWSALMhkzVM5eMkz | time=204ms
16:43:45.686 INFO: Load analysis cache (404) | time=205ms
16:43:45.736 INFO: Load project repositories
16:43:45.972 DEBUG: GET 200 https://sonarqube.example.com/batch/project.protobuf?key=test_sonar_AYicDWSALMhkzVM5eMkz | time=236ms
16:43:45.978 INFO: Load project repositories (done) | time=242ms

[Colin]

Hey there.

When upgrading – pay attention to this guide:

SonarQube becomes slow after PostgreSQL upgrade

Hello SonarQube Operators! It’s common that after performing an upgrade of PostgreSQL, a SonarQube instance connected to a PostgreSQL database can become painfully slow. Thankfully, we know how the issue can usually be addressed (it works almost every time!): by executing the following command against your Postgres database. VACUUM FULL ANALYZE For a PostgreSQL database, a VACUUM FULL command reclaims unused storage, while the ANALYZE command updates statistics used by the planner to determi…

@jeverakes

After restoring the old database, i can successfully scan still, but i still get a 404 on the cache, however it is able to load the projects.

This is fine and expected in certain contexts. The information is only available at DEBUG level. You can safely ignore it.

[jeverakes (Justin Everakes)]

Hi Colin,

Once again, thank you for your help. The full vacuum and analyze fixed the issue I was having.

The only thing that still seems to be still going on is that whenever sonarqube is restarted, it says all of the projects/branches are out of sync, and wants to index them again. Any thoughts on why that keeps happening?

Thanks!

[Colin]

@jeverakes

The only thing that still seems to be still going on is that whenever sonarqube is restarted, it says all of the projects/branches are out of sync, and wants to index them again. Any thoughts on why that keeps happening?

Hey there.

It sounds like the /data/ folder of your Sonarqube instance isn’t persisting between restarts. This folder holds the Elasticsearch data (which, when no longer present on startup, triggers a reindex). Take a look at the documentation on Persistency when deploying Sonarqube on Kubernetes.