Hello,
We have Sonar deployed with a helm chart. After upgrading to version 9.9-developer it seems there was an issue w/ the database migration. The sonar instance is stuck in a crash loop. Running a db restore has been ineffective. Reverting the version in values.yaml to 9.4 did not fix this.
What can we do to resolve this? I’ve attached the error logs from the pods.
sonarqube-lts.txt (14.9 KB)
sonarqube-postgresql-errorlogs.txt (5.8 KB)
Hi,
Welcome to the community!
It’s not clear to me what the sonarqube-lts.txt file is. It seems to be all the SonarQube application logs concatenated into one file?
This:
2023.03.04 18:39:15 INFO app[][o.s.a.SchedulerImpl] Stopping SonarQube
2023.03.04 18:39:15 INFO app[][o.s.a.SchedulerImpl] Sonarqube has been requested to stop
2023.03.04 18:39:15 INFO app[][o.s.a.SchedulerImpl] Stopping [Compute Engine] process...
2023.03.04 18:39:15 INFO app[][o.s.a.SchedulerImpl] Stopping [Web Server] process...
2023.03.04 18:39:15 INFO web[][o.s.p.ProcessEntryPoint] Gracefully stopping process
2023.03.04 18:39:15 INFO web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2023.03.04 18:39:15 INFO web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.
2023.03.04 18:39:15 INFO web[][o.s.s.app.WebServer] Web Server stopped
2023.03.04 18:39:15 INFO app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.03.04 18:39:15 INFO app[][o.s.a.SchedulerImpl] Stopping [ElasticSearch] process...
2023.03.04 18:39:16 INFO app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.03.04 18:39:16 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2023.03.04 18:39:16 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
indicates graceful termination to me, so I’m not sure why you think it’s “stuck in a crash loop”.
Earlier in that file, we see this:
2023.03.04 18:37:25 WARN app[][startup] ################################################################################
2023.03.04 18:37:25 WARN app[][startup] The database must be manually upgraded. Please backup the database and browse /setup. For more information: https://docs.sonarqube.org/latest/setup/upgrading
2023.03.04 18:37:25 WARN app[][startup] ################################################################################
Did you do that?
Ann
Hi Ann,
By crash loop I meant that it would continuously crash after ArgoCD tried to deploy the pods again.
After troubleshooting, we found that we were unable to properly restore the postgresql db to the backed up version (see Unable to restore sonar DB)
because the default sonarUser does not have permissions to drop sonarDB which is required for the restore process.
I did follow the upgrade guide and started the migration process here:
http://yourSonarQubeServerURL/setup
But the sonar instance crashed.
Please let me know if there’s something we can try to fix this.
Thanks,
Austin
Related to: Sonarqube Crashes after upgrade from 9.4 to 9.9
After a failed upgrade to version 9.9-developer, I tried restoring a postgres backup taken on version 9.3-developer. This is for a kuberentes deployment of sonar. I used this documentation for reference: Backup and restore
I ran
pg_dump -h localhost -p 5432 -U sonarUser -d sonarDB > db_backup.sql
I have tried restoring this backup with different commands with no success:
pg_restore -h localhost -p 5432 -U sonarUser -d sonarDB db_backup.sql
psql -h localhost -p 5432 -U sonarUser -d sonarDB -f ./tmp/db_backup.sql
pg_restore -h localhost -p 5432 -U sonarUser -d sonarDB --set ON_ERROR_STOP=on db_backup.sql
I have tried restoring to 9.9, a reverted 9.3, as well as a test instance (which runs fine) running 9.9.
I ran into similar errors for each attempt (see attached screenshots). Am I missing a step in the restore process?
Hi,
This is a bit out of scope for this community. Perhaps the Postgres folks can help?
Ann
Hi Ann,
We are not able to dump the DB to begin the restore b/c the default ‘sonarUser’ set in the postgresql server doesn’t not have permissions to do do. If you are able to provide the password for the ‘postgres’ user we can try it again.
Due to this blocker, we created a new DB and restored our backup to it in the same server but there seems to have been an issue in the sonar helm chart. It does not point to the new DB we want to use.
Below is the first few of our values.yaml lines pointing the jdbc driver to the new replacement_db.
sonarqube-lts:
image:
tag: 9.9.0-developer
jdbcOverwrite:
# If enable the JDBC Overwrite, make sure to set `postgresql.enabled=false`
enable: true
# The JDBC url of the external DB
jdbcUrl: "jdbc:postgresql://sonarqube-test-postgresql:5432/replacement_db"
jdbcUsername: "sonarUser"
jdbcPassword: "sonarPass"
It seems like that this config map (sonarqube 8.0.0+463 · sonarsource/sonarqube) is not being created as it should be on deployment.
Please let me know if there’s anything else we can try or if there is a more thorough restore doc.
Thanks,
Austin
Hi Austin,
I’ve combined your two threads since they appear to be about the same thing.
That’s outside the scope of this community. The SonarQube DB user needs to have full permissions on its schema. If you’re dropping and restoring the schema, that’s outside of our operating procedures and something you’ll need to work through yourself. But it seems simple enough to me that if your process crashes because you haven’t granted the account permissions… that you grant the account permissions.
Ann
Hi Ann,
This thread was created to discuss why the upgrade from 9.4 to 9.9 was unsuccessful. The second thread was created to get help for restoring our production Sonarqube instance.
The default sonarUser account is the only user account we have access to in the postgresdb. This user account was created per specifications of Sonarqube’s helm chart, which is why I am asking for help here. The sonarUser account does not come with many permissions and is not able to elevate it’s own permissions. If this is in fact possible please let me know how.
Since a restore for a postgresql DB (which is what Sonarqube uses in it’s deployment) requires dropping the sonarDB first, it should be mentioned in the Sonar backup docs that this is not possible for a helm deployment as the only user we have access to does not have permissions to do so.
As I mentioned in my response on the other thread, please let me know if there’s any support we can get for upgrading properly or restoring to our original instance.
Hello @pengcakes, thanks a lot for taking the time to participate in the community.
Regarding the backup and restore part, we tried replicating your issue locally, and we ended up in the same situation when the database was already populated.
What we did locally with docker-compose that mimick the helm setup.
After that we have access to the SQ instance still with the previous data.
Regarding your workflow we end up with the same error when we restore a second time, or if SQ was running before the restore.
On the initial topic i see in the log you have elasticsearch: 143 error, that usually mean there is a memory issue, could you try increasing the memory of your Pods ( at the resource level ) and also add that variable SONAR_SEARCH_JAVAOPTS with an increased -xmx memory value.
For details please follow the official documentation here with some details in that existing post
Please let me know if you need some more information.
Jérémy.
Hi Jeremy,
Thanks for replicating this locally. We were able to find a way around this issue with a similar solution to the one you provided. Unfortunately, we ran into project and issue loading problems in 9.9. We’re currently looking at reverting to an earlier version and including the new memory options you provided to fix this.
Thanks,
Austin
Hello @pengcakes, i am happy to ear that you found a workaround.
Have you been able to test the memory increase ?