8.9 Upgrade Issues with project reloading

We tried upgrading from 7.9 to 8.9.2 LTS and it took 48 hours to reload all of our 6000+ projects to 100%
But the analysis never triggered after upgrade. We tried restarting our sonar cluster nodes and it took it back to 0% reloading again…and it never moves after that. CE logs not getting rotated and the status is stuck at 0% for more than 24 hours now with not much info in logs as well.
Should we delete the ES7 folder and retry? will it start reloading projects again and start moving?
Any help or suggestions here is appreciated please…

Hello, welcome to the community forum!

Could you please explain what was the issue?

What database are you using?

Yes, it would restart the reloading of projects. I don’t think you should do it right now, as you might end up in the same situation as before. Let’s understand first what happened the first time.

@pierreguillot Thanks for response.
After 2 days of upgrade, the project reloading got completed at 100%. But when we tested the analysis it never got triggered or sometimes never went past quality profile check. So after consulting sonar vendor support we were told to restart the cluster and try again. But when we restarted it took us back to 0% project reloading again and never ever proceeded beyond 0 since last 24 hours. We have tried so many other options but no luck. We upgraded from 7.9 (using postgres 9.6) to 8.9.2 (using postgres 13+). Since postgres 13 is not supported in 7.9 sonar we had to do RDS postgres upgrade and sonar 8.9.2 upgrade together. Hope this helps?

Thanks for the clarification. What kind of cluster are we discussing? docker-compose? Kubernetes?

Do you mean that you were able to run the scanner, but the background processing on the SonarQube Server never started?

we are using aws ec2. 3 search nodes and 2 app nodes.
Yes the processing never happened after upgrade

I have a really naive question: When you upgraded, did you make sure all /data/es7 were empty? You might had some leftovers that messed up the elasticsearch data.

If you want to try again to reindex your data, make sure you shut down all SQ app nodes, all SQ search nodes, wipe out the /data/es7 folders for every search node, and restart the cluster.

Once the reindexation is complete, if you have background tasks processing issues, please report here with the errors so we can help on that.

No we did not empty ES folders. We are trying now. will keep you posted