Sonarqube is failing to start due to ES

SonarQube Server / Community Build
,
1

I seem to have an issue with sonarqube failing to start after the pod was restarted in my kubernetes cluster and trying to fix it without having to start from scratch.
In the past I was able to nuke my pod and the helm deployment would kick in and just recreate the pod and attach to their persistent pvc’s

Sonarqube Community 8.5.1
Sonarqube Developer 8.5.1

This instance was created several months ago managed via terraform / helm

resource "helm_release" "sonarqube" {
      name                = "sonarqube"
      repository          = "https://oteemo.github.io/charts"
      chart               = "sonarqube"
      timeout             = 1800
      force_update        = "true"
      recreate_pods       = "true"
      version             = "9.5.1"

      set {
        name  = "image.tag"
        value = "developer"
      }
      set {
        name  = "persistence.enabled"
        value = "true"
      }
      set {
        name  = "service.type"
        value = "LoadBalancer"
      }
      set {
        name  = "service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-resource-group"
        value = "network"
      }
      set {
        name  = "service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
        value = "true"
      }
    }

Recently there was an issue with the pod restarting and it seems there’s a elastic version downgrade?
Is that even possible if I’m targetting the sonar tag 9.5.1? this wasn’t changed recently right?

2021.05.29 19:57:34 INFO  es[][o.e.e.NodeEnvironment] heap size [503.6mb], compressed ordinary object pointers [true]
2021.05.29 19:57:34 ERROR es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: cannot downgrade a node from version [7.12.1] to version [7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
Caused by: java.lang.IllegalStateException: cannot downgrade a node from version [7.12.1] to version [7.11.2]
        at org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:83) ~[elasticsearch-7.11.2.jar:7.11.2]     
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:423) ~[elasticsearch-7.11.2.jar:7.11.2]     
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:320) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:351) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
        ... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: cannot downgrade a node from version [7.12.1] to version [7.11.2]
        at org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:83)
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:423)
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:320)
        at org.elasticsearch.node.Node.<init>(Node.java:351)
        at org.elasticsearch.node.Node.<init>(Node.java:278)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
        at org.elasticsearch.cli.Command.main(Command.java:79)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
For complete error details, refer to the log at /opt/sonarqube/logs/sonarqube.log
2021.05.29 19:57:34 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
2021.05.29 19:57:34 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2021.05.29 19:57:34 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

if anyone has any idea, it would be great if you can point me in the right direction.

Thanks
-Harri

2

Hi @HarriK,

It looks like you are jumping in the SQ version of your deployment. The es version can not change in the same version of sonarqube with just the edition changing. Can you double check your previous version as well as the version that is failing?

The observed error can be fixed with cleaning the es directory in the pv, but it indicates that the version of sq has also been changed and that can not be downgraded

3

I haven’t made any configuration changes that would change the SQ version which is weird. I’m also pinning the versions of SQ Helm version so that really shouldn’t be an issue unless that portion isn’t working well. I currently still have access to the current “upgraded” db, is there any indicator of what SQ version I should be using from that? After cleaning the PV I see the following issue now:

2021.05.30 16:21:32 INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
2021.05.30 16:21:32 INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
2021.05.30 16:21:33 ERROR web[][o.s.s.p.PlatformImpl] Web server startup failed: Database was upgraded to a more recent version of SonarQube. A backup must probably be restored or the DB settings are incorrect.
2021.05.30 16:21:33 INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2021.05.30 16:21:33 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2021.05.30 16:21:33 INFO  app[][o.s.a.SchedulerImpl] Process[web] is stopped
2021.05.30 16:21:33 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2021.05.30 16:21:33 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

Thanks

-Harri

4

I can just guess that you were running on latest as es was updated to the version you were mentioning with https://jira.sonarsource.com/browse/SONAR-14690 so released with sq 8.9

1 Like
5

oh gosh … so what really happened was the kubernetes upgrade from 1.17.xx to 1.18.xx. We were trailing behind with v1.18.xx
Apparently there are certain helm/sonar prereqs kube version for the sonar install which kind of screwed things up.
I didn’t dig too deep but there might be a mismatch in the versions where it did upgrade our version based on the detected version of kubernetes.
There might be a discrepancy in the allowed sonarqube version after the pod restart as it was trying to load 8.8 whereas it had us upgrade to 8.9 in the previous restart.
if that makes sense?

Solution was to push the kube version to 1.19.xx from our 1.18.xx, restart the pod, let sonar do it’s thing.

Lesson learned… keep nightly db backups anyway for peace of mind…

Thanks for the feedback, glad this community is still active.

-Harri

3 Likes