Error: [.... ElasticsearchUncaugh] cannot downgrade a node from version [7.17.1] to version [7.16.2]

[ruchir_bharadwaj]

I am trying to migrate sonarQube deployed on one kubernetes cluster to other kubernetes cluster.
My source sonarQube version is 8.2-community and on destination I am tying to upgrade it to the lastest LTS version .i.e 8.9.8-community

The process I followed on very top level is

• Migrated PVC for sonarQube Extension and sonarQube data (postgress ) using a tool velero form source k8s to destination k8s

• Then used helm chart from sonar-qube offical helm chart repo to install the latest LTS chart version

helm upgrade --install -n sonarqube sonarqube sonarqube/sonarqube-lts --set persistence.enabled=true --set persistence.existingClaim=sonarqube-sonarqube --set postgresql.persistence.existingClaim=data-sonarqube-postgresql-0

PS: I have provided options in helm chart for it to use existing PVC

however when sonar qube is getting started its failing with below error

INFO  es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2022.07.11 15:01:37 ERROR es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: cannot downgrade a node from version [7.17.1] to version [7.16.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.2.jar:7.16.2]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.2.jar:7.16.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.2.jar:7.16.2]
Caused by: java.lang.IllegalStateException: cannot downgrade a node from version [7.17.1] to version [7.16.2]
	at org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:80) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:482) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:356) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.node.Node.<init>(Node.java:427) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.2.jar:7.16.2]
	... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: cannot downgrade a node from version [7.17.1] to version [7.16.2]
	at org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:80)
	at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:482)
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:356)
	at org.elasticsearch.node.Node.<init>(Node.java:427)
	at org.elasticsearch.node.Node.<init>(Node.java:309)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
	at org.elasticsearch.cli.Command.main(Command.java:77)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)

Any pointer to troubleshoot above ?

[Colin]

Hey there.

What version of kubernetes are you using?

Sonarqube is failing to start due to ES

oh gosh … so what really happened was the kubernetes upgrade from 1.17.xx to 1.18.xx. We were trailing behind with v1.18.xx Apparently there are certain helm/sonar prereqs kube version for the sonar install which kind of screwed things up. I didn’t dig too deep but there might be a mismatch in the versions where it did upgrade our version based on the detected version of kubernetes. There might be a discrepancy in the allowed sonarqube version after the pod restart as it was trying to load 8.…

[ruchir_bharadwaj]

@Sonarqube is failing to start due to ES

so what really happened was the kubernetes upgrade from 1.17.xx to 1.18.xx. We were trailing behind with v1.18.xx Apparently there are certain helm/sonar prereqs kube version for the sonar inst

its 1.21

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.11", GitCommit:"38d3c1f3d5306401bcf39a71bad3b5a5106033d7", GitTreeState:"clean", BuildDate:"2022-03-16T14:08:11Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:11:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

[Philip_Bradshaw]

@Colin we are also experiencing the same issue, moving from 8.9.3-developer to 9.0.0-developer.

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: cannot downgrade a node from version [7.13.4] to version [7.12.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.12.1.jar:7.12.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.12.1.jar:7.12.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.12.1.jar:7.12.1]
Caused by: java.lang.IllegalStateException: cannot downgrade a node from version [7.13.4] to version [7.12.1]
        at org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:83) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:423) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:320) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.node.Node.<init>(Node.java:352) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.12.1.jar:7.12.1]
        ... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: cannot downgrade a node from version [7.13.4] to version [7.12.1]
        at org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:83)
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:423)
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:320)
        at org.elasticsearch.node.Node.<init>(Node.java:352)
        at org.elasticsearch.node.Node.<init>(Node.java:278)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
        at org.elasticsearch.cli.Command.main(Command.java:79)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
For complete error details, refer to the log at /opt/sonarqube/logs/sonarqube.log```

[jacek.poreda]

Hello @ruchir_bharadwaj ,

Search nodes are Stateful Set, so the state of your previous installation will be reused for the new one. Have you tried to stop search nodes, dropping persistence volumes for those nodes, and start with the new version? I haven’t experienced that issue, but I think it could happen if you have upgraded Elasticsearch and you are trying to reuse the same data directory.

What’s the strategy for your upgrade? Do you put down all the nodes and then start new one or are you trying to have different versions of SonarQube running at the same time?