SonarQube failing on Kubernetes EKS with persisten storage and Rolling update deploymentStrategy

[p0bailey (Phillip Bailey)]

Hello there,

I’m currently running sonarqube 8.2-developer with persistent storage enabled on AWS EKS (Kubernetes) using https://github.com/Oteemo/charts/tree/master/charts/sonarqube chart. I’m able to install and run SonarQube, however when I perform a Rolling update the new pod scheduled to go up and replace the old fails with

maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1]

However, I managed to overcome the issue by changing the deploymentStrategy to Recreate but this unacceptable because would create downtime and defeats the purpose of a graceful upgrade and rollback.

Best regards,

Phillip

This issue has already been reported on below topic but got into the dark.

Kubernetes rolling deployment fails for SonarQube with persistent volumes

Hi Team, We are running SonarQube lts version 6.7.1-alpine in k8s cluster(deployed as k8s deployment object with replica count 1) and sonarqube data/extensions dirs are backed by k8s persistent volumes to have persistency for these two them. But we are facing issue while doing the k8s rolling deployments, where existing sonar pod is still running and a parallel gets started which gets failed to start, as elasticsearch does not get the node lock(already taken by existing as both are using the sa…

---

k logs -p sonarqube-staging-sonarqube-799f945695-dbrvn
2020.05.06 21:35:58 INFO app[o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2020.05.06 21:35:58 INFO app[o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2020.05.06 21:35:58 INFO app[o.s.a.ProcessLauncherImpl] Launch process[[key=‘es’, ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2020.05.06 21:35:58 INFO app[o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2020.05.06 21:35:58 INFO app[o.e.p.PluginsService] no modules loaded
2020.05.06 21:35:58 INFO app[o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2020.05.06 21:36:00 WARN es[o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.4.jar:6.8.4]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:300) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]
… 6 more
2020.05.06 21:36:00 WARN app[o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
2020.05.06 21:36:00 INFO app[o.s.a.SchedulerImpl] Process[es] is stopped
2020.05.06 21:36:00 INFO app[o.s.a.SchedulerImpl] SonarQube is stopped

[ganncamp (G Ann Campbell)]

Hi,

We offer official Docker images now, but don’t support kubernetes. Perhaps you should ask the folks who offered the chart?

 

Ann

[p0bailey (Phillip Bailey)]

Hi Ann,

thanks for you reply. The issue that I’m experiencing is the same reported in the below topic. The chart works as expected, however sonar fails due the lock on ES.

Kubernetes rolling deployment fails for SonarQube with persistent volumes

Hi Team, We are running SonarQube lts version 6.7.1-alpine in k8s cluster(deployed as k8s deployment object with replica count 1) and sonarqube data/extensions dirs are backed by k8s persistent volumes to have persistency for these two them. But we are facing issue while doing the k8s rolling deployments, where existing sonar pod is still running and a parallel gets started which gets failed to start, as elasticsearch does not get the node lock(already taken by existing as both are using the sa…

[ganncamp (G Ann Campbell)]

Hi,

ES will lock up if it thinks there’s too little free disk space. Freeing-up / providing disk space and restarting SonarQube with flip the locks open. The es.log might help in understanding how far off the mark you are.

 
Ann

[p0bailey (Phillip Bailey)]

Hi Ann,

Seem that the issue is not related to the space on the storage as I’m using AWS EFS which has unlimited storage. The issue looks like that Sonar in not capable to to a rolling update due the node limitation.

maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

This is really a show stopper which prevents to do seamless Kubernetes deployments.

Kind regards,

Phillip

2020.05.09 10:58:13 WARN es[o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.4.jar:6.8.4]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:300) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]

[p0bailey (Phillip Bailey)]

Hi Ann,

so we spotted the problem on Sonar after a discussion on GitHub. So basically sonar is not capable to run more than one instance at the same time with the developer license. However, with Kubernetes threre’s the concept od Rolling update which allow to have a zero downtime upgrade and that will require to have a new instance up and running before evicting the old one. This is a big showstopper because makes impractical to use Sonar in the cloud with a contairer orchestrator. Yes, there’s the datacenter edition but not all companies can afford that, moreover not even the enterprise version which doesn’t come cheap is capable to run on Kubernetes.

Kind regards,

Phillip

[Sonarqube statefulset] Provide option to switch between statefulset vs deployment for sonarqube.

[p0bailey (Phillip Bailey)]

After a deep investigation on the matter at this point in time a stetefulset is not a viable solution.

Phillip

[jgallucci32 (John Gallucci)]

It looks like the Oteemo charts use a deployment and give you the option of changing the deployment strategy with the deploymentStrategy helm variable.
sonarqube 9.4.1 · oteemo/oteemo-charts (artifacthub.io)