SonarQube failing on Kubernetes EKS with persisten storage and Rolling update deploymentStrategy

Hello there,

I’m currently running sonarqube 8.2-developer with persistent storage enabled on AWS EKS (Kubernetes) using https://github.com/Oteemo/charts/tree/master/charts/sonarqube chart. I’m able to install and run SonarQube, however when I perform a Rolling update the new pod scheduled to go up and replace the old fails with

maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1]

However, I managed to overcome the issue by changing the deploymentStrategy to Recreate but this unacceptable because would create downtime and defeats the purpose of a graceful upgrade and rollback.

Best regards,

Phillip

This issue has already been reported on below topic but got into the dark.


k logs -p sonarqube-staging-sonarqube-799f945695-dbrvn
2020.05.06 21:35:58 INFO app[o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2020.05.06 21:35:58 INFO app[o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2020.05.06 21:35:58 INFO app[o.s.a.ProcessLauncherImpl] Launch process[[key=‘es’, ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2020.05.06 21:35:58 INFO app[o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2020.05.06 21:35:58 INFO app[o.e.p.PluginsService] no modules loaded
2020.05.06 21:35:58 INFO app[o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2020.05.06 21:36:00 WARN es[o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.4.jar:6.8.4]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:300) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]
… 6 more
2020.05.06 21:36:00 WARN app[o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
2020.05.06 21:36:00 INFO app[o.s.a.SchedulerImpl] Process[es] is stopped
2020.05.06 21:36:00 INFO app[o.s.a.SchedulerImpl] SonarQube is stopped

Hi,

We offer official Docker images now, but don’t support kubernetes. Perhaps you should ask the folks who offered the chart?

 
:woman_shrugging:
Ann

Hi Ann,

thanks for you reply. The issue that I’m experiencing is the same reported in the below topic. The chart works as expected, however sonar fails due the lock on ES.

Hi,

ES will lock up if it thinks there’s too little free disk space. Freeing-up / providing disk space and restarting SonarQube with flip the locks open. The es.log might help in understanding how far off the mark you are.

 
Ann

Hi Ann,

Seem that the issue is not related to the space on the storage as I’m using AWS EFS which has unlimited storage. The issue looks like that Sonar in not capable to to a rolling update due the node limitation.

maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

This is really a show stopper which prevents to do seamless Kubernetes deployments.

Kind regards,

Phillip

2020.05.09 10:58:13 WARN es[o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.4.jar:6.8.4]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:300) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]

Hi Ann,

so we spotted the problem on Sonar after a discussion on GitHub. So basically sonar is not capable to run more than one instance at the same time with the developer license. However, with Kubernetes threre’s the concept od Rolling update which allow to have a zero downtime upgrade and that will require to have a new instance up and running before evicting the old one. This is a big showstopper because makes impractical to use Sonar in the cloud with a contairer orchestrator. Yes, there’s the datacenter edition but not all companies can afford that, moreover not even the enterprise version which doesn’t come cheap is capable to run on Kubernetes.

Kind regards,

Phillip

After a deep investigation on the matter at this point in time a stetefulset is not a viable solution.

Phillip

It looks like the Oteemo charts use a deployment and give you the option of changing the deployment strategy with the deploymentStrategy helm variable.
sonarqube 9.4.1 · oteemo/oteemo-charts (artifacthub.io)

1 Like