Sonarqube in multi pod deployment setup on Kubernetes Platform

Hi,
I am in process to setup sonarqube(Version: 7.9.6) on Kubernetes platform.
I have setup deployment with replicas=2 so that 2 pods would be created for High availability.
I have setup PVC with access mode “ReadWriteMany” so that PVC can be mounted on multiple pods with read-write permission.

As I have checked, Sonarqube application is up and UI is accessible.

It is observed that only 1 pod (out of 2) is in “Running” state whereas other pod is with status “CrashLoopBackOff”.

As per error logs, it seems, SonarQube is not able to write into /opt/sonarqube/data directory (which is one of PVC) for ElasticSearch configuration.
Please note that I have verified that PVC is getting mounted to multiple pods with test setup earlier.

Please find error logs as below from failing pod:

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.0.jar:6.8.0]

... 6 more

13:13:03.102 [main] WARN org.sonar.application.config.AppSettingsLoaderImpl - Configuration file not found: /opt/sonarqube/conf/sonar.properties

2021.05.19 13:13:03 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp

2021.05.19 13:13:03 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001

2021.05.19 13:13:03 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch

2021.05.19 13:13:03 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

2021.05.19 13:13:03 INFO app[][o.e.p.PluginsService] no modules loaded

2021.05.19 13:13:03 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]

2021.05.19 13:13:05 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1

2021.05.19 13:13:05 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped

2021.05.19 13:13:05 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped

2021.05.19 13:13:05 ERROR es[][o.e.b.Bootstrap] Exception

java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.0.jar:6.8.0]

at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.0.jar:6.8.0]

2021.05.19 13:13:05 WARN es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.0]

at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.0.jar:6.8.0]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.0.jar:6.8.0]

... 6 more

As I googled, I found removing es6 directory would help. I already tried but its not working.

Can anyone please help to resolve this issue ?

Thanks,
Vaibhav Jariwala

Hey there.

It is not possible to use multiple nodes – only a single node can be connected to your SonarQube database at once.

Running SonarQube as a cluster (for high availability) is only supported with the Data Center Edition of SonarQube, currently not supported for Kubernetes.

Take note that in SonarQube v8.9 LTS we introduced a helm chart and guidelines for operating SonarQube on Kubernetes.

Ok Thanks @Colin_SonarSource for letting us know.

Can you please clarify below queries as well:

  1. Is Sonarqube as a cluster (for high availability) available for LTS 7.9.6 with Data Center Edition of SonarQube version ?
  2. Is Sonarqube as a cluster (for high availability) available for LTS 7.9.6 on Kubernetes platform Data Center Edition of SonarQube?

Hey @Colin_SonarSource /Team,
May I get update on posted query ?
Thanks.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.