Blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

SonarQube Server / Community Build
1

All of a sudden sonarqube will not come up. getting this error from elastic

2021.08.05 19:40:47 INFO  web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
2021.08.05 19:40:48 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
        at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:183)
        at org.elasticsearch.action.support.replication.TransportReplicationAction.blockExceptions(TransportReplicationAction.java:255)
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:100)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:780)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:172)
        at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:100)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:420)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:533)
        at org.elasticsearch.action.bulk.TransportBulkAction.executeIngestAndBulk(TransportBulkAction.java:271)
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:222)

any idea?

2

Hi @jelbatn ,

this looks like the information you are looking for are in a file called es.log. with this error i assume that there might not be enough storage available on your system so the bundled ES will mark all indexes as read only.

2 Likes
3

Once you made some more space available on your disk, you can restart SonarQube: It will switch the indexes as write-read again at startup.

4

Thank you for your response
I did have disk space issue which was cleared I have about 66GB available now but still not coming up.

this is what I see in the es.log as it shows net usable_space [66.7gb]

2021.08.06 09:27:45 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [FtRPkD05TgGDM-sSXcx8KA]
2021.08.06 09:27:45 INFO  es[][o.e.n.Node] version[6.6.2], pid[23], build[default/tar/3bd3e59/2019-03-06T15:16:26.864148Z], OS[Linux/3.10.0-1160.36.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b01]
2021.08.06 09:27:45 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/opt/sonarqube/temp/es6, -XX:ErrorFile=/opt/sonarqube/logs/es_hs_err_pid%p.log, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/elasticsearch, -Des.path.conf=/opt/sonarqube/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [percolator]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2021.08.06 09:27:45 INFO  es[][o.e.p.PluginsService] no plugins loaded
2021.08.06 09:27:47 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2021.08.06 09:27:48 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2021.08.06 09:27:48 INFO  es[][o.e.n.Node] initialized
2021.08.06 09:27:48 INFO  es[][o.e.n.Node] starting ...
2021.08.06 09:27:48 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2021.08.06 09:27:51 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{xYR6IdK4SjKAO6bEa_d8ag}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2021.08.06 09:27:51 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{xYR6IdK4SjKAO6bEa_d8ag}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{xYR6IdK4SjKAO6bEa_d8ag}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2021.08.06 09:27:51 INFO  es[][o.e.n.Node] started
2021.08.06 09:27:52 INFO  es[][o.e.g.GatewayService] recovered [6] indices into cluster_state
2021.08.06 09:28:14 INFO  es[][o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[metadatas][0]] ...]).
2021.08.06 09:28:22 INFO  es[][o.e.n.Node] stopping ...
2021.08.06 09:28:22 INFO  es[][o.e.n.Node] stopped
2021.08.06 09:28:22 INFO  es[][o.e.n.Node] closing ...
2021.08.06 09:28:22 INFO  es[][o.e.n.Node] closed
2021.08.06 09:28:26 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/opt/sonarqube/data (/dev/sdb1)]], net usable_space [66.7gb], net total_space [149.9gb], types [xfs]
2021.08.06 09:28:26 INFO  es[][o.e.e.NodeEnvironment] heap size [494.9mb], compressed ordinary object pointers [true]
2021.08.06 09:28:26 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [FtRPkD05TgGDM-sSXcx8KA]
2021.08.06 09:28:26 INFO  es[][o.e.n.Node] version[6.6.2], pid[22], build[default/tar/3bd3e59/2019-03-06T15:16:26.864148Z], OS[Linux/3.10.0-1160.36.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b01]
2021.08.06 09:28:26 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/opt/sonarqube/temp/es6, -XX:ErrorFile=/opt/sonarqube/logs/es_hs_err_pid%p.log, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/elasticsearch, -Des.path.conf=/opt/sonarqube/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [percolator]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2021.08.06 09:28:27 INFO  es[][o.e.p.PluginsService] no plugins loaded
2021.08.06 09:28:28 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2021.08.06 09:28:29 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2021.08.06 09:28:29 INFO  es[][o.e.n.Node] initialized
2021.08.06 09:28:30 INFO  es[][o.e.n.Node] starting ...
2021.08.06 09:28:30 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2021.08.06 09:28:33 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{2z4LQicGROyymdh9yFvIiA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2021.08.06 09:28:33 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{2z4LQicGROyymdh9yFvIiA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{2z4LQicGROyymdh9yFvIiA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2021.08.06 09:28:33 INFO  es[][o.e.n.Node] started
2021.08.06 09:28:33 INFO  es[][o.e.g.GatewayService] recovered [6] indices into cluster_state
2021.08.06 09:28:53 INFO  es[][o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[metadatas][0]] ...]).
2021.08.06 09:29:00 INFO  es[][o.e.n.Node] stopping ...
2021.08.06 09:29:01 INFO  es[][o.e.n.Node] stopped
2021.08.06 09:29:01 INFO  es[][o.e.n.Node] closing ...
2021.08.06 09:29:01 INFO  es[][o.e.n.Node] closed
2021.08.06 09:29:04 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/opt/sonarqube/data (/dev/sdb1)]], net usable_space [66.7gb], net total_space [149.9gb], types [xfs]
2021.08.06 09:29:04 INFO  es[][o.e.e.NodeEnvironment] heap size [494.9mb], compressed ordinary object pointers [true]
2021.08.06 09:29:04 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [FtRPkD05TgGDM-sSXcx8KA]
2021.08.06 09:29:04 INFO  es[][o.e.n.Node] version[6.6.2], pid[23], build[default/tar/3bd3e59/2019-03-06T15:16:26.864148Z], OS[Linux/3.10.0-1160.36.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b01]
2021.08.06 09:29:04 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/opt/sonarqube/temp/es6, -XX:ErrorFile=/opt/sonarqube/logs/es_hs_err_pid%p.log, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/elasticsearch, -Des.path.conf=/opt/sonarqube/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [percolator]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2021.08.06 09:29:05 INFO  es[][o.e.p.PluginsService] no plugins loaded
2021.08.06 09:29:07 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2021.08.06 09:29:08 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2021.08.06 09:29:08 INFO  es[][o.e.n.Node] initialized
2021.08.06 09:29:08 INFO  es[][o.e.n.Node] starting ...
2021.08.06 09:29:08 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2021.08.06 09:29:11 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{6iszYuHIR0SYJ-M-h-XrUw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2021.08.06 09:29:11 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{6iszYuHIR0SYJ-M-h-XrUw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{FtRPkD05TgGDM-sSXcx8KA}{6iszYuHIR0SYJ-M-h-XrUw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2021.08.06 09:29:11 INFO  es[][o.e.n.Node] started
2021.08.06 09:29:12 WARN  es[][o.e.g.G.InternalPrimaryShardAllocator] [rules][1]: failed to list shard for shard_started on node [FtRPkD05TgGDM-sSXcx8KA]
org.elasticsearch.action.FailedNodeException: Failed node [FtRPkD05TgGDM-sSXcx8KA]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:236) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:151) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:210) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:698) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:744) [elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-6.6.2.jar:6.6.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube][127.0.0.1:9001][internal:gateway/local/started_shards[n]]
Caused by: org.elasticsearch.ElasticsearchException: failed to load started shards
        at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:169) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:687) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) ~[elasticsearch-6.6.2.jar:6.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.6.2.jar:6.6.2]
        ... 3 more
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:23189, file:/opt/sonarqube/data/es6/nodes/0/indices/_OlCfFd6ToiIJRo7k5U4JQ/_state/state-23189.st]
5

Looks like your elasticsearch indexes are corrupted. Please have a look at the troubleshooting documentation and follow the procedure to rebuild your indexes.