Must-share information (formatted with Markdown ):
which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension) - 8.9.10
how is SonarQube deployed: zip, Docker, Helm - it’s on my local pc and executed as command line
what are you trying to achieve - I execute sonar on my local to get the code coverage before I check in code in the code base. When I restart my laptop sonarqube is unable to restart once I log back in.es.log (23.2 KB)sonar.20230417.log (3.8 KB)
what have you tried so far to achieve this - My alternative is to reinstall that is cumbersome and time consuming option.
Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!
ganncamp
April 18, 2023, 4:18pm
2
Hi,
Welcome to the community!
I think this is relevant:
opened 08:40AM - 26 Feb 18 UTC
closed 01:30PM - 26 Feb 18 UTC
**Elasticsearch version** (`bin/elasticsearch --version`):
Version: 6.2.2-SNAPS… HOT, Build: d8602bc/2018-02-18T20:00:19.005352Z, JVM: 1.8.0_162
(because of #28713)
**Plugins installed**: []
analysis-icu
**JVM version** (`java -version`):
multiple, data nodes are:
openjdk version "1.8.0_144"
OpenJDK Runtime Environment (build 1.8.0_144-b01)
OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode)
**OS version** (`uname -a` if on a Unix-like system):
FreeBSD 11.1
**Description of the problem including expected versus actual behavior**:
Cluster of 40 data nodes (three dedicated masters and some client nodes), turns into yellow suddenly with the error log below.
The index in question was created with an older version:
"created" : "5040299"
Current segments:
https://gist.github.com/bra-fsn/d77e57dfce4799319a3ef54587621012
**Steps to reproduce**:
I'm not aware of anything special, it just happened in the morning. It didn't happened previously and since then.
We collect statistics from the cluster on the master (fmesm) nodes every 5 minutes. The logged event is very close to that, maybe they are related.
fmfe* are the data nodes.
**Provide logs (if relevant)**:
```
[2018-02-25T08:05:30,890][WARN ][o.e.c.a.s.ShardStateAction] [fmesm02] [messagebodies_540][100] received shard failed for shard id [[messagebodies_540][100]], allocation id [VxBivC1NT7q_c5uqUQZEkw], primary term [0], message [shard failure, reason [refresh failed source[schedule]]], failure [CorruptIndexException[file mismatch, expected id=cge03ew8t5un8p71dbjd9wmv6, got=cge03ew8t5un8p71dbjd9wmwr (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data/elasticsearch/fmprod.9300/data/nodes/0/indices/W8X0MiYUQ8WjJKSY3ilyBA/100/index/_z899_Lucene50_0.doc")))]]
org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=cge03ew8t5un8p71dbjd9wmv6, got=cge03ew8t5un8p71dbjd9wmwr (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data/elasticsearch/fmprod.9300/data/nodes/0/indices/W8X0MiYUQ8WjJKSY3ilyBA/100/index/_z899_Lucene50_0.doc")))
at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:295) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:96) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5010) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:538) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:491) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:556) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:672) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:300) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:275) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1401) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1383) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:861) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:698) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:97) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:902) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:812) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-02-25T08:05:30,950][INFO ][o.e.c.r.a.AllocationService] [fmesm02] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[messagebodies_540][100]] ...]).
[2018-02-25T08:05:37,900][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [fmesm02] [messagebodies_540][100]: failed to list shard for shard_store on node [ndI4wzhFR9-0mxy8UE99wQ]
org.elasticsearch.action.FailedNodeException: Failed node [ndI4wzhFR9-0mxy8UE99wQ]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1098) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.lambda$handleException$33(TcpTransport.java:1478) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) [elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1476) [elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1468) [elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1398) [elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) [transport-netty4-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: org.elasticsearch.transport.RemoteTransportException: [fmfe16][10.6.145.209:9300][internal:cluster/nodes/indices/shard/store[n]]
Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[messagebodies_540][100]]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_131]
... 1 more
Caused by: org.apache.lucene.index.CorruptIndexException: failed engine (reason: [refresh failed source[schedule]]) (resource=preexisting_corruption)
at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:627) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:458) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:150) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_131]
... 1 more
Caused by: java.io.IOException: failed engine (reason: [refresh failed source[schedule]])
at org.elasticsearch.index.engine.Engine.failEngine(Engine.java:898) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1415) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1383) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:861) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:698) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:97) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:902) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:812) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_131]
... 1 more
Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=cge03ew8t5un8p71dbjd9wmv6, got=cge03ew8t5un8p71dbjd9wmwr (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data/elasticsearch/fmprod.9300/data/nodes/0/indices/W8X0MiYUQ8WjJKSY3ilyBA/100/index/_z899_Lucene50_0.doc")))
at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:295) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:96) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5010) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:538) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:491) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:556) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:672) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:300) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:275) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1401) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1383) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:861) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:698) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:97) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:902) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:812) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) ~[elasticsearch-6.2.2-SNAPSHOT.jar:6.2.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_131]
... 1 more
[2018-02-25T08:19:07,148][INFO ][o.e.c.r.a.AllocationService] [fmesm02] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[messagebodies_540][100]] ...]).
```