SonarQube Upgrade to 10.3 - Fail to execute es requestES refresh request on indices 'components'

We recently upgraded from SonarQube 10.1 to 10.3 and since then, we sometimes get the exception:

Fail to execute es requestES refresh request on indices 'components'

Checking the es.log it reveals:

java.nio.file.FileSystemException: D:\SonarQube\data\es8\indices\1lPF1vGZTymjIlqJk2JY5w\0\index\_0_3.fnm: The process cannot access the file because it is being used by another process
	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) ~[?:?]
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) ~[?:?]
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) ~[?:?]
	at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:298) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:357) ~[?:?]
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:124) ~[elasticsearch-8.11.0.jar:?]
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.codecs.lucene94.Lucene94FieldInfosFormat.read(Lucene94FieldInfosFormat.java:130) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:206) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:157) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:229) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:536) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:598) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.11.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:454) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[elasticsearch-8.11.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2033) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2004) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.Engine.lambda$externalRefresh$8(Engine.java:1114) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:299) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.Engine.externalRefresh(Engine.java:1112) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.externalRefresh(IndexShard.java:1247) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:89) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:37) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1142) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:126) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:511) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:416) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$27(IndexShard.java:3405) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.innerAcquire(IndexShardOperationPermits.java:254) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:202) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:3376) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1079) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:413) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:357) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:1020) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

Restarting the SonarQube Windows task temporarly solves the problem, but the build after that is not working again.

This did not happen in 10.1 - Any ideas?

By the way, we also followed the following suggestion:

(Besides the fact, that it es already es8 and not es7)

It keeps coming again and again. We actually have two instances, one community edition, the other one is a payed developer edition

Hi,

You have some other process on the machine that’s interfering with SonarQube’s files. Perhaps Windows Defender or some other “helpful” process or virus scan.

 
HTH,
Ann

We did not change the server configuration, other than the SonarQube version.

I might add: It helps to stop the SonarQube Windows Service, clear all the es8 files and start the service again. So SonarQube is blocking itself (and not a Windows Defender process or something like that)

Hey there.

Are these running on the same machine?

No, these are separate VMs with their own independent database.

Still frustrating situation as we need to kill the service sometimes five times a day.
Therefore I created a workaround for our Developers, so they can kill & reset the SonarQube service via a web application, feel free (at your own risk) to implement it, as long this problem exists

public static void ResetServiceAndDeleteFolder(string serviceName, string folderPath)
{
  ServiceController service = new ServiceController(serviceName);

  // Stop the service
  service.Stop();
  service.WaitForStatus(ServiceControllerStatus.Stopped);

  // Delete the folder and its content
  Directory.Delete(folderPath, true);

  // Start the service again
  service.Start();
  service.WaitForStatus(ServiceControllerStatus.Running);
}

servicename: SonarQube
folderPath example: D:\SonarQube\data\es8

The next time it happens, it might help to try and find which process is holding the file – in the example you gave, D:\SonarQube\data\es8\indices\1lPF1vGZTymjIlqJk2JY5w\0\index_0_3.fnm.


So far, everytime this issue happens and I am checking, the file isn’t locked anymore.