SonarQube Upgrade to 10.3 - Fail to execute es requestES refresh request on indices 'components'

SonarQube Server / Community Build
,
1

We recently upgraded from SonarQube 10.1 to 10.3 and since then, we sometimes get the exception:

Fail to execute es requestES refresh request on indices 'components'

Checking the es.log it reveals:

java.nio.file.FileSystemException: D:\SonarQube\data\es8\indices\1lPF1vGZTymjIlqJk2JY5w\0\index\_0_3.fnm: The process cannot access the file because it is being used by another process
	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) ~[?:?]
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) ~[?:?]
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) ~[?:?]
	at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:298) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:357) ~[?:?]
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:124) ~[elasticsearch-8.11.0.jar:?]
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.codecs.lucene94.Lucene94FieldInfosFormat.read(Lucene94FieldInfosFormat.java:130) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:206) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:157) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:229) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:536) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:598) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.11.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:454) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434) ~[elasticsearch-8.11.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.8.0.jar:?]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.8.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2033) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2004) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.Engine.lambda$externalRefresh$8(Engine.java:1114) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:299) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.engine.Engine.externalRefresh(Engine.java:1112) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.externalRefresh(IndexShard.java:1247) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:89) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:37) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1142) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:126) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:511) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:416) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$27(IndexShard.java:3405) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.innerAcquire(IndexShardOperationPermits.java:254) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:202) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:3376) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1079) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:413) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:357) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:1020) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

Restarting the SonarQube Windows task temporarly solves the problem, but the build after that is not working again.

This did not happen in 10.1 - Any ideas?

2

By the way, we also followed the following suggestion:

(Besides the fact, that it es already es8 and not es7)

It keeps coming again and again. We actually have two instances, one community edition, the other one is a payed developer edition

3

Hi,

You have some other process on the machine that’s interfering with SonarQube’s files. Perhaps Windows Defender or some other “helpful” process or virus scan.

 
HTH,
Ann

Sonar Community Roundup, December 9 - December 15
4

We did not change the server configuration, other than the SonarQube version.

5

I might add: It helps to stop the SonarQube Windows Service, clear all the es8 files and start the service again. So SonarQube is blocking itself (and not a Windows Defender process or something like that)

6

Hey there.

Are these running on the same machine?

7

No, these are separate VMs with their own independent database.

8

Still frustrating situation as we need to kill the service sometimes five times a day.
Therefore I created a workaround for our Developers, so they can kill & reset the SonarQube service via a web application, feel free (at your own risk) to implement it, as long this problem exists

public static void ResetServiceAndDeleteFolder(string serviceName, string folderPath)
{
  ServiceController service = new ServiceController(serviceName);

  // Stop the service
  service.Stop();
  service.WaitForStatus(ServiceControllerStatus.Stopped);

  // Delete the folder and its content
  Directory.Delete(folderPath, true);

  // Start the service again
  service.Start();
  service.WaitForStatus(ServiceControllerStatus.Running);
}

servicename: SonarQube
folderPath example: D:\SonarQube\data\es8

9

The next time it happens, it might help to try and find which process is holding the file – in the example you gave, D:\SonarQube\data\es8\indices\1lPF1vGZTymjIlqJk2JY5w\0\index_0_3.fnm.

10

image
image1038×464 60.3 KB

So far, everytime this issue happens and I am checking, the file isn’t locked anymore.