SonarQube server keeps crashing a few minutes after start/restart

hi @cwata thanks for your kind suggestion. unfortunately, i do not use any (extra) plugin at all since I am new to SonarQube as well. The data folder is created by SonarQube server itself and all default plugins are downloaded/installed by SonarQube server itself. I do not interfere with that process.

I deleted the data folder many times and the issue remains the same :frowning:

I checked firewall and pretty sure SonarQube (java.exe and URL) is allowed

Hi @nick.pham.hcl

I’ve tried to reproduce by downloading your version and running locally. My OS is Windows 10 Enterprise version 20H2. I can start SonarQube without a problem by running ./StartSonar.bat in a terminal window from the windows bin folder.

Did you perhaps install SonarQube as a Service at some point from the same location? I suppose the service might prevent you from being able to delete the installation path, but I’m at a loss for what other process would be accessing the content.

hi @cwata I am not sure why the latest SonarQube server developer version might have spawned a few instances (of itself?) since there are quite a number (> 2 that was normal before) of java.exe instances that are running on our Windows server. maybe they compete with each other?? we have no other running process or software that requires Java

I have newer masked logs here and if SonarQube/SonarSource Support can reply to me in time, it would be greatly appreciated please, since it has been a few working days already.

2022.05.27 15:30:48 ERROR es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube[masked-here]\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.1.jar:7.17.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube[masked-here]\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:328) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:429) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.1.jar:7.17.1]
... 6 more
2022.05.27 15:20:51 INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceFilter@5a184013 [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/components/update_key, ...]}]
2022.05.27 15:20:51 INFO web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9999
2022.05.27 15:20:51 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.IllegalStateException: org.springframework.context.annotation.AnnotationConfigApplicationContext@7b59a53a has been closed already
at org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1138)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1171)
at org.sonar.core.platform.SpringComponentContainer.getOptionalComponentByType(SpringComponentContainer.java:145)
at org.sonar.server.platform.platformlevel.PlatformLevel.getOptional(PlatformLevel.java:97)
at org.sonar.server.platform.platformlevel.PlatformLevel.lambda$getWebServer$0(PlatformLevel.java:149)
at java.base/java.util.Optional.flatMap(Optional.java:294)
at org.sonar.server.platform.platformlevel.PlatformLevel.getWebServer(PlatformLevel.java:149)
at org.sonar.server.platform.platformlevel.PlatformLevel4.configureLevel(PlatformLevel4.java:511)
at org.sonar.server.platform.platformlevel.PlatformLevel.configure(PlatformLevel.java:70)
at org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:193)
at org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:177)
at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:105)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:344)
at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:105)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:328)

Hi @nick.pham.hcl,

Thanks for some newer logs! Sounds like an ElasticSearch problem. Have you seen this post? I know you mentioned in the first original post that you deleted the data/web folder, but did you try removing the data/es7 folder?

Good luck! Sounds like progress in the right direction :+1:

hi @cwata thank you so much for your kind suggestion. i have deleted the data folder and started the SonarQube server again. it created the data folder again itself but still keeps failing

2022.05.30 10:10:57 INFO  web[][o.s.p.ProcessEntryPoint] Starting Web Server
2022.05.30 10:11:01 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start Web Server
java.lang.IllegalStateException: Fail to create or clean-up directory F:\Web\sonarqube.healthcarelogic.com\data\web\deploy
	at org.sonar.server.app.TomcatContexts.addStaticDir(TomcatContexts.java:84)
	at org.sonar.server.app.TomcatContexts.configure(TomcatContexts.java:59)
	at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:67)
	at org.sonar.server.app.WebServer.start(WebServer.java:55)
	at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:97)
	at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:81)
	at org.sonar.server.app.WebServer.main(WebServer.java:104)
Caused by: java.nio.file.FileSystemException: F:\Web\sonarqube.healthcarelogic.com\data\web\deploy\plugins\abap\sonar-abap-plugin-3.10.0.3628.jar: The process cannot access the file because it is being used by another process.

	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
	at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:274)
	at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
	at java.base/java.nio.file.Files.delete(Files.java:1142)
	at org.sonar.core.util.FileUtils$DeleteRecursivelyFileVisitor.visitFile(FileUtils.java:227)
	at org.sonar.core.util.FileUtils$DeleteRecursivelyFileVisitor.visitFile(FileUtils.java:222)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
	at org.sonar.core.util.FileUtils.deleteDirectoryImpl(FileUtils.java:162)
	at org.sonar.core.util.FileUtils$CleanDirectoryFileVisitor.visitFile(FileUtils.java:206)
	at org.sonar.core.util.FileUtils$CleanDirectoryFileVisitor.visitFile(FileUtils.java:192)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
	at org.sonar.core.util.FileUtils.cleanDirectoryImpl(FileUtils.java:149)
	at org.sonar.core.util.FileUtils.cleanDirectory(FileUtils.java:89)
	at org.sonar.server.app.TomcatContexts$Fs.createOrCleanupDir(TomcatContexts.java:123)
	at org.sonar.server.app.TomcatContexts.addStaticDir(TomcatContexts.java:82)
	... 6 common frames omitted
2022.05.30 10:11:01 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2022.05.30 10:11:01 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2022.05.30 10:11:01 INFO  web[][o.s.s.n.NotificationDaemon] Notification service stopped
2022.05.30 10:11:01 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [sonarqube] appears to have started a thread named [OkHttp TaskRunner] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.15/java.lang.Object.wait(Native Method)\n java.base@11.0.15/java.lang.Object.wait(Object.java:462)\n app//okhttp3.internal.concurrent.TaskRunner$RealBackend.coordinatorWait(TaskRunner.kt:294)\n app//okhttp3.internal.concurrent.TaskRunner.awaitTaskToRun(TaskRunner.kt:218)\n app//okhttp3.internal.concurrent.TaskRunner$runnable$1.run(TaskRunner.kt:59)\n java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.15/java.lang.Thread.run(Thread.java:834)

Hi @nick.pham.hcl

That’s unfortunate! Perhaps you can investigate how to track down which processes are currently accessing this .jar file. On Windows I see it’s possible by searching for associated handles.

If you can track down the executable that has a lock on this file, I think you’ll be on your way. Good luck!!! :+1:

@ganncamp could you please kindly help us with this issue?

i found more log

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube.healthcarelogic.com\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])

and i think you responded in Failed to obtain node locks

your help is greatly appreciated

thank you

@ganncamp please note that I cannot keep deleting the data folder (or write.lock) every 2 days. sonarqube keeps crashing every 2 days and throws the error above :frowning:

Hi,

If I understand well, the issue is not this any more.

Now you have a new one.

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube.healthcarelogic.com\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])

Can you confirm?

1 Like

thanks for getting back to me @Fan_Yang

I do not know why the issue The process cannot access the file because it is being used by another process. occurred since i have only 1 sonarqube instance. Fortunately, yes this is not happening for a few days.

And yes the issue failed to obtain node locks ... is still happening

Your kind recommendation is greatly appreciated.

Sorry for the delay.

Please run an Elasticsearch reindex procedure, as mentioned in our troubleshooting documentation:

How do I trigger a full ElasticSearch reindex?

  1. Stop SonarQube
  2. Delete the data/es7 directory
  3. Restart SonarQube

@Fan_Yang thanks for your suggestion. however, if you have read my comments from the beginning of this thread, I have done all 3 steps you advised me to do 5 times. Nothing has changed. SonarQube keeps crashing.

Hi @nick.pham.hcl

It’s interesting that SonarQube crashes every 2 days. Have you noticed a pattern with this? Perhaps your system has scheduled activities every 2 days that trigger this problem?

Otherwise, did you happen to check out the earlier link to try searching for any locks that exist on the file?

It sounds to me like something on your machine is starting another instance of SonarQube every X days. I could imagine that if this is really a periodic occurrence, then a system scheduled event (such as a reboot, maybe?) is failing to stop SonarQube correctly, thus the lock is not released and the newly started instance cannot obtain a lock. Just a guess!

At the end, you can also try increasing the max_local_storage_nodes so that whatever this other process is can obtain a lock. If then in another two days you get this problem again, then you can be pretty certain that your system is somehow starting new instances of SonarQube.

As always, good luck! We’ll figure this out!

1 Like

hi @cwata thank you for your kind suggestion. i have disabled all task schedulers and ensured only 1 instance of sonarqube is running. lets see if it runs after a few days.

i am keen to hear suggestions from @Fan_Yang and SonarSourcer as well please

thank you

Hi,

What @cwata said makes sense to me. This is unusual behavior.
Please keep monitoring and let us know if the issue persists.

Hi @nick.pham.hcl,

Just curious how it’s going! Did you find the root of the problem?

Cheers!

hi @cwata,

it turns out the Task Scheduler that I used to start the SonarQube on Windows start/restart caused this issue.

It is very weird that I had been using the same Task Scheduler for a year with no issue. I only got all errors reported in this thread on SonarQube developer version 9.4. Older versions are fine for a year :frowning:

I have disabled the Task Scheduler and will have to start the SonarQube manually for now.

I hope @Fan_Yang could please give me some advice as to why SonarQube developer version 9.4 throws errors and how to start SonarQube automatically without running into issues.

thanks

Hey @nick.pham.hcl,

could you also mark your last reply as the solution? It could help others with the same problem as you to find the solution :slight_smile:

Glad that you found what was starting your instances. Looking forward to hearing what the underlying issue is with the newest version. Til then, I think that you can configure the exact behavior of the task scheduler. So perhaps you can add a condition that the server is not already running before starting another instance? Not sure whether custom conditions are possible, but Google can help out there.

Good luck!

1 Like

Hi @nick.pham.hcl,

First of all, I’m glad to see that you are able to resolve the issue in your local environment.

Why are you trying to restart SonarQube regularly?

I think it’s worth continuing to find the root cause of this issue. Could you let me know which version you were using before 9.4?

Could you reproduce the issue, zip your $SONARQUBE_HOME/logs/ directory and share it with me via private message?

thanks @cwata

hi @Fan_Yang I do not want to start/restart the SonarQube regularly. As mentioned earlier, I wish to start the SonarQube server when my Windows DevOps VM starts/restarts. Since this is a DevOps VM, as similar as other VMs, sometime it needs patches or we need to install more dependencies/agents, and I want SonarQube to start immediately after Windows start.

this approach had worked for a year on older versions of SonarQube Developer. I am not sure why it fails to support this kind of Task Scheduler anymore

your further help is greatly appreciated @Fan_Yang

thank you both