SonarQube server keeps crashing a few minutes after start/restart

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension): latest 9.4.0.54424
  • what are you trying to achieve: get it up and running as normal. all my pipelines are failing because of this
  • what have you tried so far to achieve this: deleted folders, terminated all JAVA.exe, reinstalled, checked firewall, added SonarQube as allowed to firewall, checked logs, everything

the file sonar-abap-plugin-3.10.0.3628.jar is in [SONARQUBE-ROOT]\data\web\deploy\plugins folder that was/is created by SonarQube itself. I have been deleting this data\web folder and SonarQube keeps recreating it with required default plugins, but it still keeps crashing reporting the file was used by something else.

what is the “another process”? only SonarQube is using it.

I even terminate all JAVA.exe many times and run SonarQube again but no luck. Keeps crashing after less than 5 minutes

Hi Nick,

I’m relatively new to Sonar, so I don’t know this issue exactly. Any time I see the error message “The process cannot access the file” though, I think immediately of permissions. Might be worth checking which permissions your account has to the folder!

A related question: have you tried starting SonarQube without the plugin? If so, it could be the plugin is at fault. Then you could address this issue more directly.

Good luck!

1 Like

hi @cwata thanks for your kind suggestion. unfortunately, i do not use any (extra) plugin at all since I am new to SonarQube as well. The data folder is created by SonarQube server itself and all default plugins are downloaded/installed by SonarQube server itself. I do not interfere with that process.

I deleted the data folder many times and the issue remains the same :frowning:

I checked firewall and pretty sure SonarQube (java.exe and URL) is allowed

Hi @nick.pham.hcl

I’ve tried to reproduce by downloading your version and running locally. My OS is Windows 10 Enterprise version 20H2. I can start SonarQube without a problem by running ./StartSonar.bat in a terminal window from the windows bin folder.

Did you perhaps install SonarQube as a Service at some point from the same location? I suppose the service might prevent you from being able to delete the installation path, but I’m at a loss for what other process would be accessing the content.

hi @cwata I am not sure why the latest SonarQube server developer version might have spawned a few instances (of itself?) since there are quite a number (> 2 that was normal before) of java.exe instances that are running on our Windows server. maybe they compete with each other?? we have no other running process or software that requires Java

I have newer masked logs here and if SonarQube/SonarSource Support can reply to me in time, it would be greatly appreciated please, since it has been a few working days already.

2022.05.27 15:30:48 ERROR es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube[masked-here]\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.1.jar:7.17.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube[masked-here]\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:328) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:429) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.1.jar:7.17.1]
... 6 more
2022.05.27 15:20:51 INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceFilter@5a184013 [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/components/update_key, ...]}]
2022.05.27 15:20:51 INFO web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9999
2022.05.27 15:20:51 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.IllegalStateException: org.springframework.context.annotation.AnnotationConfigApplicationContext@7b59a53a has been closed already
at org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1138)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1171)
at org.sonar.core.platform.SpringComponentContainer.getOptionalComponentByType(SpringComponentContainer.java:145)
at org.sonar.server.platform.platformlevel.PlatformLevel.getOptional(PlatformLevel.java:97)
at org.sonar.server.platform.platformlevel.PlatformLevel.lambda$getWebServer$0(PlatformLevel.java:149)
at java.base/java.util.Optional.flatMap(Optional.java:294)
at org.sonar.server.platform.platformlevel.PlatformLevel.getWebServer(PlatformLevel.java:149)
at org.sonar.server.platform.platformlevel.PlatformLevel4.configureLevel(PlatformLevel4.java:511)
at org.sonar.server.platform.platformlevel.PlatformLevel.configure(PlatformLevel.java:70)
at org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:193)
at org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:177)
at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:105)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:344)
at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:105)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:328)

Hi @nick.pham.hcl,

Thanks for some newer logs! Sounds like an ElasticSearch problem. Have you seen this post? I know you mentioned in the first original post that you deleted the data/web folder, but did you try removing the data/es7 folder?

Good luck! Sounds like progress in the right direction :+1:

hi @cwata thank you so much for your kind suggestion. i have deleted the data folder and started the SonarQube server again. it created the data folder again itself but still keeps failing

2022.05.30 10:10:57 INFO  web[][o.s.p.ProcessEntryPoint] Starting Web Server
2022.05.30 10:11:01 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start Web Server
java.lang.IllegalStateException: Fail to create or clean-up directory F:\Web\sonarqube.healthcarelogic.com\data\web\deploy
	at org.sonar.server.app.TomcatContexts.addStaticDir(TomcatContexts.java:84)
	at org.sonar.server.app.TomcatContexts.configure(TomcatContexts.java:59)
	at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:67)
	at org.sonar.server.app.WebServer.start(WebServer.java:55)
	at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:97)
	at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:81)
	at org.sonar.server.app.WebServer.main(WebServer.java:104)
Caused by: java.nio.file.FileSystemException: F:\Web\sonarqube.healthcarelogic.com\data\web\deploy\plugins\abap\sonar-abap-plugin-3.10.0.3628.jar: The process cannot access the file because it is being used by another process.

	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
	at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:274)
	at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
	at java.base/java.nio.file.Files.delete(Files.java:1142)
	at org.sonar.core.util.FileUtils$DeleteRecursivelyFileVisitor.visitFile(FileUtils.java:227)
	at org.sonar.core.util.FileUtils$DeleteRecursivelyFileVisitor.visitFile(FileUtils.java:222)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
	at org.sonar.core.util.FileUtils.deleteDirectoryImpl(FileUtils.java:162)
	at org.sonar.core.util.FileUtils$CleanDirectoryFileVisitor.visitFile(FileUtils.java:206)
	at org.sonar.core.util.FileUtils$CleanDirectoryFileVisitor.visitFile(FileUtils.java:192)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
	at org.sonar.core.util.FileUtils.cleanDirectoryImpl(FileUtils.java:149)
	at org.sonar.core.util.FileUtils.cleanDirectory(FileUtils.java:89)
	at org.sonar.server.app.TomcatContexts$Fs.createOrCleanupDir(TomcatContexts.java:123)
	at org.sonar.server.app.TomcatContexts.addStaticDir(TomcatContexts.java:82)
	... 6 common frames omitted
2022.05.30 10:11:01 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2022.05.30 10:11:01 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2022.05.30 10:11:01 INFO  web[][o.s.s.n.NotificationDaemon] Notification service stopped
2022.05.30 10:11:01 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [sonarqube] appears to have started a thread named [OkHttp TaskRunner] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.15/java.lang.Object.wait(Native Method)\n java.base@11.0.15/java.lang.Object.wait(Object.java:462)\n app//okhttp3.internal.concurrent.TaskRunner$RealBackend.coordinatorWait(TaskRunner.kt:294)\n app//okhttp3.internal.concurrent.TaskRunner.awaitTaskToRun(TaskRunner.kt:218)\n app//okhttp3.internal.concurrent.TaskRunner$runnable$1.run(TaskRunner.kt:59)\n java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.15/java.lang.Thread.run(Thread.java:834)

Hi @nick.pham.hcl

That’s unfortunate! Perhaps you can investigate how to track down which processes are currently accessing this .jar file. On Windows I see it’s possible by searching for associated handles.

If you can track down the executable that has a lock on this file, I think you’ll be on your way. Good luck!!! :+1:

@ganncamp could you please kindly help us with this issue?

i found more log

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube.healthcarelogic.com\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])

and i think you responded in Failed to obtain node locks

your help is greatly appreciated

thank you

@ganncamp please note that I cannot keep deleting the data folder (or write.lock) every 2 days. sonarqube keeps crashing every 2 days and throws the error above :frowning:

Hi,

If I understand well, the issue is not this any more.

Now you have a new one.

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[F:\Web\sonarqube.healthcarelogic.com\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])

Can you confirm?

1 Like

thanks for getting back to me @Fan_Yang

I do not know why the issue The process cannot access the file because it is being used by another process. occurred since i have only 1 sonarqube instance. Fortunately, yes this is not happening for a few days.

And yes the issue failed to obtain node locks ... is still happening

Your kind recommendation is greatly appreciated.

Sorry for the delay.

Please run an Elasticsearch reindex procedure, as mentioned in our troubleshooting documentation:

How do I trigger a full ElasticSearch reindex?

  1. Stop SonarQube
  2. Delete the data/es7 directory
  3. Restart SonarQube

@Fan_Yang thanks for your suggestion. however, if you have read my comments from the beginning of this thread, I have done all 3 steps you advised me to do 5 times. Nothing has changed. SonarQube keeps crashing.

Hi @nick.pham.hcl

It’s interesting that SonarQube crashes every 2 days. Have you noticed a pattern with this? Perhaps your system has scheduled activities every 2 days that trigger this problem?

Otherwise, did you happen to check out the earlier link to try searching for any locks that exist on the file?

It sounds to me like something on your machine is starting another instance of SonarQube every X days. I could imagine that if this is really a periodic occurrence, then a system scheduled event (such as a reboot, maybe?) is failing to stop SonarQube correctly, thus the lock is not released and the newly started instance cannot obtain a lock. Just a guess!

At the end, you can also try increasing the max_local_storage_nodes so that whatever this other process is can obtain a lock. If then in another two days you get this problem again, then you can be pretty certain that your system is somehow starting new instances of SonarQube.

As always, good luck! We’ll figure this out!

1 Like

hi @cwata thank you for your kind suggestion. i have disabled all task schedulers and ensured only 1 instance of sonarqube is running. lets see if it runs after a few days.

i am keen to hear suggestions from @Fan_Yang and SonarSourcer as well please

thank you

Hi,

What @cwata said makes sense to me. This is unusual behavior.
Please keep monitoring and let us know if the issue persists.

Hi @nick.pham.hcl,

Just curious how it’s going! Did you find the root of the problem?

Cheers!

hi @cwata,

it turns out the Task Scheduler that I used to start the SonarQube on Windows start/restart caused this issue.

It is very weird that I had been using the same Task Scheduler for a year with no issue. I only got all errors reported in this thread on SonarQube developer version 9.4. Older versions are fine for a year :frowning:

I have disabled the Task Scheduler and will have to start the SonarQube manually for now.

I hope @Fan_Yang could please give me some advice as to why SonarQube developer version 9.4 throws errors and how to start SonarQube automatically without running into issues.

thanks

Hey @nick.pham.hcl,

could you also mark your last reply as the solution? It could help others with the same problem as you to find the solution :slight_smile:

Glad that you found what was starting your instances. Looking forward to hearing what the underlying issue is with the newest version. Til then, I think that you can configure the exact behavior of the task scheduler. So perhaps you can add a condition that the server is not already running before starting another instance? Not sure whether custom conditions are possible, but Google can help out there.

Good luck!

1 Like