Recovery process is deleting installed plugins

Hello there!
I have detected a strange behavior on SonarQube Enterprise Edition Version 9.4.

Somentimes, for some reason, our SonarQube instance fails and recovers itself from the issue.
This works perfectly because in the event of a issue it is capable of recovering on its own without the need for human help.

The problem is that during the recovery it seems to remove the installed plugins, specifically the Azure Active Directory (AAD) plugin which is the way our users have to login in.

Looking for errors in the logs we have found this in the web.log

2022.05.30 04:35:37 INFO  web[][o.s.s.p.ServerPushClient] The server push client 10.69.18.8 gone without notice, closing the connection (java.io.IOException: Broken pipe)
2022.05.30 04:35:38 ERROR web[][o.s.s.m.ElasticSearchMetricTask] Failed to query ES status
org.sonar.server.es.ElasticsearchException: Fail to execute es requestES cluster health request
	at org.sonar.server.es.EsClient.execute(EsClient.java:313)
	at org.sonar.server.es.EsClient.clusterHealth(EsClient.java:206)
	at org.sonar.server.monitoring.ElasticSearchMetricTask.updateElasticSearchHealthStatus(ElasticSearchMetricTask.java:56)
	at org.sonar.server.monitoring.ElasticSearchMetricTask.run(ElasticSearchMetricTask.java:50)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.ConnectException: Connection refused
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:914)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:299)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:287)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.server.es.EsClient.lambda$clusterHealth$27(EsClient.java:206)
	at org.sonar.server.es.EsClient.execute(EsClient.java:311)
	... 9 common frames omitted
2022.05.30 04:36:29 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2022.05.30 04:36:30 WARN  web[][o.s.p.ProcessEntryPoint$HardStopperThread] Can not stop in 1000ms
2022.05.30 04:36:30 WARN  web[][o.s.s.a.EmbeddedTomcat] Failed to stop web server
org.apache.catalina.LifecycleException: A child container failed during stop
	at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:1007)
	at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
	at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:486)
	at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
	at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:784)
	at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
	at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:450)
	at org.sonar.server.app.EmbeddedTomcat.terminate(EmbeddedTomcat.java:112)
	at org.sonar.server.app.WebServer.hardStop(WebServer.java:86)
	at org.sonar.process.ProcessEntryPoint$HardStopperThread.lambda$new$0(ProcessEntryPoint.java:219)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 INFO  web[][o.s.s.e.CoreExtensionStopper] Stopping Governance
2022.05.30 04:36:32 INFO  web[][o.s.s.e.CoreExtensionStopper] Stopping Governance (done) | time=1ms
2022.05.30 04:36:32 INFO  web[][o.s.s.n.NotificationDaemon] Notification service stopped
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [pool-32-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-12] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-14] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-15] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:32 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [logback-16] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.14/jdk.internal.misc.Unsafe.park(Native Method)\n java.base@11.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)\n java.base@11.0.14/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)\n java.base@11.0.14/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)\n java.base@11.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n java.base@11.0.14/java.lang.Thread.run(Thread.java:829)
2022.05.30 04:36:38 INFO  web[][o.s.s.p.ServerPushClient] The server push client 10.69.18.5 gone without notice, closing the connection (java.io.IOException: Broken pipe)
2022.05.30 04:41:00 INFO  web[][o.s.p.ProcessEntryPoint] Starting Web Server
2022.05.30 04:41:03 INFO  web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
2022.05.30 04:41:05 INFO  web[][o.s.s.p.LogServerVersion] SonarQube Server / 9.4.0.54424 / 5c36a2e19337da136080a276102eaaefce073c68
2022.05.30 04:41:05 INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://10.46.244.20:5432/sonardb?socketTimeout=1500
2022.05.30 04:41:10 INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
2022.05.30 04:41:10 INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is enabled
2022.05.30 04:41:11 INFO  web[][o.s.c.e.CoreExtensionsLoader] Loaded core extensions: enterprise-edition, ce-workers, developer-scanner, developer-server, governance, license, securityreport, monitoring
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin ABAP Code Quality and Security / 3.10.0.3628 / 81de0cd6517fb9d5b38f340c6e3ef4639471547d
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Apex Code Quality and Security / 1.9.0.3429 / faf3c08c0ebf3a877186442cd4a0ddd50de079a8
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin C# Code Quality and Security / 8.36.1.44192 / 855387dffd60a6e903221a41ff5282452c1b742b
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin CFamily Code Quality and Security / 6.32.0.44918 / dc90c831c08ed2dfa57771c99415e8222173c176
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin COBOL Code Quality / 4.6.2.4876 / 314414101278f77b354b02acd8808fdbff700d3f
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Configuration detection fot Code Quality and Security / 1.2.0.267 / 4f37ba9ffb37a96d5883e52ad392ed32c5c6eaab
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Flex Code Quality and Security / 2.7.0.2865 / 9e61d9774455b25eefd0b71bf8903cb030641864
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Go Code Quality and Security / 1.9.0.3429 / faf3c08c0ebf3a877186442cd4a0ddd50de079a8
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin HTML Code Quality and Security / 3.6.0.3106 / 963bdc9f051f5dcb698b418790364231aeea04f2
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin IaC Code Quality and Security / 1.7.0.2012 / a006dbf717537ce3a87bb2b6b088597ead3273d1
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin JaCoCo / 1.1.1.1157 / 83478572b9f23efac29de15e30c7758bbb0c0e47
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Java Code Quality and Security / 7.11.0.29148 / e0695da6058ee662cb747ac920253d344609f19c
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin JavaScript/TypeScript/CSS Code Quality and Security / 9.1.0.17747 / e9a6665cfd326e7eeb36dbd112510b06f439ea34
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Kotlin Code Quality and Security / 2.9.0.1147 / 6ce439214a685d4aabde6461fd609a9a9c1df1c6
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin PHP Code Quality and Security / 3.23.1.8766 / db197ac9daccf3d305ad5e40618a65a95f3d5dde
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin PL/I Code Quality and Security / 1.11.1.2727 / 6976734b43f9247c20529879eca3495cfa89f47a
C# / 9.4.1.15913 / e99ab18ffaa0aecc9781038c1279d92b213f5152
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for JS / 9.4.1.15913 / e99ab18ffaa0aecc9781038c1279d92b213f5152
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for Java / 9.4.1.15913 / e99ab18ffaa0aecc9781038c1279d92b213f5152
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for PHP / 9.4.1.15913 / e99ab18ffaa0aecc9781038c1279d92b213f5152
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for Python / 9.4.1.15913 / e99ab18ffaa0aecc9781038c1279d92b213f5152
2022.05.30 04:41:13 INFO  web[][o.s.s.p.ServerPluginManager] Deploy plugin XML Code Quality and Security / 2.5.0.3376 / 3b089c16cf0736c56f4fdb2fc88ddad59e7ca45d
2022.05.30 04:41:18 INFO  web[][o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2022.05.30 04:41:20 INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [http://localhost:9001]
2022.05.30 04:41:22 WARN  web[][o.s.a.s.w.WebService$Action] Description is not set on action api/monitoring/metrics
2022.05.30 04:41:22 WARN  web[][o.s.a.s.w.WebService$Action] Since is not set on action api/monitoring/metrics
2022.05.30 04:41:22 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/monitoring/metrics
2022.05.30 04:41:22 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/system/liveness
2022.05.30 04:41:22 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceFilter@769197bd [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/components/update_key, ...]}]
2022.05.30 04:41:23 INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2022.05.30 04:41:23 INFO  web[][o.s.s.p.DetectPluginChange] Detect plugin changes
2022.05.30 04:41:31 INFO  web[][o.s.s.s.LogServerId] Server ID: XXXXXXXXXX
2022.05.30 04:41:32 INFO  web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/system/liveness
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/plugins/download
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/analysis_cache/get
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/alm_integrations/list_bitbucketserver_projects
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/alm_integrations/search_bitbucketcloud_repos
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/alm_integrations/check_pat
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/alm_integrations/list_azure_projects
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/alm_integrations/search_bitbucketserver_repos
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/alm_integrations/search_azure_repos
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/project_dump/status
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/push/sonarlint_events
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/security_reports/download
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/security_reports/show
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/views/definition
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/governance_reports/download
2022.05.30 04:41:37 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/support/info
2022.05.30 04:41:37 INFO  web[][o.s.s.a.p.ExpiredSessionsCleaner] Purge of expired session tokens has removed 15 elements
2022.05.30 04:41:37 INFO  web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
2022.05.30 04:41:37 INFO  web[][o.s.s.a.p.ExpiredSessionsCleaner] Purge of expired SAML message ids has removed 0 elements
2022.05.30 04:41:37 INFO  web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled.
2022.05.30 04:41:38 INFO  web[][com.sonarsource.D.H] Loaded virtual plugin License for SonarLint
2022.05.30 04:41:38 INFO  web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index
2022.05.30 04:41:38 INFO  web[][o.s.s.s.RegisterMetrics] Register metrics
2022.05.30 04:41:38 INFO  web[][o.s.s.r.RegisterRules] Register rules
2022.05.30 04:41:56 INFO  web[][o.s.s.q.b.BuiltInQProfileRepositoryImpl] Load quality profiles
2022.05.30 04:41:57 INFO  web[][o.s.s.q.RegisterQualityProfiles] Register quality profiles
2022.05.30 04:41:57 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile css/Sonar way
2022.05.30 04:41:57 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile scala/Sonar way
2022.05.30 04:41:57 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile jsp/Sonar way
2022.05.30 04:41:57 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way
2022.05.30 04:41:57 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile py/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile apex/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile plsql/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile java/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile web/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile flex/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile xml/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile json/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile text/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vbnet/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cloudformation/Sonar way
2022.05.30 04:41:58 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile yaml/Sonar way
2022.05.30 04:41:59 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile swift/Sonar way
2022.05.30 04:41:59 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cpp/Sonar way
2022.05.30 04:41:59 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile c/Sonar way
2022.05.30 04:41:59 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile go/Sonar way
2022.05.30 04:41:59 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile rpg/Sonar way
2022.05.30 04:41:59 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile kotlin/Sonar way
2022.05.30 04:41:59 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vb/Sonar way
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile tsql/Sonar way
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile pli/Sonar way
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ruby/Sonar way
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cs/Sonar way
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cobol/Sonar way
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Sonar way
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/PSR-2
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Drupal
2022.05.30 04:42:00 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile terraform/Sonar way
2022.05.30 04:42:01 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile abap/Sonar way
2022.05.30 04:42:01 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way
2022.05.30 04:42:01 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile objc/Sonar way
2022.05.30 04:42:01 INFO  web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates
2022.05.30 04:42:01 INFO  web[][o.s.s.s.RenameDeprecatedPropertyKeys] Rename deprecated property keys
2022.05.30 04:42:01 INFO  web[][o.s.s.s.UpgradeSuggestionsCleaner] Dismissed messages cleanup
2022.05.30 04:42:01 INFO  web[][o.s.s.s.RegisterPlugins] Register plugins
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.SonarLintConnectionFilter@1fb777af [pattern=UrlPattern{inclusions=[/api/*], exclusions=[]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceFilter@2741d719 [pattern=UrlPattern{inclusions=[/api/issues/delete_comment.*, ...], exclusions=[/api/authentication/login.*, ...]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceReroutingFilter@7f72e376 [pattern=UrlPattern{inclusions=[/api/components/bulk_update_key, ...], exclusions=[]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.DefaultAdminCredentialsVerifierFilter@dd3df3d [pattern=UrlPattern{inclusions=[/*], exclusions=[*.css, ...]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.InitFilter@297ddb27 [pattern=UrlPattern{inclusions=[/sessions/init/*], exclusions=[]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.OAuth2CallbackFilter@ad280e [pattern=UrlPattern{inclusions=[/oauth2/callback/*], exclusions=[]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ResetPasswordFilter@68ac023e [pattern=UrlPattern{inclusions=[/*], exclusions=[*.css, ...]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LoginAction@c3c90af [pattern=UrlPattern{inclusions=[/api/authentication/login], exclusions=[]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LogoutAction@3a479772 [pattern=UrlPattern{inclusions=[/api/authentication/logout], exclusions=[]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.ValidateAction@53a63e5a [pattern=UrlPattern{inclusions=[/api/authentication/validate], exclusions=[]}]
2022.05.30 04:42:01 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.plugins.PluginsRiskConsentFilter@7b150221 [pattern=UrlPattern{inclusions=[/*], exclusions=[*.css, ...]}]
2022.05.30 04:42:02 INFO  web[][o.s.s.q.ProjectsInWarningDaemon] Counting number of projects in warning is not started as there are no projects in this situation.
2022.05.30 04:42:02 INFO  web[][o.s.s.e.CoreExtensionBootstraper] Bootstrapping Governance
2022.05.30 04:42:02 INFO  web[][o.s.s.e.CoreExtensionBootstraper] Bootstrapping Governance (done) | time=8ms
2022.05.30 04:42:02 INFO  web[][o.s.s.p.p.PlatformLevelStartup] Running Enterprise Edition
2022.05.30 04:42:03 INFO  web[][o.s.s.p.Platform] Web Server is operational
2022.05.30 04:46:09 INFO  web[][o.s.s.p.ServerPushClient] The server push client 10.69.18.5 gone without notice, closing the connection (java.io.IOException: Broken pipe)

It seems that the trigger of this behavior is some kind of ElasticSearch issue and the “Hard stopping process” or “Stopping Governance” process that follows.

Aside from trying to understand why ElasticSearch has failed, our main drawback is that the recovery process removes the AAD plugin preventing access to users.

Added the complete log for further analysis.
sonarqube_web.log (205.4 KB)

Some other info.

  • SonarQube is working on AKS with Docker Hub image (SonarQube Enterprise Edition Version 9.4 (build 54424))
  • PostgreSQL Version: 11.12 as a service on Azure and running with Helm
1 Like

Hi,

Did you set up the volumes described in the docs? Specifically, the one for sonarqube_extensions is where your 3rd-party plugins would live, and be persistent across restarts.

 
Ann

1 Like

Hello.
Thanks for the response.
That has given us a clue of where our problem may be.

We are working on it. As soon as we solve the problem we will share it.

Thanks

1 Like