Sonarqube app-node fails in k8s

Sonarqube data-center 8.7

App-node in my local docker-compose is fine, but when I put it onto k8s, it fails with the following message, not sure what went wrong. Search-nodes are working fine and they work with my local docker-compose based app-node:

2021.07.01 22:45:03 WARN app[c.h.i.impl.Node] [10.32.8.189]:9003 [SonarQube] [4.2] Terminating forcefully…
2021.07.01 22:45:03 INFO web[o.s.p.ProcessEntryPoint] Gracefully stopping process
2021.07.01 22:45:04 WARN web[o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [SQ starter] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.11/java.lang.ClassLoader.defineClass1(Native Method)\n java.base@11.0.11/java.lang.ClassLoader.defineClass(Unknown Source)\n java.base@11.0.11/java.security.SecureClassLoader.defineClass(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)\n java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)\n java.base@11.0.11/java.lang.Class.getDeclaredMethods0(Native Method)\n java.base@11.0.11/java.lang.Class.privateGetDeclaredMethods(Unknown Source)\n java.base@11.0.11/java.lang.Class.getDeclaredMethods(Unknown Source)\n app//org.picocontainer.injectors.AdaptingInjection$1.run(AdaptingInjection.java:203)\n java.base@11.0.11/java.security.AccessController.doPrivileged(Native Method)\n app//org.picocontainer.injectors.AdaptingInjection.injectionMethodAnnotated(AdaptingInjection.java:200)\n app//org.picocontainer.injectors.AdaptingInjection.methodAnnotatedInjectionAdapter(AdaptingInjection.java:171)\n app//org.picocontainer.injectors.AdaptingInjection.createComponentAdapter(AdaptingInjection.java:70)\n app//org.picocontainer.behaviors.AbstractBehaviorFactory.createComponentAdapter(AbstractBehaviorFactory.java:44)\n app//org.picocontainer.behaviors.OptInCaching.createComponentAdapter(OptInCaching.java:45)\n app//org.picocontainer.DefaultPicoContainer.addComponent(DefaultPicoContainer.java:536)\n app//org.picocontainer.DefaultPicoContainer.access$300(DefaultPicoContainer.java:84)\n app//org.picocontainer.DefaultPicoContainer$AsPropertiesPicoContainer.addComponent(DefaultPicoContainer.java:1149)\n app//org.sonar.core.platform.ComponentContainer.addComponent(ComponentContainer.java:229)\n app//org.sonar.core.platform.Module.add(Module.java:46)\n app//org.sonar.server.batch.BatchWsModule.configureModule(BatchWsModule.java:27)\n app//org.sonar.core.platform.Module.configure(Module.java:32)\n app//org.sonar.server.platform.platformlevel.PlatformLevel.configure(PlatformLevel.java:78)\n app//org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:210)\n app//org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:187)\n app//org.sonar.server.platform.PlatformImpl.access$500(PlatformImpl.java:46)\n app//org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:120)\n app//org.sonar.server.platform.PlatformImpl$1$$Lambda$499/0x0000000100569040.run(Unknown Source)\n app//org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:370)\n app//org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:120)\n app//org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:354)\n java.base@11.0.11/java.lang.Thread.run(Unknown Source)

Hi @tlong ,

can you share the part before

apart from that please note that the DCE on k8s will only go beta soon (with the resolvement of MMF-2412).

ah and as a final note: Sonarqube 8.7 is EOL. when you are starting fresh you should go with 8.9 which is the current LTS

Sorry, correction, we are using 8.9.1. When comparing logs between k8s and my local deployment, I see the app node is actually reaching out to https://update.sonarsource.org/update-center.properties. In our k8s, outbound traffic is blocked. I think it might be the reason, what is the way to disable outbound traffic in app nodes, e.g. disabling auto-update etc?

Here is more log, the outbound traffic is not the issue, the exception changes a bit but here is more logs:

2021.07.02 17:03:46 INFO web[o.s.s.p.LogServerVersion] SonarQube Server / 8.9.1.44547 / XXXXXXXXXXXX
2021.07.02 17:03:46 INFO web[o.sonar.db.Database] Create JDBC data source for XXXXXXXXXXX
2021.07.02 17:03:56 INFO web[o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
2021.07.02 17:03:56 INFO web[o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
2021.07.02 17:03:56 INFO web[o.s.s.p.WebServerImpl] Cluster enabled (startup leader)
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin ABAP Code Quality and Security / 3.9.1.3127 / a62ddf6ddd7379d398a58d32c9931a2feef61e24
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Apex Code Quality and Security / 1.8.3.2219 / d6ad7a5a47fc4785d2e80918fb7424be46e38a7f
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin C# Code Quality and Security / 8.22.0.31243 / e3cee7838d992e31dcdd90cf4f7406bb20535e8e
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin CFamily Code Quality and Security / 6.20.1.32841 / cfe20794a83197d63a97c7310114b0bf8ba0a548
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin COBOL Code Quality / 4.6.2.4876 / 314414101278f77b354b02acd8808fdbff700d3f
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin CSS Code Quality and Security / 1.4.2.2002 / faa7d4f1407df67df7ada53caf677ab783721173
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Flex Code Quality and Security / 2.6.1.2564 / bb723840701bda72510b7a47742811d20daad331
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Go Code Quality and Security / 1.8.3.2219 / d6ad7a5a47fc4785d2e80918fb7424be46e38a7f
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin HTML Code Quality and Security / 3.4.0.2754 / 38f7ff864ae15152c9f1efc3014594f7e7ca7b6e
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin JaCoCo / 1.1.1.1157 / 83478572b9f23efac29de15e30c7758bbb0c0e47
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Java Code Quality and Security / 6.15.1.26025 / 1b1e96715bfa9f6a4ae24e95cc5b91f0edce609f
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin JavaScript/TypeScript Code Quality and Security / 7.4.4.15624 / 481b2e69339b016b5d7d1eb27f0abf20dd6bd961
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Kotlin Code Quality and Security / 1.8.3.2219 / d6ad7a5a47fc4785d2e80918fb7424be46e38a7f
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin PHP Code Quality and Security / 3.17.0.7439 / 44c7760147080c157fa0ff579772f92d3c8e1ebf
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin PL/I Code Quality and Security / 1.11.1.2727 / 6976734b43f9247c20529879eca3495cfa89f47a
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin PL/SQL Code Quality and Security / 3.6.1.3873 / 342f7fcf17ecb7fbf827a2aacf630be1f4157625
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Python Code Quality and Security / 3.4.1.8066 / 22139ec73fb2f32044f66477ea52734415683668
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin RPG Code Quality / 2.5.1.2575 / 7ec2910da67ffcebf5d5944244c641d9de4b2a8a
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Ruby Code Quality and Security / 1.8.3.2219 / d6ad7a5a47fc4785d2e80918fb7424be46e38a7f
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Scala Code Quality and Security / 1.8.3.2219 / d6ad7a5a47fc4785d2e80918fb7424be46e38a7f
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Swift Code Quality and Security / 4.3.1.4892 / 2b249272bc4430519bdab769886b12c9a82084b5
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin T-SQL Code Quality and Security / 1.5.1.4340 / 11f3de5739b539749d6c2848bda8fc90135d91b6
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin VB.NET Code Quality and Security / 8.22.0.31243 / e3cee7838d992e31dcdd90cf4f7406bb20535e8e
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin VB6 Code Quality and Security / 2.7.1.2721 / b1d1c140bb9ec432799c480e2245a63d947fbef9
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Analysis / 8.9.0.11439 / 04498e7b336a73db9508145d1e8cc44b9330528d
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for C# / 8.9.0.11439 / 04498e7b336a73db9508145d1e8cc44b9330528d
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for JS / 8.9.0.11439 / 04498e7b336a73db9508145d1e8cc44b9330528d
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for Java / 8.9.0.11439 / 04498e7b336a73db9508145d1e8cc44b9330528d
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for PHP / 8.9.0.11439 / 04498e7b336a73db9508145d1e8cc44b9330528d
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin Vulnerability Rules for Python / 8.9.0.11439 / 04498e7b336a73db9508145d1e8cc44b9330528d
2021.07.02 17:03:59 INFO web[o.s.s.p.ServerPluginManager] Deploy plugin XML Code Quality and Security / 2.2.0.2973 / 16002945f0725643a7b42f090572795dd8b72a0f
2021.07.02 17:04:05 INFO app[o.s.a.SchedulerImpl] Stopping SonarQube
2021.07.02 17:04:05 WARN app[c.h.i.impl.Node] [XX.XX.XX.XX]:9003 [SonarQube] [4.2] Terminating forcefully…
2021.07.02 17:04:06 INFO web[o.s.p.ProcessEntryPoint] Gracefully stopping process
2021.07.02 17:04:08 INFO web[o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2021.07.02 17:04:09 INFO web[o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceFilter@73661705 [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, …], exclusions=[/api/components/update_key, …]}]
2021.07.02 17:04:09 INFO web[o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2021.07.02 17:04:10 WARN web[o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [SQ starter] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/java.lang.ClassLoader.findBootstrapClass(Native Method)
java.base@11.0.11/java.lang.ClassLoader.findBootstrapClassOrNull(Unknown Source)
java.base@11.0.11/java.lang.System$2.findBootstrapClassOrNull(Unknown Source)
java.base@11.0.11/jdk.internal.loader.ClassLoaders$BootClassLoader.loadClassOrNull(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
java.base@11.0.11/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)
java.base@11.0.11/java.lang.ClassLoader.defineClass1(Native Method)
java.base@11.0.11/java.lang.ClassLoader.defineClass(Unknown Source)
java.base@11.0.11/java.security.SecureClassLoader.defineClass(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
java.base@11.0.11/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)
app//org.sonar.server.platform.platformlevel.PlatformLevel4.configureLevel(PlatformLevel4.java:266)
app//org.sonar.server.platform.platformlevel.PlatformLevel.configure(PlatformLevel.java:74)
app//org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:210)
app//org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:187)
app//org.sonar.server.platform.PlatformImpl.access$500(PlatformImpl.java:46)
app//org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:120)
app//org.sonar.server.platform.PlatformImpl$1$$Lambda$501/0x0000000100569840.run(Unknown Source)
app//org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:370)
app//org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:120)
app//org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:354)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)

I also notice that the log stops before getting the server id. maybe db issues? but the search nodes are not reporting the same issue.
from the db log, I got these entries:
2021-07-02 17:57:34 UTC:10.32.16.176(48614):sonar@postgres:[17815]:LOG: could not receive data from client: Connection reset by peer
2021-07-02 17:57:34 UTC:10.32.16.176(48572):sonar@postgres:[17704]:LOG: could not receive data from client: Connection reset by peer

Maybe SONAR_AUTH_JWTBASE64HS256SECRET was not right or escaped correctly?

Log comparison
k8s pod log:
2021.07.02 20:42:16 INFO web[o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2021.07.02 20:42:19 INFO app[o.s.a.SchedulerImpl] Stopping SonarQube
2021.07.02 20:42:19 WARN app[c.h.i.impl.Node] [10.32.47.159]:9003 [SonarQube] [4.2] Terminating forcefully…
2021.07.02 20:42:19 INFO web[o.s.p.ProcessEntryPoint] Gracefully stopping process
2021.07.02 20:42:20 INFO web[o.s.s.app.WebServer] WebServer stopped
2021.07.02 20:42:20 INFO app[o.s.a.SchedulerImpl] Process[web] is stopped
2021.07.02 20:42:20 INFO app[o.s.a.SchedulerImpl] SonarQube is stopped

docker-compose log:
2021.07.02 20:39:35 INFO web[o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2021.07.02 20:39:36 INFO web[A.A.A.A.A.C] JavaScript/TypeScript frontend is enabled
2021.07.02 20:39:37 INFO web[o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
2021.07.02 20:39:39 INFO web[o.s.s.s.LogServerId] Server ID: XXXX-XXXX-XXXX-XXXX
2021.07.02 20:39:39 WARN web[o.s.s.a.LogOAuthWarning] For security reasons, OAuth authentication should use HTTPS. You should set the property ‘Administration > Configuration > Server base URL’ to a HTTPS URL.

Hi @tlong ,

the following log segment confuses me a little. it looks like SQ received a SIGTERM from somewhere

could it be that it gets killed by k8s as the pod does not get healthy in time?
If not, could you check the logs in the $SONARQUBE_HOME/logs directory? sadly not everything gets printed to stdout

Thanks, that actually was the issue, the liveness check wasn’t long enough. It is resolved now.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.