App-node in my local docker-compose is fine, but when I put it onto k8s, it fails with the following message, not sure what went wrong. Search-nodes are working fine and they work with my local docker-compose based app-node:
2021.07.01 22:45:03 WARN app[c.h.i.impl.Node] [10.32.8.189]:9003 [SonarQube] [4.2] Terminating forcefully…
2021.07.01 22:45:03 INFO web[o.s.p.ProcessEntryPoint] Gracefully stopping process
2021.07.01 22:45:04 WARN web[o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [SQ starter] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.11/java.lang.ClassLoader.defineClass1(Native Method)\n java.base@11.0.11/java.lang.ClassLoader.defineClass(Unknown Source)\n java.base@11.0.11/java.security.SecureClassLoader.defineClass(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)\n java.base@11.0.11/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)\n java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)\n java.base@11.0.11/java.lang.Class.getDeclaredMethods0(Native Method)\n java.base@11.0.11/java.lang.Class.privateGetDeclaredMethods(Unknown Source)\n java.base@11.0.11/java.lang.Class.getDeclaredMethods(Unknown Source)\n app//org.picocontainer.injectors.AdaptingInjection$1.run(AdaptingInjection.java:203)\n java.base@11.0.11/java.security.AccessController.doPrivileged(Native Method)\n app//org.picocontainer.injectors.AdaptingInjection.injectionMethodAnnotated(AdaptingInjection.java:200)\n app//org.picocontainer.injectors.AdaptingInjection.methodAnnotatedInjectionAdapter(AdaptingInjection.java:171)\n app//org.picocontainer.injectors.AdaptingInjection.createComponentAdapter(AdaptingInjection.java:70)\n app//org.picocontainer.behaviors.AbstractBehaviorFactory.createComponentAdapter(AbstractBehaviorFactory.java:44)\n app//org.picocontainer.behaviors.OptInCaching.createComponentAdapter(OptInCaching.java:45)\n app//org.picocontainer.DefaultPicoContainer.addComponent(DefaultPicoContainer.java:536)\n app//org.picocontainer.DefaultPicoContainer.access$300(DefaultPicoContainer.java:84)\n app//org.picocontainer.DefaultPicoContainer$AsPropertiesPicoContainer.addComponent(DefaultPicoContainer.java:1149)\n app//org.sonar.core.platform.ComponentContainer.addComponent(ComponentContainer.java:229)\n app//org.sonar.core.platform.Module.add(Module.java:46)\n app//org.sonar.server.batch.BatchWsModule.configureModule(BatchWsModule.java:27)\n app//org.sonar.core.platform.Module.configure(Module.java:32)\n app//org.sonar.server.platform.platformlevel.PlatformLevel.configure(PlatformLevel.java:78)\n app//org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:210)\n app//org.sonar.server.platform.PlatformImpl.startLevel34Containers(PlatformImpl.java:187)\n app//org.sonar.server.platform.PlatformImpl.access$500(PlatformImpl.java:46)\n app//org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:120)\n app//org.sonar.server.platform.PlatformImpl$1$$Lambda$499/0x0000000100569040.run(Unknown Source)\n app//org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:370)\n app//org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:120)\n app//org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:354)\n java.base@11.0.11/java.lang.Thread.run(Unknown Source)
Sorry, correction, we are using 8.9.1. When comparing logs between k8s and my local deployment, I see the app node is actually reaching out to https://update.sonarsource.org/update-center.properties. In our k8s, outbound traffic is blocked. I think it might be the reason, what is the way to disable outbound traffic in app nodes, e.g. disabling auto-update etc?
I also notice that the log stops before getting the server id. maybe db issues? but the search nodes are not reporting the same issue.
from the db log, I got these entries:
2021-07-02 17:57:34 UTC:10.32.16.176(48614):sonar@postgres:[17815]:LOG: could not receive data from client: Connection reset by peer
2021-07-02 17:57:34 UTC:10.32.16.176(48572):sonar@postgres:[17704]:LOG: could not receive data from client: Connection reset by peer
Maybe SONAR_AUTH_JWTBASE64HS256SECRET was not right or escaped correctly?
Log comparison
k8s pod log:
2021.07.02 20:42:16 INFO web[o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2021.07.02 20:42:19 INFO app[o.s.a.SchedulerImpl] Stopping SonarQube
2021.07.02 20:42:19 WARN app[c.h.i.impl.Node] [10.32.47.159]:9003 [SonarQube] [4.2] Terminating forcefully…
2021.07.02 20:42:19 INFO web[o.s.p.ProcessEntryPoint] Gracefully stopping process
2021.07.02 20:42:20 INFO web[o.s.s.app.WebServer] WebServer stopped
2021.07.02 20:42:20 INFO app[o.s.a.SchedulerImpl] Process[web] is stopped
2021.07.02 20:42:20 INFO app[o.s.a.SchedulerImpl] SonarQube is stopped
docker-compose log:
2021.07.02 20:39:35 INFO web[o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2021.07.02 20:39:36 INFO web[A.A.A.A.A.C] JavaScript/TypeScript frontend is enabled
2021.07.02 20:39:37 INFO web[o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
2021.07.02 20:39:39 INFO web[o.s.s.s.LogServerId] Server ID: XXXX-XXXX-XXXX-XXXX
2021.07.02 20:39:39 WARN web[o.s.s.a.LogOAuthWarning] For security reasons, OAuth authentication should use HTTPS. You should set the property ‘Administration > Configuration > Server base URL’ to a HTTPS URL.
the following log segment confuses me a little. it looks like SQ received a SIGTERM from somewhere
could it be that it gets killed by k8s as the pod does not get healthy in time?
If not, could you check the logs in the $SONARQUBE_HOME/logs directory? sadly not everything gets printed to stdout