Sonarqube hazelCast setup through k8s service

SonarQube
1

I am trying to setup app nodes in data center edition 8.9.1 in k8s. In our setup, we can’t assign a static IP to each app node and I am trying to setup a k8s service for each app node, each k8s service has a fixed ip that I am putting into to the sonar.cluster.hosts parameter.
However, when each app node starts up, and join the cluster, I think it is using app node’s pod ip instead of the k8s service ip to join and since the pod ip is not listed in the sonar.cluster.hosts, I am getting a lot of of not a member errors.
On the other hand, if I set the app node’s sonar.cluster.node.host to be the k8s service ip, I got another error saying “Target is this node!”.
How do I make it work? Thanks in advance.

2

Hi @tlong ,

yeah this is currently a problem and the main reason why the DCE is not yet supported on k8s. Also i would highly suggest that you wait until we can deliver MMF-2412 as your setup will probably be unreliable.

That said, you can configure the service name for the hazelcast lookup. in 75% of the cases this works to get a cluster formed.

...
     containers:
       - name: "sonarqube-dce-app"
         image: "sonarqube:8.9.1-datacenter-app"
         imagePullPolicy: "IfNotPresent"
         ports:
           - name: http
             containerPort: 9000
             protocol: TCP
           - name: hazelcast
             containerPort: 9003
             protocol: TCP
         env:
           - name: SONAR_CLUSTER_HOSTS
             value: "dce-sonarqube-dce"
...

apiVersion: v1
kind: Service
metadata:
  labels:
    app: sonarqube-dce
  name: dce-sonarqube-dce
  namespace: dce
spec:
  ports:
  - name: http
    port: 9000
    protocol: TCP
    targetPort: http
  - name: hazelcast
    port: 9003
    protocol: TCP
    targetPort: hazelcast
  selector:
    app: sonarqube-dce
    release: dce
  sessionAffinity: None
  type: ClusterIP

this will result in a cluster getting formed but the startup procedure is still a little bit unstable. i would suggest you start one replica of the application nodes and scale up after the database migration has passed and the pod is healthy.
The errors you observe are expected as hazelcast will do a DNS lookup on the service name and gets a list of all ip adresses, including the own host. This will probably get resolved with the MMF i linked above.

hope this helps

3

Thanks, our setup went past the initial setup, now the issues is that the hazel cast is picking random ports to communicate with each other, I tried to set the env variable of SONAR_CLUSTER_NODE_CE_PORT and SONAR_CLUSTER_NODE_WEB_PORT but they are not doing anything for me.

2021.07.08 18:21:38 INFO  ce[][com.hazelcast.system] [XX.XX.XX.XX]:40649 [SonarQube] [4.2] Hazelcast 4.2 (20210324 - 405cfd1) starting at [XX.XX.XX.XX]:40649
2021.07.08 18:21:41 INFO  ce[][c.h.i.impl.Node] [XX.XX.XX.XX]:40649 [SonarQube] [4.2] Using TCP/IP discovery
2021.07.08 18:21:41 WARN  ce[][c.h.cp.CPSubsystem] [XX.XX.XX.XX]:40649 [SonarQube] [4.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2021.07.08 18:21:43 INFO  ce[][c.h.i.d.Diagnostics] [XX.XX.XX.XX]:40649 [SonarQube] [4.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2021.07.08 18:21:43 INFO  ce[][c.h.c.LifecycleService] [XX.XX.XX.XX]:40649 [SonarQube] [4.2] [XX.XX.XX.XX]:40649 is STARTING