I am facing an issue where the health checks for my SonarQube pod in an AWS Kubernetes setup with ALB (Application Load Balancer) are failing, even though the pod is running fine and accessible via HTTP. The health checks from the ALB to the SonarQube pod are returning as unhealthy, and I’m unable to identify the root cause.
Environment Setup:
- SonarQube Version: 24.12.0
- Kubernetes: AWS EKS (Elastic Kubernetes Service)
- Load Balancer: AWS ALB
- Service Type: ClusterIP
- Health Check Path:
/api/system/status
- Health Check Port: 9000
- Deployed with helm helm-chart-sonarqube/charts/sonarqube at master · SonarSource/helm-chart-sonarqube · GitHub
- Service config
*`
service:
type: ClusterIP
externalPort: 9000
internalPort: 9000
labels:
annotations: {}`
ingress config
ingress:
enabled: true
# Used to create an Ingress record.
hosts:
- name: sonarqube-dev.dmain.com
# Different clouds or configurations might need /* as the default path
path: /
# For additional control over serviceName and servicePort
#serviceName:
# servicePort: somePort
# the pathType can be one of the following values: Exact|Prefix|ImplementationSpecific(default)
pathType: Prefix
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:cer
alb.ingress.kubernetes.io/group.name: alb-group name
alb.ingress.kubernetes.io/healthcheck-healthy-threshold: "2"
alb.ingress.kubernetes.io/healthcheck-interval-seconds: "30"
alb.ingress.kubernetes.io/healthcheck-path: /api/system/status
alb.ingress.kubernetes.io/healthcheck-port: "9000"
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "20"
alb.ingress.kubernetes.io/healthcheck-unhealthy-threshold: "6"
alb.ingress.kubernetes.io/inbound-cidrs: valid ips
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/security-groups: sg
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/target-type: ip
# kubernetes.io/tls-acme: "true"
# Set the ingressClassName on the ingress record
ingressClassName: alb
Problem Description:
- The SonarQube pod is running and accessible within the cluster. However, the ALB health checks are failing.
- Pod Access: When trying to access
/api/system/status
directly, it returns a405 Method Not Allowed
, but accessing the root/
returns a200 OK
status. - ALB Health Check: Despite the pod being accessible via
curl -I http://pod-ip:9000/
, the health check from the ALB fails, marking the target as unhealthy. - Health Check Logs: The ALB health check logs show that the target is unhealthy, even though the pod is functional and the
curl
command from within the pod works fine.
What I’ve Tried So Far:
- Increasing Readiness Delay: Increased the
readinessProbe
initialDelaySeconds
to allow SonarQube enough time to initialize. - Health Check Timeout: Set the
healthcheck-timeout-seconds
for the ALB target to30
seconds, but the issue persists. - Idle Timeout: Increased the
idle_timeout.timeout_seconds
in the ALB configuration to300
seconds, but still no resolution. - Correct Path in ALB: Verified that the health check path is correctly set to
/api/system/status
and the port is set to9000
.
Symptoms Observed:
- Pod Access:
curl -I http://localhost:9000/api/system/status
returns405 Method Not Allowed
.curl -I http://localhost:9000/
returns200 OK
.
- ALB Health Check Failure: ALB shows the target as unhealthy despite the pod’s readiness and the ability to access it internally.
Questions:
- Why does the health check fail even though the pod responds correctly?
- What can be configured or adjusted on the ALB side to ensure that health checks pass?
- Is there a discrepancy between the expected health check path and what SonarQube exposes for readiness?
- Is there any additional configuration required for the SonarQube pod in relation to health checks and ALB compatibility?