Upgrade from 10.4.1 to 10.5.0 broke LDAP authentication

jonesbusy · April 18, 2024, 8:32am

Hi,

We are regularly upgrading our SonarQube Enterprise instance on our dev environment to test feature of 10.x releases.

Previous 10.x released didn’t cause any issue, but during upgrade from 10.4.1 to 10.5.0 it broke our LDAP authentication.

We are deploying via official Helm chart and our configuration for LDAP is the following (only relevant section are added

We are using few LDAP server

  sonarProperties:
    sonar.forceAuthentication: true
    sonar.updatecenter.activate: false
    sonar.security.realm: LDAP
    sonar.security.localUsers: admin,runner
    sonar.security.savePassword: false
    sonar.authenticator.createUsers: true
    sonar.authenticator.downcase: true
    sonar.log.level: INFO
    ldap.servers: ****ls1ch,****ls2ch,****ls1vn,****ls2vn,****ls1svc,****ls2svc
    ldap.****ls1ch.url: ldaps://****ls1.***********:636/
    ldap.****ls1ch.user.baseDn: OU=Users,OU=Accounts,OU=CH,DC=****,DC=local
    ldap.****ls1ch.user.request: (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={login}))
    ldap.****ls1ch.group.baseDn: OU=****,OU=Groups,OU=CH,DC=****,DC=local
    ldap.****ls1ch.group.request: (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={dn}))

    ldap.****ls2ch.url: ldaps://****ls2.***********:636/
    ldap.****ls2ch.user.baseDn: OU=Users,OU=Accounts,OU=CH,DC=****,DC=local
    ldap.****ls2ch.user.request: (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={login}))
    ldap.****ls2ch.group.baseDn: OU=****,OU=Groups,OU=CH,DC=****,DC=local
    ldap.****ls2ch.group.request: (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={dn}))
    ldap.****ls1vn.url: ldaps://****ls1.***********:636/
    ldap.****ls1vn.user.baseDn: OU=****-VN,OU=Accounts,OU=VN,DC=****,DC=local
    ldap.****ls1vn.user.request: (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={login}))
    ldap.****ls1vn.group.baseDn: OU=****,OU=Groups,OU=CH,DC=****,DC=local
    ldap.****ls1vn.group.request: (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={dn}))

    ldap.****ls2vn.url: ldaps://****ls2.***********:636/
    ldap.****ls2vn.user.baseDn: OU=****-VN,OU=Accounts,OU=VN,DC=****,DC=local
    ldap.****ls2vn.user.request: (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={login}))
    ldap.****ls2vn.group.baseDn: OU=****,OU=Groups,OU=CH,DC=****,DC=local
    ldap.****ls2vn.group.request: (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={dn}))

    ldap.****ls1svc.url: ldaps://****ls1.***********:636/
    ldap.****ls1svc.user.baseDn: OU=****,OU=****,OU=Accounts,OU=CH,DC=****,DC=local
    ldap.****ls1svc.user.request: (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={login}))
    ldap.****ls1svc.group.baseDn: OU=****,OU=Groups,OU=CH,DC=****,DC=local
    ldap.****ls1svc.group.request: (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={dn}))

    ldap.****ls2svc.url: ldaps://****ls2.***********:636/
    ldap.****ls2svc.user.baseDn: OU=****,OU=****,OU=Accounts,OU=CH,DC=****,DC=local
    ldap.****ls2svc.user.request: (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={login}))
    ldap.****ls2svc.group.baseDn: OU=****,OU=Groups,OU=CH,DC=****,DC=local
    ldap.****ls2svc.group.request: (&(objectClass=group)(member:1.2.840.113556.1.4.1941:={dn}))

Bind DN and password are passed via secret

  env:
    - name: SONAR_UPDATECENTER_ACTIVATE
      value: "false"
    - name: LDAP_****1****BINDDN
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDDN
    - name: LDAP_****1****BINDPASSWORD
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDPASSWORD
    - name: LDAP_****2****BINDDN
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDDN
    - name: LDAP_****2****BINDPASSWORD
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDPASSWORD
    - name: LDAP_****1****BINDDN
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDDN
    - name: LDAP_****1****BINDPASSWORD
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDPASSWORD
    - name: LDAP_****2****BINDDN
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDDN
    - name: LDAP_****2****BINDPASSWORD
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDPASSWORD
    - name: LDAP_****1SVC_BINDDN
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDDN
    - name: LDAP_****1SVC_BINDPASSWORD
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDPASSWORD
    - name: LDAP_****2SVC_BINDDN
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDDN
    - name: LDAP_****2SVC_BINDPASSWORD
      valueFrom:
        secretKeyRef:
          name: "sonarqube-ldap"
          key: LDAP_BINDPASSWORD

We also have CA certficate in order to connect via LDAPS

  caCerts:
    enabled: true
    image: r-docker-registry-1-docker-io.*********/eclipse-temurin:11-jdk-alpine
    secret: ldap-ca-certificates

SonarQube is able to add the certficate via init container without issue


Certificate was added to keystore
Certificate was added to keystore
Certificate was added to keystore
Certificate was added to keystore

SonarQube is able to connect to all LDAP server

2024.04.18 08:14:27 INFO web[][o.s.a.l.LdapContextFactory] Test LDAP connection on ldaps://****s1.****:636/: OK
2024.04.18 08:14:27 INFO web[][o.s.a.l.LdapContextFactory] Test LDAP connection on ldaps://****s2.****:636/: OK
2024.04.18 08:14:27 INFO web[][o.s.a.l.LdapContextFactory] Test LDAP connection on ldaps://****s1.****:636/: OK
2024.04.18 08:14:27 INFO web[][o.s.a.l.LdapContextFactory] Test LDAP connection on ldaps://****s2.****:636/: OK
2024.04.18 08:14:27 INFO web[][o.s.a.l.LdapContextFactory] Test LDAP connection on ldaps://****s1.****:636/: OK
2024.04.18 08:14:27 INFO web[][o.s.a.l.LdapContextFactory] Test LDAP connection on ldaps://****s2.****:636/: OK

The logs don’t say much, except that the user is not found

2024.04.18 08:19:51 TRACE web[2eb35602-2443-4da2-b819-44c56ca9a1ac][o.s.s.p.w.UserSessionFilter] Thread[http-nio-0.0.0.0-9000-exec-3,5,main] serves /api/authentication/login
2024.04.18 08:19:51 TRACE web[2eb35602-2443-4da2-b819-44c56ca9a1ac][sql] time=4ms | sql=SELECT sa.scm_account as "scm_account", u.uuid as uuid, u.login as login, u.name as name, u.email as email, u.active as "active", u.salt as "salt", u.crypted_password as "cryptedPassword", u.hash_method as "hashMethod", u.external_id as "externalId", u.external_login as "externalLogin", u.external_identity_provider as "externalIdentityProvider", u.user_local as "local", u.reset_password as "resetPassword", u.homepage_type as "homepageType", u.homepage_parameter as "homepageParameter", u.last_connection_date as "lastConnectionDate", u.last_sonarlint_connection as "lastSonarlintConnectionDate", u.created_at as "createdAt", u.updated_at as "updatedAt" FROM users u left outer join scm_accounts sa on sa.user_uuid = u.uuid WHERE u.login=? AND u.active=true | params=******
2024.04.18 08:19:51 DEBUG web[2eb35602-2443-4da2-b819-44c56ca9a1ac][o.s.a.l.LdapSearch] Search: LdapSearch{baseDn=OU=Users,OU=Accounts,OU=CH,DC=******,DC=local, scope=subtree, request=(&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={0})), parameters=[******], attributes=null}
2024.04.18 08:19:51 DEBUG web[2eb35602-2443-4da2-b819-44c56ca9a1ac][o.s.a.l.LdapContextFactory] Initializing LDAP context {java.naming.referral=follow, java.naming.security.principal=, com.sun.jndi.ldap.connect.pool=true, java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory, java.naming.provider.url=ldaps://dc******s1.******.local:636/, java.naming.security.authentication=simple}
2024.04.18 08:19:51 DEBUG web[2eb35602-2443-4da2-b819-44c56ca9a1ac][o.s.a.l.DefaultLdapAuthenticator] User ****** not found in server <dc******s1ch>: javax.naming.NamingException: [LDAP: error code 1 - 000004DC: LdapErr: DSID-0C090CE5, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, v4563

The same configuration works with older SonarQube version.

Did anything change on Helm config or SonarQube related to authentication ?

I’m out of ideas.

Regards,

Colin · April 18, 2024, 8:43am

Hey @jonesbusy

Thanks for the report. It looks like we touched the code that handles LDAP auth (which doesn’t happen often) between 10.4 and 10.5 (here and here, the latter commit specifically touching user mapping). Beyond that, I can’t say for sure what could be the problem.

I’ve flagged this for attention.

Antoine · April 19, 2024, 1:32pm

Hey @jonesbusy,

We tried to reproduce the issue internally and we couldn’t, we used a similar config that what you have, ie. ldaps, multiple servers, custom user requests, etc. it worked fine all the time.
So we’ll have to better understand your particular case as it seems specific.

I know you said that this started happening when you moved to 10.5, but still there are a few things you should check on your side.

The root error message is not about the user being not found, but about:

[LDAP: error code 1 - 000004DC: LdapErr: DSID-0C090CE5, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, v4563

This means that the binding query with the server didn’t work (ie. the user request wasn’t even executed)

Indeed the config output you shared doesn’t have binding properties (ldap.bindDn and ldap.bindPassword) while you have them in the yaml. Have you just removed them or are they missing completely?
If they are missing, then it might be a secret issue in the helm deployment.
And before you ask: yes the Test LDAP connection succeeds even with wrong binding values, as long as the URL is valid.

I hope this will help.

Cheers
Antoine

jonesbusy · April 19, 2024, 1:53pm

Hi,

Thanks for your answer. The binddn and password are stored on a secret ‘sonarqube-ldap’.

Yes it’s started to fail between SonarQube 10.4.1 and 10.5.0.

Perhaps I didn’t mention that I also upgradde the chart from 10.4.1+2389 to 10.5.0+2748.

I double check the secret is identical to our prod and staging environment.

Then we mount env vars via the values files (like on my previous message) for each LDAP server (It’s the same username/password for each LDAP).

  - name: LDAP_****2****BINDPASSWORD
    valueFrom:
      secretKeyRef:
        name: "sonarqube-ldap"
        key: LDAP_BINDPASSWORD

jonesbusy · April 19, 2024, 1:54pm

Potential issue: SONAR-20266 fix duplicated env var caused by templating · SonarSource/helm-chart-sonarqube@f4dc8f7 · GitHub

Before the env vars were passed “as yaml”. Now there is some kind of filtering done…

jonesbusy · April 19, 2024, 2:07pm

Reading at the code it assume all env var are passed with “name/value”… But doesn’t consider env vars passed via secret like I do. Which is Kubernetes standard.

Checking on the pod I see empty values instead of the username/password

LDAP_****_BINDDN=
LDAP_***_BINDDN=

I tried to rollback the chart version to 10.4.1+2389 but keep using 10.5.0 container and it works. My env var are there on the pod and I can connect using LDAP.

So the chart is the problem.

Thanks

Antoine · April 19, 2024, 2:16pm

There we go, it’s easy to solve issues with good investigators like you Thanks for these tests, it confirms from where the issue comes from.

@jeremy.cotineau as you submitted these changes, could you have a look as to why passing secrets doesn’t work anymore? Thanks

Cheers

jeremy.cotineau · April 23, 2024, 10:10am

Huge thanks again to @jonesbusy for raising the bug so promptly.

The fixed has been merged in this PR and will also be part of the 10.5.1 patch release.

system · May 3, 2024, 7:54am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.