SAML plugin sets incorrect port during callback

We run SonarQube behind an Nginx reverse proxy and use the SAML 2.0 plugin for authentication.
On login with SAML, everything works fine up until the callback.

The callback request URL is https://<sonarqube-url>/oauth2/callback/saml, which is then redirected to https://<sonarqube-url>:8443, instead of the expected https://<sonarqube-url>.

Since we run the whole setup on a Kubernetes cluster, all requests to port 443 are forwarded to the nginx container on port 8443. We believe that the SAML plugin incorrectly takes this container port, instead of the actually exposed port when redirecting after the successful callback.

It is not possible for us to run the nginx container on port 443.

As a workaround we have exposed port 8443, however we would love to see this fixed or have some additional settings added where we could override this behavior.

Hi @dominikm,

Have you correctly configured the setting “sonar.core.serverBaseURL” ?
Because the plugin is using this URL to generate the callback URL.

Regards,
Julien Lancelot

Yes I have. It is set to https://<sonarqube-url>.

Then I think you need to configure your reverse proxy, as neither the SAML authentication or SonarQube will change the URL by appending the port 8443.

Thank you for the clarification on how the plugin works. I have looked into our nginx configuration, but I don’t see any obvious configuration error.

Here is the config:

server {
    listen 8080;
    return 301 https://$host$request_uri;
}
server {
    server_name <sonarqube-url>;
    listen 8443;
    ssl on;
    ssl_certificate    ...;
    ssl_certificate_key ...;

    location / {
        client_max_body_size 100m;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass http://<internal-sonarqube-server>:9000;
    }
}

Anybody has some ideas on what might be going wrong?

Hi Dominik,

Quick clarification on your setup:

Since we run the whole setup on a Kubernetes cluster, all requests to port 443 are forwarded to the nginx container on port 8443. We believe that the SAML plugin incorrectly takes this container port, instead of the actually exposed port when redirecting after the successful callback.

I take it this means the you port-forward 8443 to 443 on the container? So, if this were Docker, you would -p 8443:443, correct?

If correct, how is SonarQube run? I take it it’s in its own container, exposing port 9000? Are the NginX and SQ proxy on a shared, specific network (in the Docker world, this would be linking containers together)? Or are they both on a “global” internal network (like a DMZ)?

Lastly, I imagine SQ is configured as a SP, and the URL (entityId?) on the IdP side is correctly pointing to the public HTTPS address (meaning, it would route traffic through Kubernetes, and not directly to the NginX container)?

Hi Wouter,

it’s just the other way around, sorry if I was unclear above. Since the Nginx container cannot bind to ports below 1024 without root, the container runs on 8080 and 8443. So if this were Docker, it would be -p 443:8443 -p 80:8080.

SonarQube is run in its own container, exposing port 9000 only locally, not it is not accessible from the outside. It lives in the same network as Nginx, so Nginx has access to SonarQube. Just like in a Docker network, the service names of the containers act as hostnames in the network.

So, when accessing SonarQube from the outside, this is basically what happens:
https://<sonarqube-url> → port forwarding to 8443 → proxied to http://<internal-sonarqube>:9000

Lastly, I imagine SQ is configured as a SP, and the URL (entityId?) on the IdP side is correctly pointing to the public HTTPS address (meaning, it would route traffic through Kubernetes , and not directly to the NginX container)?

That is correct, yes.

Hi, thanks for the clarification.

That’s odd. I wonder where the port 8443 gets picked up then, and by whom. As far as I know, the $host variable doesn’t contain any port (right?), so:

proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;

would not pass NginX’s port to SQ. Say, if the <sonarqube-url> is sq.example.com, the HTTP headers would look like this:

X-Forwarded-Host sq.example.com
X-Forwarded-Server sq.example.com

Furthermore, sonar.core.serverBaseURL=https://sq.example.com, so I would be inclined to think the issue with the redirect is not on SQ’s side. I don’t see how SQ can even know that there’s a non-standard port involved.

It’s hard to know without actually seeing the entire setup :slight_smile: . But I would imagine the issue comes from either NginX, when it processes the redirect (which should flow through it again), or from your SAML IdP.

The typical workflow of an auth request would be:

  1. User accesses the App
  2. NginX handles the request
  3. NginX passes the request to SQ
  4. SQ prepares the SAML data, and redirects to the IdP for authentication
  5. Request passes through NginX again – This is probably where the problem starts
  6. Request is handled by the IdP
  7. IdP redirects back to the SP
  8. NginX get the request, passes it to SQ again
  9. SQ gets the SAML data, authenticates the user
  10. SQ redirects to itself (like, home page)
  11. Request goes through NginX again
  12. NginX forwards the request to SQ
  13. Done

But to make matters even more complex, here, you have Kubernetes in front of the whole shebang :slight_smile: .

Points I would investigate:

  1. Where is the final redirect going to (after successful auth)? If, until that point, the port was not in the URL yet, it might be on NginX’s side (redirecting to itself, like Location /projects, bypassing Kubernetes?)
  2. How is the request passed from SQ -> NginX -> ?Kubernetes? -> IdP ? I could imagine the port is added there at some point.

Let me know how it goes :slight_smile:.

It sure is strange!
I recorded the network log from the point I started the SAML request up to the final redirect after the authentication was successful and the port gets in. In short, this is how the flow looks like:

  1. https://sq.example.com/sessions/init/saml?return_to=%2f
    • 302 redirected to https://idp.example.com/...
    • goes through Kubernetes to Nginx of IdP
    • IdP Nginx handles and passes request to IdP
  2. https://idp.example.com/.../login?...
    • normal login to the IdP
    • after login redirects to SQ callback
  3. https://sq.example.com/oauth2/callback/saml
    • callback handles SAML data and authenticates the user
    • 302 redirected to https://sq.example.com:8443

I can replicate this final redirect by accessing /oauth2/callback/saml outside of this flow. So if I just access the above URL as is, I also get redirected to https://sq.example.com:8443, which is why we first suspected the plugin as the culprit. However, as you correctly pointed out, it doesn’t really make sense. Also, when looking at the response headers for this final redirect, the Server header is set to nginx/1.12.2, so it would make sense that the problem is with Nginx somewhere.

My only problem is: why doesn’t the port appear on other SQ URLs? Why does this only happen in this one case?

Altough the port does not appear until the very final callback, I could imagine that maybe the Nginx of our IdP (which as you guessed correctly is also running on Kubernetes) is somehow responsible by setting a strange header? I’m totally shooting in the dark here, but since the SQ and the SQ Nginx setup looks correct, I really don’t know where else to look.

I saw that there are also plugins for GitHub and Azure Active Directory. I will try to setup one of these and see if I can replicate the problem. If the problem appears again it should be a very strong indicator that the problem is with our Nginx. However, if everything works fine, then the problem is somewhere else. I will report back once I tried it out!

Yes, please do :-). I’m interested to know what triggers this. Also, could this be related: https://github.com/kubernetes/ingress-nginx/issues/1798 ?

I set up GitHub authentication and was able to replicate the problem, so it has definitely nothing to do with our IdP setup. The problem has to be with SQ’s Nginx somewhere.

Thank you very much for sharing this! I only took a quick look for now, but it might be related. I will take a closer look tomorrow and keep you updated :slight_smile:

Good news: I finally managed to solve it! It was indeed an incorrect Nginx configuration

I looked at the GitHub issue that you shared but realized that unfortunately it does not relate to our problem. By chance I found this beautiful web tool called nginxconfig.io. I entered my desired configuration, looked at the output and copied the relevant parts to my config, deployed it, and it worked. So yeah…

In case someone else stumbles across this problem, here is my final configuration that solved the problem:

server {
    listen 8080;
    return 301 https://$host$request_uri;
}
server {
    server_name sq.example.com;
    listen 8443;
    ssl on;
    ssl_certificate     /path/to/cert.crt;
    ssl_certificate_key /path/to/cert.key;

    access_log /bitnami/nginx/logs/sonarqube_access.log;
    error_log /bitnami/nginx/logs/sonarqube_error.log;
    ssl_session_timeout  5m;

    location / {
        client_max_body_size 100m;
        proxy_http_version	1.1;
        proxy_cache_bypass	$http_upgrade;

        proxy_set_header Upgrade             $http_upgrade;
        proxy_set_header Connection          "upgrade";
        proxy_set_header Host                $host;
        proxy_set_header X-Real-IP           $remote_addr;
        proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto   https;
        proxy_set_header X-Forwarded-Host    $host;
        proxy_set_header X-Forwarded-Port    $server_port;
        proxy_pass                           http://internal-sonarqube:9000;
    }
}

I’m still not exactly sure which header was the culprit, but I have a strong feeling that it has to do with X-Forwarded-Port.

Thanks @Wouter_Admiraal for your help with this issue!

2 Likes

Thanks @dominikm for sharing the result! I’m sure this will be helpful to others in the future.