Prometheus scraping of SonarQube is failing with 403

I’m using SonarQube 10.0-enterprise installed on Kubernetes using Helm. I am trying to have Prometheus scrape SonarQube, as described in the Monitoring section here, but Prometheus is getting 403s.

The behavior can be reproduced with curl like so:

$ curl -i -H "Authorization: Bearer <token>"  http://<pod IP>:9000/api/monitoring/metrics
HTTP/1.1 403
vary: accept-encoding
Content-Type: application/json
Transfer-Encoding: chunked
Date: Tue, 25 Apr 2023 16:49:56 GMT

{"errors":[{"msg":"Insufficient privileges"}]}

I have:

  • set the sonar.web.systemPasscode
  • verified sonar.web.systemPasscode appears in the sonarqube-sonarqube-config ConfigMap
  • verified the token in my curl command matches the value of sonar.web.systemPasscode
  • verified I can access /api/monitoring/metrics as a logged-in user in my web browser, so the endpoint is working at least
  • have tried setting sonar.forceAuthentication to both true and false

I do see requests to /api/monitoring/metrics in access.log returning 403:

10.22.91.3 - - [25/Apr/2023:17:15:29 +0000] "GET /api/monitoring/metrics HTTP/1.1" 403 - "-" "Prometheus/2.37.5" "AYe46K1E7nc2tr02AAqE"
10.100.199.32 - - [25/Apr/2023:17:15:50 +0000] "GET /api/monitoring/metrics HTTP/1.1" 403 - "-" "curl/7.87.0" "AYe46K1E7nc2tr02AAqJ"

However, I do not see any evidence of requests to /api/monitoring/metrics in web.log, even though log level is set to TRACE.

At this point, I don’t know what else to try. Any help would be greatly appreciated.

1 Like

Hi,

Are you sure the token you’re using has permissions to the services you’re trying to call?

 
Ann

Ann,
Thanks for your quick response!

Sorry, I’m new to SonarQube – is this supposed to be a user token? The documentation makes no mention of this. I thought the whole point of sonar.web.systemPasscode was to disassociate the token from a user to prevent privilege-escalation attacks, especially since this token is in plain-text in the ConfigMap!

If this is supposed to be a user token, what is the best-practice here? Create a user with no permissions and use a token from that user?

If this is not supposed to be a user token, then how do I create this token and assign it the correct permissions?

Thanks,
Fred

Hi Fred,

In fact, sonar.web.systemPasscode is something that slipped by me. I’m going to flag this for more expert eyes.

 
Ann

Thank you!

Warm welcome @fbierhaus ,

Thank you for your post. From the information you provided it seems like SonarQube doesn’t recognize the sonar.web.systemPasscode that you are passing in curl or Prometheus.

To confirm that, could you check if you receive 403 status code also for the request to another endpoint that allows system passcode, in example GET api/system/health ? This is needed to make sure we focus on the right problem.

I have two more questions:

  • verified the token in my curl command matches the value of sonar.web.systemPasscode
  1. Do you mean that you verified that equality of the passcode between curl and what is inside sonar.properties ?

  2. Could you check in your web.log when starting SonarQube whether a line like this is logged?

INFO web[o.s.s.u.SystemPasscodeImpl] System authentication by passcode is enabled

Thank you in advance for this additional information.

Hmm, I get 401 for /api/system/health:

$ curl  -i -H 'Authorization: Bearer <token>'  http://<pod IP>:9000/api/system/health
HTTP/1.1 401
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'; base-uri 'none'; connect-src 'self' http: https:; img-src * data: blob:; object-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; worker-src 'none'
X-Content-Security-Policy: default-src 'self'; base-uri 'none'; connect-src 'self' http: https:; img-src * data: blob:; object-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; worker-src 'none'
X-WebKit-CSP: default-src 'self'; base-uri 'none'; connect-src 'self' http: https:; img-src * data: blob:; object-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; worker-src 'none'
Content-Length: 0
Date: Thu, 27 Apr 2023 18:32:21 GMT

Here are the logs for that request:

2023.04.27 18:32:21 TRACE web[AYfD9GvKqB7Go6W1AADH][o.s.s.p.w.UserSessionFilter] Thread[http-nio-0.0.0.0-9000-exec-7,5,main] serves /api/system/health
2023.04.27 18:32:21 TRACE web[AYfD9GvKqB7Go6W1AADH][sql] time=1ms | sql=SELECT t.uuid as "uuid", t.user_uuid as "userUuid", t.name as "name", t.token_hash as "tokenHash", t.last_connection_date as "lastConnectionDate", t.created_at as "createdAt", t.project_key as "projectKey", t.type as "type", t.expiration_date as "expirationDate", p.name as "projectName", p.uuid as "projectUuid" FROM user_tokens t LEFT JOIN projects p on t.project_key = p.kee WHERE t.token_hash=? | params=fd3d752e61d244bc3776e794a4d65f3de2f3fdd4ebf18cde4a46d153f4f1947f6c8313919bfec1f1edf9b8407cab50d9
2023.04.27 18:32:21 DEBUG web[AYfD9GvKqB7Go6W1AADH][auth.event] login failure [cause|Token doesn't exist][method|SONARQUBE_TOKEN][provider|LOCAL|local][IP|10.100.199.214|][login|]

Looks like it is trying to query the db for a value specified in a properties file?

To answer your other questions:

  1. Yes, that’s what I mean – the Bearer token used by curl matches the value in sonar.properties.

  2. Yes, I see the following line in web.log:

$ grep "System authentication by passcode is enabled" web.log
2023.04.27 18:22:24 INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is enabled

Finally, for completeness, here are the logs for a /api/monitoring/metrics request:

2023.04.27 18:34:16 TRACE web[AYfD9GvKqB7Go6W1AADm][o.s.c.p.PriorityBeanFactory] Returning cached instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@324a0017-org.sonar.db.DBSessionsImpl'
2023.04.27 18:34:16 TRACE web[AYfD9GvKqB7Go6W1AADm][o.s.c.p.PriorityBeanFactory] Returning cached instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@324a0017-org.sonar.server.setting.ThreadLocalSettings'
2023.04.27 18:34:16 TRACE web[AYfD9GvKqB7Go6W1AADm][o.s.c.p.PriorityBeanFactory] Returning cached instance of singleton bean 'jdk.internal.loader.ClassLoaders$AppClassLoader@324a0017-org.sonar.server.authentication.UserSessionInitializer'
2023.04.27 18:34:16 TRACE web[AYfD9GvKqB7Go6W1AADm][o.s.s.p.w.UserSessionFilter] Thread[http-nio-0.0.0.0-9000-exec-1,5,main] serves /api/monitoring/metrics
2023.04.27 18:34:16 TRACE web[AYfD9GvKqB7Go6W1AADm][sql] time=1ms | sql=select gr.role from group_roles gr where gr.component_uuid is null and gr.group_uuid is null

Thank you so much for your help!

1 Like

Hi again,

We are still looking into this.
Meanwhile could you:

  1. Try to replace Authentication: Bearer with the header X-Sonar-Passcode and let us know if this changes anything?
  2. Let us know if there is maybe any proxy (like Nginx in example) between you (well… curl) and SonarQube?

Thank you.

There is no proxy in the middle. In fact, to eliminate any possibility of contamination, here is the request when shelled into the SonarQube container directly:

$ wget -SqO- --header="X-Sonar-Passcode: <sonar.web.systemPasscode>" http://localhost:9000/api/monitoring/metrics
  HTTP/1.1 403
  vary: accept-encoding
  Content-Type: application/json
  Transfer-Encoding: chunked
  Date: Fri, 28 Apr 2023 16:07:37 GMT
  Keep-Alive: timeout=60
  Connection: keep-alive

As you can see X-Sonar-Passcode yields the same result.

Oh, BTW, when I call /api/monitoring/metrics with a user token, it works. So the issue appears to be limited to sonar.web.systemPasscode.

Any update on this? Is this a bug introduced in 10.0?

Thanks,
Fred

I was able to get the metrics by using curl -u like this:

curl -u yourtokenhere: https://sonarqube.example.com/api/monitoring/metrics

The token was created with admin user and is configured in config/sonar.properties like such:

sonar.web.systemPasscode=yourtokenhere
2 Likes