I recently tried upgrading out sonarqube server from 8.9 LTS to 9.9 LTS (zip file deployment method). The server ran through its migrations and came online, but the quality gates tab of the web ui was unable to load and threw an empty String error:
I also tried running our pipeline workflows against the upgraded server, and hit similar issues with qualitygate-related APIs returning similar empty String errors. I enabled trace-level logging on the server and tried clicking onto the quality gates tab again a couple times, but I didnât see any errors that were obviously related to the issue Iâm describing here. I am seeing quite a few connection refused errors in sonar.log:
2023.04.16 16:50:27 TRACE app[][tracer] curl -iX GET 'http://127.0.0.1:9001/'
2023.04.16 16:50:27 DEBUG app[][o.e.c.RestClient] updated [[host=http://127.0.0.1:9001]] already in blacklist
2023.04.16 16:50:27 DEBUG app[][o.a.h.i.n.c.MainClientExec] [exchange: 8] start execution
2023.04.16 16:50:27 DEBUG app[][o.a.h.c.p.RequestAddCookies] CookieSpec selected: default
2023.04.16 16:50:27 DEBUG app[][j.io.serialization] Builtin factory: null -> new: null
2023.04.16 16:50:27 DEBUG app[][o.a.h.c.p.RequestAuthCache] Re-using cached 'basic' auth scheme for http://127.0.0.1:9001
2023.04.16 16:50:27 DEBUG app[][o.a.h.c.p.RequestAuthCache] No credentials for preemptive authentication
2023.04.16 16:50:27 DEBUG app[][o.a.h.i.n.c.InternalHttpAsyncClient] [exchange: 8] Request connection for {}->http://127.0.0.1:9001
2023.04.16 16:50:27 DEBUG app[][o.a.h.i.n.c.PoolingNHttpClientConnectionManager] Connection request: [route: {}->http://127.0.0.1:9001][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 30]
2023.04.16 16:50:27 DEBUG app[][o.a.h.i.n.c.PoolingNHttpClientConnectionManager] Connection request failed
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.base/java.lang.Thread.run(Thread.java:833)
But there arenât any of these errors logged during the timeframe where I was trying to observe this quality gates error in the logs (I was testing and looking for logged events in this timeframe: from 2023.04.16 16:39:30 to 2023.04.16 16:40:00).
The only thing I can think of that is non-standard about our server setup is that weâre using the Qualinsight code smells plugin (v4.0.0). So, I tried rolling back to a pre-upgrade database snapshot and reverting the server setup to 8.9 LTS, and I verified that the plugin was loaded and things were behaving normally. Next, I turned off the sonar server, removed the plugin jar file from the extensions/plugins directory, and turned the server back on and verified that the plugin was no longer loaded. I then tried upgrading this setup to 9.9 LTS, but again hit the same issue with the quality gates tab not loading and related APIs returning an empty String error.
Iâve attached trimmed logs from the tested timeframe, Iâve omitted the ce.log file because itâs extremely large but if you think it would be helpful I can find a way to provide it. access.log (7.4 KB) es.log (219.5 KB) sonar.log (11.7 KB) web.log (869.8 KB)
Thanks in advance for any assistance or insight youâre able to provide.
I believe thatâs just from me shutting down the server after I validated that the upgrade path was still broken for our instance. I donât believe it has anything to do with the issue Iâm trying to resolve here, I was just trying to note that I had observed those errors in the logfile. I enabled trace level logging to try to figure out what the issue with the quality gates view/api was, after seeing nothing useful logged at the debug level.
I spun up a staging server using an image of our current main server + a snapshot of its database (running sonarqube 8.9), ran through the upgrade to 9.9, and reproduced the issue with the quality gates tab. Here are all the log files:
I am trying to reproduce the issue you are having in my environment.
If you have access to the database could you run these 2 queries and share the results here? They will not contain any confidential data, but if you think so, feel free to remove any confidential information from the results:
select * from quality_gates
select text_value from properties where âqualitygate.defaultâ = prop_key
The results will help with the investigation. Thank you in advance.
and hit similar issues with qualitygate-related APIs returning similar empty String errors
At the same time, could you please provide some concrete examples of other /api/qualitygates/* endpoints that also return HTTP Status code 400? This will help us to narrow down the issue.
Hereâs the results of the first query: sonarqube-qg.txt (164.8 KB)
Redacted the names of the quality gates since most are named after corresponding projects, just to be on the safe side.
The second query returned zero results.
I did some more digging and, at least among the GET qualitygate methods, it seems that list is actually the only one thatâs failing, my mistake in assuming it was all of them. get_by_project, project_status, search, and show all worked as expected. I didnât try any of the POST methods via curl, but I was able to navigate to the page of an individual quality gate via the corresponding projectâs webpage, and it mostly loaded as expected and I was able to update conditions on it. The only part that didnât load was the list of other QGs on the side of the page, and again I saw that empty String banner pop up at the top of the page. So it seems that only the list method is impacted, based on those findings.
Thank you very much for the additional information that you provided. There is one thing that worries me a lot, namely somehow you have duplicated value in the uuid column of quality_gates table. Because of that SonarQube can behave unpredictably. Normally this should never happen as there should be unique constraint on this column (primary key).
The duplicated uuid is your default quality gate: AXJD2TGSPWUcWP28Tcjv.
To find out more about the issue we should:
Check if the duplication in the uuid column was already there on the 8.9 backup of your database
Check if the unique constraint on the uuid column of the quality_gates table was there on the 8.9 backup
Double check that this constraint doesnât exist in your database after the upgrade to 9.9
Are the values in name column the same for this duplicated uuid ? Does any of these values ring any bells to you? Does any of these values seem obsolete?
What DB vendor are you using?
This could be a symptom that your database schema is not in a great shape after the upgrade and we need to do a little bit more digging to try to get closer to the answer âwhyâ.
I see the same duplication in our main instance which is currently running 8.9
on the 8.9 instance:
sonar=> \d+ quality_gates;
Table "public.quality_gates"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
-------------+-----------------------------+-----------+----------+---------+----------+-------------+--------------+-------------
name | character varying(100) | | not null | | extended | | |
created_at | timestamp without time zone | | | | plain | | |
updated_at | timestamp without time zone | | | | plain | | |
is_built_in | boolean | | not null | | plain | | |
uuid | character varying(40) | | not null | | extended | | |
Indexes:
"pk_quality_gates" PRIMARY KEY, btree (uuid)
Access method: heap
on the 9.9 instance:
sonar=> \d+ quality_gates;
Table "public.quality_gates"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
-------------+-----------------------------+-----------+----------+---------+----------+-------------+--------------+-------------
name | character varying(100) | | not null | | extended | | |
created_at | timestamp without time zone | | | | plain | | |
updated_at | timestamp without time zone | | | | plain | | |
is_built_in | boolean | | not null | | plain | | |
uuid | character varying(40) | | not null | | extended | | |
Indexes:
"pk_quality_gates" PRIMARY KEY, btree (uuid)
Access method: heap
I donât see any unique constraint on either version. If thereâs another command that would yield more useful info, please let me know.
One of the entries w/ duplicate uuid is a QG for one of our projects, the other is named âSonar way Updatedâ. Iâd imagine that they are both user created but I canât say for sure. If you think it would help, I could try to create another copy of our 8.9 instance, delete one or both of those QGs, and then run the update to 9.9 and see if that changes anything.
Weâre using postgresql, version 14.4 on aws rds. We recently updated from an older version, 10.x I think, I donât remember the exact minor version. That database version update was done prior to the sonarqube version update attempt
Just to see, I tried destroying one of the offending entries via the api and restarted the server (this being on our 9.9 staging instance), and unfortunately itâs still unable to list quality gates
I am getting closer to finding out the root cause of the issue. I managed to reproduce the exact issue by modifying data in my database.
It seems to me for now that the issue is caused by empty (or less likely null) value in the quality_gate_conditions.value_error column in your database.
Could you:
Share the results of the following query: select uuid, value_error from quality_gate_conditions where qgate_uuid='AXJD2TGRPWUcWP28Tcir' order by created_at asc
Search the column quality_gate_conditions.value_error for any values that donât look like numbers (including nulls or empty strings). You can achieve that in postgres in example by running: select uuid, qgate_uuid from quality_gate_conditions where value_error !~ '^[-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?$';
sonar=> select uuid, value_error from quality_gate_conditions where qgate_uuid='AXJD2TGRPWUcWP28Tcir' order by created_at asc;
uuid | value_error
----------------------+-------------
AXolbh7801QwETeDn-qh | 5
AXolbh7801QwETeDn-qG | 75
AXolbh7801QwETeDn-qs | 95
AXolbh6x01QwETeDn-b2 |
AXolbh8V01QwETeDn-2u | 0
AYSJ23PBhPMgR6WhlGro | 95
(6 rows)
sonar=> select uuid, qgate_uuid from quality_gate_conditions where value_error !~ '^[-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?$';
uuid | qgate_uuid
----------------------+----------------------
AXolbh6x01QwETeDn-b2 | AXJD2TGRPWUcWP28Tcir
(1 row)
Thanks for including the exact queries, I appreciate it. I modified the second query to include returning the value_error, and it looks like it has an empty value as you described:
It is hard to say, without knowing the whole history of your usage of SonarQube how this value over there got emptied. As a temporary fix you can put over there 0 and your quality profiles page should be unblocked.
If this is true, then we can work together to understand what the expected value should be there.
Ok, looks like that was the problem, things seem to be working now on the 9.9 staging server! Thanks so much for your help Ĺukasz and Ann, this has been driving me crazy for a while. I have no clue what happened to cause that empty cell, but I might have to do some access review on our setup just in case. Iâm not too worried about fixing the value, it looks like it belongs to a QG that was only ever used for testing (maybe some test managed to mangle the entry?). Anyway, thanks again for your help
