Projects not listed after on-premises Enterprise server migration and version upgrade

Current State

Recently due to company divestiture activities, we have had to migrate data from our existing SonarQube Enterprise server (9.2.4) to a newly stood-up Enterprise server (9.9 LTS). During the migration, the new server was setup first with 9.9 LTS before the data was moved over from the existing server.

Now, on the new server, we cannot see the projects listed under the Projects tab, even after running a new baseline pipeline build. If I as user with Administrator privileges goes to the Projects Management list (Administration/Projects/Management), I can see all of the projects listed.

What we have tried

  1. Clearing search results as recommended here - Stack Overflow, here - SQ Community, and here - SQ Community.
  2. Building baseline and pull request builds within our Azure DevOps (on-prem) pipelines. The builds go off fine and upload results fine, and I can see the updates happening in the Projects Management list, but they do not show up on the Projects list.
  3. Deleting a project fully and re-scanning. This does allow for the project to show up, but then loses all of the history and issue resolution and therefore is not a desirable outcome.

Relevant Version Information

Existing Server

Server version: 9.2.4.50792

System
Docker No
External User Authentication LDAP
High Availability No
Official Distribution Yes
Force authentication No
Home Dir C:\Program Files\SonarQube\sonarqube-9.2.4.50792
Data Dir D:/SonarData
Temp Dir D:/SonarTemp
Processors 4
Database
Database Microsoft SQL Server
Database Version 13.00.6419
Driver Microsoft JDBC Driver 9.2 for SQL Server
Driver Version 9.2.0.0
New Server

Server version: 9.9.0.65466

System
Edition Enterprise
Docker No
High Availability No
Official Distribution Yes
Force authentication Yes
Home Dir E:\SonarQube
Data Dir E:\SonarQube\var\sonarqube\data
Temp Dir E:\SonarQube\var\sonarqube\temp
Processors 4
Database
Database Microsoft SQL Server
Database Version 16.00.1000
Driver Microsoft JDBC Driver 11.2 for SQL Server
Driver Version 11.2.2.0
Default transaction isolation TRANSACTION_READ_COMMITTED

Path forward?

Any further recommendations would be very much appreciated. I’d really prefer to not have to delete all the projects, re-scan, and then have to re-do all of the previously resolved issues.

Hi,

How did you migrate those projects?

 
Ann

Fair question, I have that back to the person that did the migration and will update here once he gets in and gets that answered. I do know that the server was stood up first with v9.9 before the data was brought over, rather than copying the installation over and then doing upgrade processes (which is the mechanism that I recommended).

1 Like

@ganncamp

Process was that the new server was stood up fresh with 9.9 and the other attendant software (MS SQL, etc.). Once it was created, the database from the original server was essentially copy/pasted to the new server.

Hi,

So by copy/pasted, I’m going to read that as basically being a DB backup restore. And that’s a decent way to do it.

Nothing obvious is jumping out at me, so it’s time to see what your server logs say.

 
Ann

1 Like

I got a copy of the logs and took an initial look through. Nothing super obvious is jumping out at me beyond some expired access tokens that I have now fixed, but were definitely not the cause (fixing the PATs didn’t change the project listing).

One other odd behavior that I’m seeing is that on the original server prior to migration, I had made a Portfolio to group together one program’s projects. On that one, I am able to see all the projects and everything looks happy.

image

On the new server, I see the same portfolio, but the last one tells me I don’t have permission and shows me a yellow triangle warning icon. If I try to click that project in the Projects Management view, SonarQube takes me to the login page and tells me I don’t have permissions, even if I try logging in again. On both server, I am part of the sonar-administrators group.

image

In the access.log I see the following: {IP REDACTED} - - [14/Nov/2023:07:35:44 -0800] "GET /api/views/projects_status?portfolio=PROJNAME&status=ERROR&ps=3 HTTP/1.1" 200 - "http://149.145.159.55:9000/portfolio?id=PROJNAME" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "AYvIo1Nybym6+iudAAGd"

That is the only error I see there that’s not related to the PAT (which, again, is now resolved and should no longer show up).

This feels like some funky permissions issue, and I have tried including other folks into a newly-made group to see those projects, but no additional projects are seen even by them.

Following up on the previous post, as a test I tried unchecking the last project from the portfolio, and when I did that and clicked ‘Reload’ on that pane, it was no longer in the list. When I went to the ‘All’ view, that project was not visible to me. When I went back to Projects Management and clicked on the ‘gear’ icon, it gave me an option for Restore Access. When I clicked that, I was able to see the project in the portfolio view again to add, and when I added it, there’s no longer the yellow warning icon. When I go to the Projects page, just that project is now also visible, but none of the rest that had not displayed are showing.

In my mind this is further leaning in the direction of a funky permissions error, but the other projects in Projects Management do not give me the Restore Access option, only Edit Permissions or Apply Permissions Template. I have tried multiple things with permissions, including or excluding myself from groups other than sonar-administrators and sonar-users, but no changes have effect.

Hi,

First, is there anything in your browser console when you’re looking at a page that is missing projects you should be able to see?

Second, did your method of authentication change from the old server to the new one?

 
Ann

Ann,

I will check again, but after having seen something in another thread about that, I looked but didn’t see any errors.

Authentication-wise, the domain controller did change, though (visible) user IDs stayed the same and both domain controllers are Active Directory.

Hi,

Barring errors in your browser console, I’m out of ideas. Let’s just cross that off, though, before I call for help.

 
Ann

Ann,

I’m going to try a quick experiment this morning with another user who was on the original server. I’m going to remove them from the new server, log in “fresh” and assign them one of our groups, and see if that lets them see those projects. I will update here once I’ve gotten that complete, and at least rule in/out a AD permissions issue.

1 Like

@ganncamp

Here’s the things I’ve now tried, with no success:

  1. Remove an existing user, have them log in with their Active Directory credentials, assign them to various groups. No change.
  2. Swap visibility on projects from private to public, disassociate with groups. No change.
  3. Create a non-AD user, log in with separate browser (Edge instead of Chrome), assign to groups, cannot see public projects or private projects associated with groups. No change.

I also went back and looked at Chrome’s Inspect function, refreshed the page while at the base Projects view, and all the GET commands come back with a 200 OK. I tested this in both Edge and Chrome, same behavior both ways with my admin (AD) and regular (non-AD) users.

Definitely stumped here.

Hi,

Yeah, me too. I’ve flagged this for more expert eyes.

 
Ann

1 Like

Hello @mikeschlag, I would like to gather more details on how the migration of the database was performed.

My understanding is that a 9.9LTS instance was made completely operational with no projects in it. Then the content of the original database (Related to a 9.2.4) got “copy/pasted” to the database already attached to the new 9.9 instance. Is that correct?

At which stage the database schema was migrated from 9.2.4 to 9.9? There are differences in the structures of the tables between these two versions, SonarQube performs the upgrade of the Database Schema at startup when it notices that it is based on an older version, was that the case even with this instance?

@Matteo_Mara @ganncamp

Woke up this morning to a nice surprise, and the server admin (who did the server standup and migration) figured out the issue. I’m waiting to hear back from him on what he did to get things working, and once I have that I will post here so that the solution is visible to all.

2 Likes

Thank you very much @mikeschlag for this news.
Really looking forward to reading the solution.

So it turns out that it apparently was related to ElasticSearch and whatever caching it had done. When he originally went to try to fix to clear out the directory (%SONAR_HOME%\data) there was nothing but a single folder that itself was empty (probably should have been a clue to us). This morning he dug in more to the configuration and found that the data folder was actually located for our install at %SONAR_HOME%\var\sonarqube\data. He stopped the server, cleared that out, started back up, and now everything displays as expected.

Glad it was a simple fix, and hopefully others in the future can use this as a reminder to go look in their configuration to see where that data is actually residing.

Appreciate your help and responsiveness @ganncamp and @Matteo_Mara!

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.