If you have ended up in a scenario where your organization is running multiple distinct SonarQube servers, and realized the multiple reasons why you should consolidate them, in this guide you will learn how.
Choose an approach
There are two main approaches to choose from when consolidating several SonarQube servers into one. To help you choose the right one for you, consider the following questions:
- Do you need to preserve your project analysis history? Remember that, thanks to Cleaning As You Code, you should be focusing on new code rather than legacy code.
- Do you have lots of issues flagged as Won’t Fix, False Positive or containing important comments you must keep?
In most if not all cases, we recommend favoring the Agile Consolidation approach. If you answered yes to any of the above questions, however, you may consider the Full Data Consolidation approach.
1.- Agile Consolidation approach
This approach requires the least administration work and allows achieving consolidation rapidly. It works no matter the edition of SonarQube you are using.
Note that with this approach you will lose (1) Project history data and (2) Issue metadata such as Won’t Fix, False Positive flags and any related comments.
- Provision whatever you need in the target platform (Projects, Users, Permissions, Global Settings, Quality Gates, Quality Profiles, Portfolios, extra plugins if required…)
- Configure your CI Tool to start using the new platform for all pipelines
- Freeze the source platforms, setting them in “read only mode”, so that users can still have access to historical data for a limited period of time, e.g. 6 months. Do this by removing all global and project permissions from all non-admin users, except Browse and See Source Code permissions.
- Done !
2.- Full Data Consolidation approach
This approach preserves historical data and all issue metadata, but it is the most costly and time-consuming.
Prerequisites
Please note that you require Enterprise Edition or above in order to be able to complete the steps in this approach. In addition, all your SonarQube instances will need to be aligned:
- Same SonarQube version
- Same SonarQube Edition: you may need to request for temporary licenses from your Sales Representative (Note: If consolidating servers above version 9.2, you won’t need for the servers to be on the same edition)
- Same plugins, with same versions
- Same custom metrics, if any
- Same custom rules, if any
Data to consolidate
To technically achieve full data consolidation of multiple SonarQube servers into one, we need to move each separate server’s data into the single final SonarQube instance. That data may fall into one of the following categories:
- Users, groups and permissions
- Quality Profiles
- Quality Gates
- Project data
- Applications
- Portfolios
Note: you will need to configure the global settings of your target instance manually, e.g., SMTP, authentication, ALM integration, Security Engine Custom Configuration, etc.
Let’s accomplish data consolidation step by step, in order:
1. Users, groups and permissions
When operating SonarQube EE or DCE, you should be delegating authentication to your organization’s authentication provider. In such a case, Users do not need to be consolidated, as they will be automatically created in SonarQube upon a first valid login.
Groups should be created anew in the target final SonarQube instance. Usually, one of your existing SonarQube servers will be chosen as the final server to consolidate all instances into. In that case, a comprehensive set of groups should already exist in your server. If the other instances define different sets of groups, you will need to assess if all of them make sense when put together, and create or remove groups in the final server accordingly if you identify any gaps. This will be a good opportunity to define a single, consistent set of groups for your organization.
Global Permissions will need to be manually set in your target final instance if required. If you manage these at group level, which is the recommended best practice, you should now be ready to do so. The same applies to Permission Templates.
In the case of unique Project, Portfolio and Application level Permissions, these cannot be set until you consolidate Project Data. When consolidating, it is recommended to handle such permissions via Permission Templates whenever possible.
2. Quality Profiles
If you need to consolidate custom Quality Profiles, you should:
- Export each Quality Profile using the Back up feature (Quality Profiles → Your QP → Gear icon → Back up) on the source SonarQube instance to export the profile to an XML file.
- Use the Restore Profile feature on the target instance to import the file (Quality Profiles → Restore).
If you have lots of Quality Profiles to consolidate, you can automate this process using the following API methods: GET api/qualityprofiles/backup and POST api/qualityprofiles/restore. Remember that SonarQube API is documented under http(s)://<your_sonarqube_url>/web_api.
3. Quality Gates
Be aware that in most scenarios, a small set of Quality Gates will cater most if not all needs of your developers. Therefore, the consolidation step is a good opportunity to define a carefully designed set of Quality Gates that:
- Helps you adopt the Cleaning As You Code methodology
- Sets reasonable, attainable goals for your developers
- Raises the bar gradually
If you need to consolidate any number of Quality Gates, you have to recreate such Quality Gates manually in your target SonarQube instance or use a set of API methods under the api/qualitygates collection.
4. Project Data
The bulk of the data to consolidate is almost always project data. The process to consolidate projects is made easier thanks to the Project Move feature, available in Enterprise Edition and above.
Moving all your projects to your target SonarQube consolidated instance can be either a manual or an automated process. If you have few projects, and as explained in the documentation, these are the main steps to follow:
- Provision each project in the new instance using the same project key. Keys must be unique: note that, if the same key is being used by more than 2 projects across your instances, you should resolve the conflict by setting a different key for conflicting projects before export.
- Configure the project’s permissions, Quality Profiles and Quality Gates as needed
- Export project data from the source instance and then Import it in the target one
- Run a first analysis so that source code is included in the project
Automating the above process for a large number of projects requires you to write your own script based on the following API methods:
GET api/projects/searchPOST api/projects/createGET api/settings/valuesPOST api/settings/setPOST api/project_dump/exportPOST api/project_dump/import
Source code data is not exported/imported when moving projects, and that’s why it is necessary to run a first analysis after importing your projects. It is important to note specifically which project-related data is exported/imported with the above process:
- Historical analysis data
- Project issues, including all associated metadata
- Active Branches and Pull Requests
- Project Settings such as Pull Request decoration or New Code period
5. Applications
Applications have to be consolidated manually, or alternatively, using a set of API methods under the api/applications collection.
6. Portfolios
Portfolio definitions have to be consolidated manually, or alternatively, using a set of API methods under the api/views collection: GET api/views/definition and POST api/views/define.
In addition, if you have any portfolio definitions with the “Manual” project selection mode setting, you will need to manually add again projects to the portfolio via the UI or use the POST api/views/add_project API method. However, if the project selection mode is any other (“By tags”, “By regular expression” or “All projects”), projects shall be automatically added once Portfolio computation is triggered. You should trigger this recomputation on all your consolidated portfolios to see them get populated, or alternatively, wait until any of its member projects get analyzed.