Sonar doesn't start after upgrading from 8.5.1 to LTS

Hello all,

We’re currently running docker 8.5.1 developers edition in docker - and when I’m trying to upgrade to lts-developers / lts-enterprise (we just bought the license) I get the following error (after the db upgrade):

org.sonar.server.exceptions.BadRequestException: Rule with UUID AXX959T2azZyf-hGQAI- not found

I logged in the postgresql db and I noticed the data of some tables isn’t really consistent (active_rules & rules_parameters):

select * from active_rules ar where rule_uuid not in (select uuid from rules r where r.uuid = ar.rule_uuid );
select * from rules_parameters rp where rule_uuid not in (select uuid from rules r where r.uuid = rp.rule_uuid );

All the active rules / rule parameters that exist in our db without a proper rule linked to them are related to plugins / languages that we have never used, like python, swift, vb.net…so, I don’t really understand why we can have these kind of issues.

If I do the upgrade with a clean db, there is no issue - but honestly, this isn’t an option for us as we would like to keep the existing projects / results.

Regards,

Javier G.

PS: I found a similar ticket in the forum Sonar doesn't start after updating from 8.4 to 8.6 but since the versions aren’t exactly the same, I decided to create a new post.

Hi @jgutierrezglez ,

Let’s follow the same logic for troubleshooting that we see in the related post (thank you for linking that).

Please list the contents of lib/extensions and /extensions/plugins directories first. This will help check if there are other plugins that could be causing problems.

Hello @Joe,

Thanks for you quick response

The list of content of lib/extensions is:

sonar-abap-plugin-3.9.1.3127.jar
sonar-apex-plugin-1.8.3.2219.jar
sonar-cfamily-plugin-6.20.1.32841.jar
sonar-cobol-plugin-4.6.2.4876.jar
sonar-csharp-plugin-8.22.0.31243.jar
sonar-css-plugin-1.4.2.2002.jar
sonar-flex-plugin-2.6.1.2564.jar
sonar-go-plugin-1.8.3.2219.jar
sonar-html-plugin-3.4.0.2754.jar
sonar-jacoco-plugin-1.1.1.1157.jar
sonar-java-plugin-6.15.1.26025.jar
sonar-javascript-plugin-7.4.4.15624.jar
sonar-kotlin-plugin-1.8.3.2219.jar
sonar-php-plugin-3.17.0.7439.jar
sonar-pli-plugin-1.11.1.2727.jar
sonar-plsql-plugin-3.6.1.3873.jar
sonar-python-plugin-3.4.1.8066.jar
sonar-rpg-plugin-2.5.1.2575.jar
sonar-ruby-plugin-1.8.3.2219.jar
sonar-scala-plugin-1.8.3.2219.jar
sonar-security-csharp-frontend-plugin-8.9.0.11439.jar
sonar-security-java-frontend-plugin-8.9.0.11439.jar
sonar-security-js-frontend-plugin-8.9.0.11439.jar
sonar-security-php-frontend-plugin-8.9.0.11439.jar
sonar-security-plugin-8.9.0.11439.jar
sonar-security-python-frontend-plugin-8.9.0.11439.jar
sonar-swift-plugin-4.3.1.4892.jar
sonar-tsql-plugin-1.5.1.4340.jar
sonar-vb-plugin-2.7.1.2721.jar
sonar-vbnet-plugin-8.22.0.31243.jar
sonar-xml-plugin-2.2.0.2973.jar

All these jars are coming with the docker image that I’m using (sonarqube:8.5.1-enterprise / sonarqube:lts-enterprise)

The list of content of /extensions/plugins is:

sonar-dependency-check-plugin-2.0.8.jar
sonar-findbugs-plugin-4.0.3.jar
sonar-pmd-plugin-3.3.1.jar

These plugins are added by us (mounting the volume and copying them - so, they’re accessible by sonar)

Regards,

Javier G.

Hi @jgutierrezglez ,

Can you please turn on TRACE logs and paste the portion of logs just before the failure?

  • Change your Log Level to TRACE (Administration > System > Log Level) - no need to restart SonarQube
  • Go to browser: $YOUR_SONARQUBE_HOSTNAME/setup to restart the upgrade
  • Wait for the failure and copy and paste the log portion here

Also, please provide the results of the following SQL queries:

select * from rules where uuid='AXX959T2azZyf-hGQAI-';
select * from rules_parameters where rule_uuid='AXX959T2azZyf-hGQAI-'; 
select * from active_rules where rule_uuid='AXX959T2azZyf-hGQAI-'; 

select * from rules_profiles rp where uuid in (select profile_uuid from active_rules ar where rule_uuid='AXX959T2azZyf-hGQAI-');

Hello @Joe,

The migration process isn’t a problem - it works well without no issue. The issue appears when Sonar tries to start after the migration - and it cannot find some rules that we have never used (python)

Attached to this ticket you can find the trace of the failure after setting the log level to TRACE:

sonar.log (34.4 KB)

The results of the queries requested in your previous comment:

sonar=# select * from rules where uuid='AXX959T2azZyf-hGQAI-';
 name | plugin_rule_key | plugin_key | plugin_config_key | plugin_name | scope | description | priority | status | language | def_remediation_function | def_remediation_gap_mult | def_remediation_base_effort | g
ap_description | system_tags | is_template | description_format | rule_type | security_standards | is_ad_hoc | is_external | created_at | updated_at | uuid | template_uuid 
------+-----------------+------------+-------------------+-------------+-------+-------------+----------+--------+----------+--------------------------+--------------------------+-----------------------------+--
---------------+-------------+-------------+--------------------+-----------+--------------------+-----------+-------------+------------+------------+------+---------------
(0 rows)

sonar=# select * from rules_parameters where rule_uuid='AXX959T2azZyf-hGQAI-'; 
 name | description | param_type | default_value | uuid | rule_uuid 
------+-------------+------------+---------------+------+-----------
(0 rows)

sonar=# select * from active_rules where rule_uuid='AXX959T2azZyf-hGQAI-'; 
 failure_level | inheritance |  created_at   |  updated_at   |         uuid         |     profile_uuid     |      rule_uuid       
---------------+-------------+---------------+---------------+----------------------+----------------------+----------------------
             2 |             | 1606282651366 | 1606282651366 | AXX96CcAazZyf-hGQC8F | AXX96CbmazZyf-hGQC6R | AXX959T2azZyf-hGQAI-
(1 row)

sonar=# select * from rules_profiles rp where uuid in (select profile_uuid from active_rules ar where rule_uuid='AXX959T2azZyf-hGQAI-');
 name | language | is_built_in | rules_updated_at | created_at | updated_at | uuid 
------+----------+-------------+------------------+------------+------------+------
(0 rows)

Regards,

Javier G.

Hi @jgutierrezglez ,

Thank you for the logs and queries. It looks like the link between the active_rule and the rules_profile are missing, which matches the TRACE logging.

It looks like you run these queries in psql. Can you also check this query on your pg_dump backup and see if you get the same results?

Hello @Joe,

I get exactly the same results - do you have any about to recover / recreate this missing link? If I exclude the python plugin, and I get a similar issue for other languages (VB.net) - all of them languages that we have never used.

Also restored a backup of our db and try to run the same version of sonar that we’re running in prod (without no issues) - 8.5.1-developer and I don’t have this problem, so, it’s clearly related how these rules are registered in the LTS version / 8.5.1-enterprise

Regards,

Javier G.

Hi @jgutierrezglez , there’s no clear-cut way to recover this link. I will reach out to our internal team member who can assist you.

EDIT: I’ve only seen one similar case to your issue. This customer noticed inconsistencies between their current DB and the pg_dump DB, e.g. a row with a specific UUID was visible in the table rules in their pg_dump DB but not their current DB. So they recreated their DB and imported the pg_dump DB, but still encountered errors but different ones (e.g. "could not create unique index “live_measures_component”), so they removed the duplicates and their SQ server started to work.

I can recommend a similar method as the previous customer did:

  • Refresh your database’s statistics and rebuild your database’s indices.
  • Create a backup of your database.
  • Create a new SonarQube database, which should be up and running.
  • Create the sonar schema on this new SonarQube database.
  • Download and expand a copy of SonarQube that exactly matches the version you’re running (i.e. use a new copy of SonarQube).
  • Configure your SonarQube sonar.properties to connect to this new, empty database
    • If you’ve placed your SonarQube copy on the same server that runs your primary SonarQube instance, you’ll also need to configure non-default ports for your copy SonarQube instance.
  • Startup the copy of SonarQube and it will connect to this empty database and populate the schema
  • Once your copy instance is up and running (this indicates that the schema is fully populated), you can stop and delete SonarQube copy.
  • Import/restore the DB backup into this freshly minted DB.
  • Refresh the database statistics and rebuild indices on the target database before restarting the original SonarQube server

Joe

Hello @Joe,

I tried your suggestion and it didn’t work. After trying other options, I randomly found that the issue was caused by one of the plugins that were installed (doing the upgrade without any plugin installed didn’t trigger any issue) - and since we don’t really need any of those plugins anymore, I just removed them and upgraded our instance.

I cannot really say which one of the plugins (sonar-dependency-check-plugin-2.0.8.jar / sonar-findbugs-plugin-4.0.3.jar / sonar-pmd-plugin-3.3.1.jar) was causing this problem and why - but I can confirm it was due to one of them.

Cheers,

Javier G.

Hello @jgutierrezglez ,

Thank you for informing me. That definitely sounds like the cause of the issue since we don’t check compatibility of all 3rd party, community plugins. I’m glad you were able to resolve your issue.

Joe

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.