SonarQube 8.4.1 Analysis Bug

Sonarqube 8.4.1 Community Edition
Postgresql 11
Upgraded SonarQube from 6.7.5 --> 7.9.3 --> 8.4.4 along with a MySQL to PostregSQL migration using the mysql migrator tool

We successfully performed a migration and upgrade of our Sonarqube instance. However we’re now seeing sporadic failures that are occurring during the scan phase with some projects.

java.lang.IllegalStateException: Failed to perform DB migration for project
	at org.sonar.ce.task.projectanalysis.step.DbMigrationsStep.execute(DbMigrationsStep.java:48)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
	at org.sonar.ce.task.projectanalysis.step.DbMigrationsStep.execute(DbMigrationsStep.java:41)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:235)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:217)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:162)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:137)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:89)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.postgresql.util.PSQLException: ERROR: column "id" does not exist
  Position: 8
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:153)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:103)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
	at org.sonar.server.platform.db.migration.step.SelectImpl.scroll(SelectImpl.java:74)
	at org.sonar.server.platform.db.migration.step.MassUpdate.execute(MassUpdate.java:93)
	at org.sonar.ce.task.projectanalysis.dbmigration.PopulateFileSourceLineCount.execute(PopulateFileSourceLineCount.java:57)
	at org.sonar.server.platform.db.migration.step.DataChange.execute(DataChange.java:44)
	at org.sonar.ce.task.projectanalysis.step.DbMigrationsStep.execute(DbMigrationsStep.java:46)
	... 21 more
  • potential workaround
    • So far we found https://jira.sonarsource.com/browse/SONAR-12282 that may be similar. However these project were running successfully previously when on 6.7.5. They are now just failing to right and in order to fix it we need to delete the projects.
    • The postrgesql query for SONAR-12282 that’s in the bug report doesn’t work since we’re now on 8.4.1 and we’re trying to identify the scope of the issue.

Continuance of SonarQube 8.4 Possible Database Upgrade Bug

Hey there.

This issue was recently reported internally and so far is known to occur when existing projects never analysed between SonarQube v7.2 and v8.3 (which would include the 6.7 -> 7.9 -> 8.4 upgrade path) when analysed on SonarQube v8.4.

No ticket for this yet, but we’ll keep you posted on this thread.

Hey Colin, thanks for getting back to us. It seems the query to detect affected projects in the original bug ticket we posted does not work on our Sonar 8.4.1 postgresql data. Since this seems like a re-occurance could you help us update that query so we can see what projects are affected and communicate out to our customers? Here is the one I created by modifying the existing queries but it seems to return empty:

select
    p.kee,p.name
from
  projects p
inner join snapshots s on
  s.component_uuid = p.uuid
  and s.islast = true
where
  p.qualifier = 'TRK'
 and exists (
    select
      pm.component_uuid, pm.metric_uuid, count(1)
    from
      project_measures pm
    where
      pm.analysis_uuid = s.uuid
    group by
      pm.component_uuid, pm.metric_uuid
    having
      count(1) > 1
 )

Hey there.

While a similar issue in that there is an effect on projects not analysed since a certain version, the underlying issue and the query is not at all the same.

While this query has resulted in some odd results on some (“real”) databases facing the issue (with some null values being returned in place of project keys), I was able to get an accurate summary of affected projects.

select projects.kee, count(1)
from file_sources
LEFT JOIN components
ON file_sources.project_uuid = components.uuid
LEFT JOIN projects 
ON components.root_uuid = projects.uuid
where file_sources.line_count='-1'
GROUP BY projects.kee;

Want to give this a try and let me know?

If the theory is right – that projects not analyzed since v6.7 LTS are affected, you should be able to cross-check with the global Administration > Projects > Management page and filtering by Last Analysis Date to when you upgraded your SonarQube instance to v7.9 LTS.

Hey Colin, the assumption is that if the project was not scanned while we were on v6.7 then when we upgraded it becomes affected by this bug? If that’s the case that query returns the below project as a result and it was listed as last scanned on: March 17th 2020. Which is when we were on v6.7.5.

“agency-ui:bugfix_agency-groups-role-permissions-for-edit-add-groups-and-memberships”

Could this be a re-occurance of the issue because we upgraded to v7.9LTS and then immediately to v8.4.1 without forcing the projects to rescan? Also is there a possible fix beyond deleting as it affects 95% of all our projects? Roughly 44,000 projects.

Oy, ticket is here. I have a fix ready, should be published to master in a day or so. I guess we will need to discuss with SQ PM to see if it make sense to backport this fix to 8.4.

Hi again Pierre, would this fix resolve our issue with a minor version upgrade or would we also be required to re-migrate our data? Additionally could you explain to us the technical issue and what causes it?

No need to re-migrate your data. The story here is that we introduced a new line_count column on the file_sources table, with SQ 7.2. As this information is coming from the analyzers, from the SonarQube point of view there is no way to populate this new column at startup during a regular migration. What was decided at the time is to do a “progressive migration” and use every project analysis to populate this field. That’s why you have issues with projects analyzed since a long time ago only.

The bug here was introduced with SQ 8.4, where we replaced all auto-generated ID by java-generated UUID for the database (SONAR-13221). We basically “missed” this step of the analysis were a SQL query was hardcoded, still using the old schema with IDs: https://github.com/SonarSource/sonarqube/blob/7f657d7e476bb650814ae7cd4287a1c1dd147fe3/server/sonar-ce-task-projectanalysis/src/main/java/org/sonar/ce/task/projectanalysis/dbmigration/PopulateFileSourceLineCount.java#L52

So the minor upgrade will do just fine :slight_smile:

That’s great Pierre! Is there an ETA for 8.4.2 or 8.5 being released with this fix?

8.5 is planned for end of September (minor version are on a 2-month lifecycle). 8.4.2 is not planned yet.

Do you encounter this issue on your production instance, or in a test environment?

This is in production, we missed it in our upgrade testing. Could we simply populate those columns for users with a SQL query? Deleting these projects would be incredibly painful for our customers as it would wipe out all their historical data and also wipe away their configured group permissions.

-Brandon

Also running into this with a production instance and would lose years of data. Any chance there’s a workaround we can implement in the meantime?

Hi, i’m back from holiday. We are preparing a 8.4.2 for this week, including a backport of this fix.

2 Likes

I’d <3 that more than once if it would allow me to :slight_smile: Thanks @pierreguillot!

Just did the upgrade from all LTS 6.7LTS -> 7.9LTS -> 8.4.1 and BAM, just got the “Project analysis has failed”. Looking forward for a fix. And thanks for this awesome tools. Also, this happened in production :slight_smile:

Hi,

There is “sonarqube-8.4.2.36762.zip” in download URL: https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.4.2.36762.zip. Is the new version 8.4.2.36762 resolves this issue. ?

The technical release is done, and that’s what you found the link for! :slight_smile: Links still have to be updated / announcements made, but you can go ahead and try it out.

1 Like

Thanks to the SonarSource team for putting together this hotfix!

:grin:Hi,my sonarqube version is community edition 8.0 and my database is oracle12c. I didn’t do a version upgrade, I just migrated the application and database to the new machine. After the migration, an error message will occasionally appear. The error message is: Failed to perform DB migration for project…Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed out… This kind of error occurs every few days. The same sonarqube project sometimes appears, sometimes not. How can I handle this? Should I upgrade the version, or can I migrate the database back to sonar’s internal database? Or upgrade the oracle driver?

Hey @jingjing4

Instead of bumping an old thread, I recommend raising a new thread. I’ll close this one.