Sonarqube 8.4.1 Community Edition
Postgresql 11
Upgraded SonarQube from 6.7.5 --> 7.9.3 --> 8.4.4 along with a MySQL to PostregSQL migration using the mysql migrator tool
We successfully performed a migration and upgrade of our Sonarqube instance. However we’re now seeing sporadic failures that are occurring during the scan phase with some projects.
java.lang.IllegalStateException: Failed to perform DB migration for project
at org.sonar.ce.task.projectanalysis.step.DbMigrationsStep.execute(DbMigrationsStep.java:48)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
at org.sonar.ce.task.projectanalysis.step.DbMigrationsStep.execute(DbMigrationsStep.java:41)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:235)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:217)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:162)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:137)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:89)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.postgresql.util.PSQLException: ERROR: column "id" does not exist
Position: 8
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:153)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:103)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
at org.sonar.server.platform.db.migration.step.SelectImpl.scroll(SelectImpl.java:74)
at org.sonar.server.platform.db.migration.step.MassUpdate.execute(MassUpdate.java:93)
at org.sonar.ce.task.projectanalysis.dbmigration.PopulateFileSourceLineCount.execute(PopulateFileSourceLineCount.java:57)
at org.sonar.server.platform.db.migration.step.DataChange.execute(DataChange.java:44)
at org.sonar.ce.task.projectanalysis.step.DbMigrationsStep.execute(DbMigrationsStep.java:46)
... 21 more
potential workaround
So far we found https://jira.sonarsource.com/browse/SONAR-12282 that may be similar. However these project were running successfully previously when on 6.7.5. They are now just failing to right and in order to fix it we need to delete the projects.
The postrgesql query for SONAR-12282 that’s in the bug report doesn’t work since we’re now on 8.4.1 and we’re trying to identify the scope of the issue.
This issue was recently reported internally and so far is known to occur when existing projects never analysed between SonarQube v7.2 and v8.3 (which would include the 6.7 -> 7.9 -> 8.4 upgrade path) when analysed on SonarQube v8.4.
No ticket for this yet, but we’ll keep you posted on this thread.
Hey Colin, thanks for getting back to us. It seems the query to detect affected projects in the original bug ticket we posted does not work on our Sonar 8.4.1 postgresql data. Since this seems like a re-occurance could you help us update that query so we can see what projects are affected and communicate out to our customers? Here is the one I created by modifying the existing queries but it seems to return empty:
select
p.kee,p.name
from
projects p
inner join snapshots s on
s.component_uuid = p.uuid
and s.islast = true
where
p.qualifier = 'TRK'
and exists (
select
pm.component_uuid, pm.metric_uuid, count(1)
from
project_measures pm
where
pm.analysis_uuid = s.uuid
group by
pm.component_uuid, pm.metric_uuid
having
count(1) > 1
)
While a similar issue in that there is an effect on projects not analysed since a certain version, the underlying issue and the query is not at all the same.
While this query has resulted in some odd results on some (“real”) databases facing the issue (with some null values being returned in place of project keys), I was able to get an accurate summary of affected projects.
select projects.kee, count(1)
from file_sources
LEFT JOIN components
ON file_sources.project_uuid = components.uuid
LEFT JOIN projects
ON components.root_uuid = projects.uuid
where file_sources.line_count='-1'
GROUP BY projects.kee;
Want to give this a try and let me know?
If the theory is right – that projects not analyzed since v6.7 LTS are affected, you should be able to cross-check with the global Administration > Projects > Management page and filtering by Last Analysis Date to when you upgraded your SonarQube instance to v7.9 LTS.
Hey Colin, the assumption is that if the project was not scanned while we were on v6.7 then when we upgraded it becomes affected by this bug? If that’s the case that query returns the below project as a result and it was listed as last scanned on: March 17th 2020. Which is when we were on v6.7.5.
Could this be a re-occurance of the issue because we upgraded to v7.9LTS and then immediately to v8.4.1 without forcing the projects to rescan? Also is there a possible fix beyond deleting as it affects 95% of all our projects? Roughly 44,000 projects.
Oy, ticket is here. I have a fix ready, should be published to master in a day or so. I guess we will need to discuss with SQ PM to see if it make sense to backport this fix to 8.4.
Hi again Pierre, would this fix resolve our issue with a minor version upgrade or would we also be required to re-migrate our data? Additionally could you explain to us the technical issue and what causes it?
No need to re-migrate your data. The story here is that we introduced a new line_count column on the file_sources table, with SQ 7.2. As this information is coming from the analyzers, from the SonarQube point of view there is no way to populate this new column at startup during a regular migration. What was decided at the time is to do a “progressive migration” and use every project analysis to populate this field. That’s why you have issues with projects analyzed since a long time ago only.
This is in production, we missed it in our upgrade testing. Could we simply populate those columns for users with a SQL query? Deleting these projects would be incredibly painful for our customers as it would wipe out all their historical data and also wipe away their configured group permissions.
Just did the upgrade from all LTS 6.7LTS -> 7.9LTS -> 8.4.1 and BAM, just got the “Project analysis has failed”. Looking forward for a fix. And thanks for this awesome tools. Also, this happened in production
The technical release is done, and that’s what you found the link for! Links still have to be updated / announcements made, but you can go ahead and try it out.
Hi,my sonarqube version is community edition 8.0 and my database is oracle12c. I didn’t do a version upgrade, I just migrated the application and database to the new machine. After the migration, an error message will occasionally appear. The error message is: Failed to perform DB migration for project…Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed out… This kind of error occurs every few days. The same sonarqube project sometimes appears, sometimes not. How can I handle this? Should I upgrade the version, or can I migrate the database back to sonar’s internal database? Or upgrade the oracle driver?