which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
SonarQube 8.2 Community & Developer editions
what are you trying to achieve
Manually execute a “dbcleaner” task, to see how/when old items (branches, …) are deleted.
what have you tried so far to achieve this
Reason is we use SonarQube 8.2 Community Edition and the non-supported sonarqube-community-branch-plugin
Olds PRs analysis are never removed, although the plugin code does support the feature.
So the idea would be to trig a dbcleaner manually and see if the plugin actually works fine.
Note that we also run a Developer Edition, and are sometimes facing similar issues, ie. 2 months old PRs still showing in SonarQube…
There’s no way to manually trigger housekeeping tasks. If a re-analysis of the main branch doesn’t do it for you, then I suggest you contact the maintainers of that community plugin by filing an issue on the project; it’s not supported here.
I asked about version because there was an issue in previous versions where it required a main branch analysis to trigger branch housekeeping. I’m pretty sure that’s fixed by the LTS, though so that shouldn’t be the issue. However, I’m not super-sure that PR analysis triggers housekeeping. When’s the last time you ran a branch analysis on this project?
And BTW, to make sure it’s clear, cleanup is done on a project when that project is analyzed. So analyzing ProjectA doesn’t trigger a general DB cleanup; it only triggers housekeeping on ProjectA.
Thanks for clarification regarding db cleanup trigger.
So there is no “weekly or monthly” kind of cleanup of the whole instance? From ops perspective it could be interesting to keep instance/database healthy.
For the particular example, main branch analysis was last performed few days ago (i did trig it manually from Jenkins to see if it would clean things up).
As our instance is quite “young” and not used that much, i can perfectly enable debug logs for a while and start again a main branch analysis on this project.
Would that help? Any particular log/class to watch for?
Note that i have renamed the thread, topic is clearer now
Nope. AFAIK, the only “cron job” in SonarQube is a daily check to see if it’s time to send the license reminder notification.
I get that. At the same time, we’re kinda running under the assumption that you analyze regularly to keep your projects healthy. That, in turn should keep the DB healthy.
You note that you’ve renamed the thread (). So now I notice that it says specifically “some projects”. Do you see any commonalities or patterns among those projects?
i’ve checked on few projects (instance is quite recent):
2 projects had its main branch analyzed 5 days ago, and several PR branches analysis older than 1 month
1 project had its main branch analyzed more than 1 month ago, and several PR branches analysis older than 1 month (may be normal when reading details you provided and DB cleaning trigger)
1 project had its main branch analyzed 3 days ago, and several PR branches analysis but all younger than 1 month (seems it is working on this project…)
I ran another analysis of above test project showing the issue, with DEBUG logs analysis.
Here is an excerpt for CE Logs:
2020.07.27 08:55:24 INFO ce[][o.s.s.l.ServerLogging] Level of logs changed to DEBUG
2020.07.27 08:56:21 INFO ce[][o.s.c.t.CeWorkerImpl] Execute task | project=scm:sweng.sandbox-platform | type=REPORT | branch=master | branchType=BRANCH | id=AXOPfLR9grqhUAt7SPaT | submitter=scm-analyzer
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExtractReportStep] Analysis report is 142 KB uncompressed
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=7ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Execute DB migrations for current project | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=8ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Verify billing | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=5 | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load quality profiles | status=SUCCESS | time=144ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load Quality gate | status=SUCCESS | time=4ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.p.NewCodePeriodResolver] Resolving first analysis as new code period as there is only one existing version
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load new code period | status=SUCCESS | time=4ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.f.FileMoveDetectionStep] Either no files added or no files removed. Do nothing.
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Detect file moves | reportFiles=3 | dbFiles=3 | addedFiles=0 | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load duplications | duplications=0 | status=SUCCESS | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.d.CrossProjectDuplicationStatusHolderImpl] Cross project duplication is disabled because it's disabled in the analysis report
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute cross project duplications | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute size measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute coverage measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute comment measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Copy custom measures | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute duplication measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute size measures on new code | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute language distribution | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute test measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute complexity measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load measure computers | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute Quality Profile status | status=SUCCESS | time=3ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] Execution time for each component visitor:
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - LoadComponentUuidsHavingOpenIssuesVisitor | time=2ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - IntegrateIssuesVisitor | time=14ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - CloseIssuesOnRemovedComponentsVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - MaintainabilityMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - NewMaintainabilityMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - ReliabilityAndSecurityRatingMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - NewReliabilityAndSecurityRatingMeasuresVisitor | time=1ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - SecurityReviewMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - NewSecurityReviewMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - LastCommitVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep] - MeasureComputersVisitor | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Execute component visitors | status=SUCCESS | time=26ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Checks executed after computation of measures | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute Quality Gate measures | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute Quality profile measures | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Generate Quality profile events | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Generate Quality gate events | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=5ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=52 | status=SUCCESS | time=5ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist live measures | insertsOrUpdates=287 | status=SUCCESS | time=27ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist duplication data | insertsOrUpdates=0 | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist new ad hoc Rules | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist issues | cacheSize=0 bytes | inserts=0 | updates=0 | merged=0 | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist project links | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist events | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Enable analysis | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Update last usage date of quality profiles | status=SUCCESS | time=2ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepOneFilter] -> Keep one snapshot per day between 2020-06-29 and 2020-07-26
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl:
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepOneFilter] -> Keep one snapshot per week between 2019-07-29 and 2020-06-29
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl:
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepOneFilter] -> Keep one snapshot per month between 2015-08-03 and 2019-07-29
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl:
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepWithVersionFilter] -> Keep analyses with a version prior to 2018-07-30
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl:
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DeleteAllFilter] -> Delete data prior to: 2015-08-03
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl:
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.purge.PurgeDao] <- Delete aborted builds
**2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.purge.PurgeDao] <- Purge stale branches**
**2020.07.27 08:56:21 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Purge db | status=SUCCESS | time=23ms**
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.IndexAnalysisStep] Call org.sonar.server.measure.index.ProjectMeasuresIndexer@3abd3578
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.IndexAnalysisStep] Call org.sonar.server.component.index.ComponentIndexer@6f804527
2020.07.27 08:56:22 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.IndexAnalysisStep] Call org.sonar.server.issue.index.IssueIndexer@7d5c7e3a
2020.07.27 08:56:22 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Index analysis | status=SUCCESS | time=138ms
2020.07.27 08:56:22 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Update need issue sync for branch | status=SUCCESS | time=2ms
2020.07.27 08:56:22 INFO ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Send issue notifications | newIssuesNotifs=0 | newIssuesDeliveries=0 | myNewIssuesNotifs=0 | myNewIssuesDeliveries=0 | changesNotifs=0 | changesDeliveries=0 | status=SUCCESS | time=1ms
I just tried to reproduce the problem and I also debugged it and it seems to work fine. However, as you mentioned, another user reported the same problem with the LTS.
Unfortunately, the debug logs won’t show more details about it.
Since sending a reproducer is not viable in this case, If you can access the DB directly (read only), you could look for the PR in the table ‘PROJECT_BRANCHES’ and check if:
The value of ‘exclude_from_purge’ is false;
The value of ‘update_at’ refers to a date that is older than 30 days.
That could help determine if one of those 2 conditions is not being fulfilled cause the P/R to not be deleted.
In fact it is not. Main branch last analysis on 8.2 was on Wednesday, June 3, 2020 2:06:35.989 PM, so only 15 days after PR branch update_date.
I then ran again Main branch analysis but post 8.4 upgrade. And as the 8.4 upgrade did modify all updated_at fields of the database then dbcleaner process would not find this PR branch still suitable for purge…
We’ll check back in 30 days see if things are evolving correctly
I realized that there is an inconsistency between the number of days displayed in the “Branches & Pull Requests” configuration page and the actual criteria used for the purge. Maybe the only problem we have is that that inconsistency is misleading. I’ve created a ticket so that we can align things: https://jira.sonarsource.com/browse/SONAR-13708