[v8.2 Developer Edition] PR branches for some projects are not purged

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    SonarQube 8.2 Community & Developer editions

  • what are you trying to achieve
    Manually execute a “dbcleaner” task, to see how/when old items (branches, …) are deleted.

  • what have you tried so far to achieve this
    Reason is we use SonarQube 8.2 Community Edition and the non-supported sonarqube-community-branch-plugin
    Olds PRs analysis are never removed, although the plugin code does support the feature.
    So the idea would be to trig a dbcleaner manually and see if the plugin actually works fine.

Note that we also run a Developer Edition, and are sometimes facing similar issues, ie. 2 months old PRs still showing in SonarQube…

Hi,

There’s no way to manually trigger housekeeping tasks. If a re-analysis of the main branch doesn’t do it for you, then I suggest you contact the maintainers of that community plugin by filing an issue on the project; it’s not supported here.

 
Ann

Thanks

we are also seeing the same issue on a Developer edition instance, so without using a community plugin:

Configuration for Branch&PR dbcleaner is left by default.
Last analysis of main branch was yesterday.

Hi,

What version are you using?

 
Ann

Hello
Was 8.2 until monday, now it is 8.4.1 (docker images)

Hi,

I asked about version because there was an issue in previous versions where it required a main branch analysis to trigger branch housekeeping. I’m pretty sure that’s fixed by the LTS, though so that shouldn’t be the issue. However, I’m not super-sure that PR analysis triggers housekeeping. When’s the last time you ran a branch analysis on this project?

And BTW, to make sure it’s clear, cleanup is done on a project when that project is analyzed. So analyzing ProjectA doesn’t trigger a general DB cleanup; it only triggers housekeeping on ProjectA.

 
Ann

Thanks for clarification regarding db cleanup trigger.
So there is no “weekly or monthly” kind of cleanup of the whole instance? From ops perspective it could be interesting to keep instance/database healthy.

I think our issue is similar to this post, although it is about LTS version: Sonarqube - Short-lived branches are never purged

For the particular example, main branch analysis was last performed few days ago (i did trig it manually from Jenkins to see if it would clean things up).

As our instance is quite “young” and not used that much, i can perfectly enable debug logs for a while and start again a main branch analysis on this project.
Would that help? Any particular log/class to watch for?

Note that i have renamed the thread, topic is clearer now :wink:

Hi,

Nope. AFAIK, the only “cron job” in SonarQube is a daily check to see if it’s time to send the license reminder notification.

I get that. At the same time, we’re kinda running under the assumption that you analyze regularly to keep your projects healthy. That, in turn should keep the DB healthy. :smiley:

You note that you’ve renamed the thread (:+1:). So now I notice that it says specifically “some projects”. Do you see any commonalities or patterns among those projects?

 
Ann

Hello

i’ve checked on few projects (instance is quite recent):

  • 2 projects had its main branch analyzed 5 days ago, and several PR branches analysis older than 1 month
  • 1 project had its main branch analyzed more than 1 month ago, and several PR branches analysis older than 1 month (may be normal when reading details you provided and DB cleaning trigger)
  • 1 project had its main branch analyzed 3 days ago, and several PR branches analysis but all younger than 1 month (seems it is working on this project…)

I ran another analysis of above test project showing the issue, with DEBUG logs analysis.
Here is an excerpt for CE Logs:

2020.07.27 08:55:24 INFO  ce[][o.s.s.l.ServerLogging] Level of logs changed to DEBUG
2020.07.27 08:56:21 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=scm:sweng.sandbox-platform | type=REPORT | branch=master | branchType=BRANCH | id=AXOPfLR9grqhUAt7SPaT | submitter=scm-analyzer
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExtractReportStep] Analysis report is 142 KB uncompressed
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=7ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Execute DB migrations for current project | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=8ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Verify billing | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=5 | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load quality profiles | status=SUCCESS | time=144ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load Quality gate | status=SUCCESS | time=4ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.p.NewCodePeriodResolver] Resolving first analysis as new code period as there is only one existing version
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load new code period | status=SUCCESS | time=4ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.f.FileMoveDetectionStep] Either no files added or no files removed. Do nothing.
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Detect file moves | reportFiles=3 | dbFiles=3 | addedFiles=0 | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load duplications | duplications=0 | status=SUCCESS | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.d.CrossProjectDuplicationStatusHolderImpl] Cross project duplication is disabled because it's disabled in the analysis report
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute cross project duplications | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute size measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute coverage measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute comment measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Copy custom measures | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute duplication measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute size measures on new code | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute language distribution | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute test measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute complexity measures | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Load measure computers | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute Quality Profile status | status=SUCCESS | time=3ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   Execution time for each component visitor:
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - LoadComponentUuidsHavingOpenIssuesVisitor | time=2ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - IntegrateIssuesVisitor | time=14ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - CloseIssuesOnRemovedComponentsVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - MaintainabilityMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - NewMaintainabilityMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - ReliabilityAndSecurityRatingMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - NewReliabilityAndSecurityRatingMeasuresVisitor | time=1ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - SecurityReviewMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - NewSecurityReviewMeasuresVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - LastCommitVisitor | time=0ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.ExecuteVisitorsStep]   - MeasureComputersVisitor | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Execute component visitors | status=SUCCESS | time=26ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Checks executed after computation of measures | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute Quality Gate measures | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Compute Quality profile measures | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Generate Quality profile events | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Generate Quality gate events | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=5ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=4ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=52 | status=SUCCESS | time=5ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist live measures | insertsOrUpdates=287 | status=SUCCESS | time=27ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist duplication data | insertsOrUpdates=0 | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist new ad hoc Rules | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist issues | cacheSize=0 bytes | inserts=0 | updates=0 | merged=0 | status=SUCCESS | time=1ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist project links | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist events | status=SUCCESS | time=2ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=SUCCESS | time=0ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Enable analysis | status=SUCCESS | time=3ms
2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Update last usage date of quality profiles | status=SUCCESS | time=2ms
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepOneFilter] -> Keep one snapshot per day between 2020-06-29 and 2020-07-26
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl: 
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepOneFilter] -> Keep one snapshot per week between 2019-07-29 and 2020-06-29
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl: 
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepOneFilter] -> Keep one snapshot per month between 2015-08-03 and 2019-07-29
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl: 
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.KeepWithVersionFilter] -> Keep analyses with a version prior to 2018-07-30
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl: 
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DeleteAllFilter] -> Delete data prior to: 2015-08-03
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.p.p.DefaultPeriodCleaner] <- Delete analyses of component AXFQizNDWDVujoyaWUBl: 
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.purge.PurgeDao] <- Delete aborted builds
**2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.d.purge.PurgeDao] <- Purge stale branches**
**2020.07.27 08:56:21 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Purge db | status=SUCCESS | time=23ms**
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.IndexAnalysisStep] Call org.sonar.server.measure.index.ProjectMeasuresIndexer@3abd3578
2020.07.27 08:56:21 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.IndexAnalysisStep] Call org.sonar.server.component.index.ComponentIndexer@6f804527
2020.07.27 08:56:22 DEBUG ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.p.s.IndexAnalysisStep] Call org.sonar.server.issue.index.IssueIndexer@7d5c7e3a
2020.07.27 08:56:22 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Index analysis | status=SUCCESS | time=138ms
2020.07.27 08:56:22 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Update need issue sync for branch | status=SUCCESS | time=2ms
2020.07.27 08:56:22 INFO  ce[AXOPfLR9grqhUAt7SPaT][o.s.c.t.s.ComputationStepExecutor] Send issue notifications | newIssuesNotifs=0 | newIssuesDeliveries=0 | myNewIssuesNotifs=0 | myNewIssuesDeliveries=0 | changesNotifs=0 | changesDeliveries=0 | status=SUCCESS | time=1ms

Hi,

I just tried to reproduce the problem and I also debugged it and it seems to work fine. However, as you mentioned, another user reported the same problem with the LTS.
Unfortunately, the debug logs won’t show more details about it.

Since sending a reproducer is not viable in this case, If you can access the DB directly (read only), you could look for the PR in the table ‘PROJECT_BRANCHES’ and check if:

  • The value of ‘exclude_from_purge’ is false;
  • The value of ‘update_at’ refers to a date that is older than 30 days.

That could help determine if one of those 2 conditions is not being fulfilled cause the P/R to not be deleted.

1 Like

Hello

Here are results for the exercise with project “sandbox-platform” having one master main branch and one clearly outdated PR branch:

exclude_from_purge = f
created_at = 1589969926775 (Wednesday, May 20, 2020 10:18:46.775 AM)
updated_at = 1589979992879 (Wednesday, May 20, 2020 1:06:32.879 PM)

So this PR should have been properly detected and deleted by the dbcleaning process…

Note: as we upgraded in the meantime to v8.4 the 22nd of July, all “updated_at” timestamp have been modified to roughly the date of the upgrade:

created_at = 1589969926775 (Wednesday, May 20, 2020 10:18:46.775 AM)
updated_at = 1595407852101 (Wednesday, July 22, 2020 8:50:52.101 AM)

this is when backups come handy :smiley:

So when the P/R failed to be purged, was it before the upgrade? And was a branch analyzed 30 days after its update at date?

In fact it is not. Main branch last analysis on 8.2 was on Wednesday, June 3, 2020 2:06:35.989 PM, so only 15 days after PR branch update_date.

I then ran again Main branch analysis but post 8.4 upgrade. And as the 8.4 upgrade did modify all updated_at fields of the database then dbcleaner process would not find this PR branch still suitable for purge…

We’ll check back in 30 days see if things are evolving correctly

ok, sounds good!

I realized that there is an inconsistency between the number of days displayed in the “Branches & Pull Requests” configuration page and the actual criteria used for the purge. Maybe the only problem we have is that that inconsistency is misleading. I’ve created a ticket so that we can align things: https://jira.sonarsource.com/browse/SONAR-13708

1 Like