Elimination of JGit or at most as fallback

A while back we finally switched from SVN to Git.
In doing so, I forgot to replace the SVN plugin by the Git plugin in SonarQube though.
So since then, the VCS integration was not used and the analysis builds needed around 2 hours with a resourcs graph like this:

Now with upgrading SonarQube from 6.5 to 7.5 I of course also added the Git plugin.
Unfortunately now the builds need around 24 - 30 hours (yes, about a day, this is not a typo) with a resource graph like this:

After some analysis, I found out that the problem is the Git analysis which took up 21.5 hours of the 23.5 hours the build in the second screenshot needed.

As an experiment I took the Git SCM plugin code from tag 1.7.0 and replaced all usage of JGit by native Git commands and rerun an analysis and now we are back to the old duration around 2 hours and a sensible resources graph like this:

So here some advantages of using native Git executable instead of JGit:

  • It is much more CPU friendly
  • It is much more RAM friendly
  • It fully supports Git (JGit will probably need another 5 years to add worktree or .mailmap support)
  • Probably other points I don’t think of currently

Is there any chance you will use native Git instead of JGit in the future, or accept a PR that changes this?
An implementation could also determine whether native Git is accessible and use JGit as a fallback if that would be preferred.
Actually most work would already be finished, as the only thing that was added in master since 1.7.0 was the IgnoreCommand, everything else more or less stayed the same.

PS: the current state of the git plugin on GitHub is not buildable as the dependency on the plugin API is not published. :-/

1 Like

Ok, my last example was bad.
I was not aware that only changed files get blamed which is good of course.
In the slow analysis it was the first one since the upgrade and also the first one with git as before the git-plugin was missing as mentioned, so I think because of that all files got blamed.

I run the analysis again with sonar.scm.forceReloadAll = true to replicate the behavior and still got significant better performance. 4.5 hours for the full build with 2 hours of blaming and resources graph like this:

PS: I ported now my changes also to latest master and also implemented the new ignore command with native Git.

4 Likes

Hi,

Very interesting experiment! Thanks you for taking the time to investigate.

Could you share with us your implementation of the git plugin?

We will discuss internally the option of using native command instead of JGit, we’ll come back to you soon!

Thanks again
Benoit

3 Likes

Sure, if it will be decided for it, will it be native-or-failed or native-or-JGit-fallback?
Latter of course means that any functionality needs to be implemented for both tools, but it would work even if no native Git is in the path.

That’s something we’ll discuss, but probably native-or-JGit-fallback for retro-compatibility!

Here is the commit that completely replaces JGit (except for test fixture setup) by native git which is also configurable: https://github.com/Vampire/sonar-scm-git/commits/native-git-instead-of-jgit

Only documentation is not updated, code-wise it should be pretty production ready I think.
It passes all tests (I just removed a couple that tested for behavior with JGit exceptions being thrown) and it seems to me it is working fine and fast here.

Maybe I’ll later on also provide a variant that uses JGit as fallback if no native Git executable is configured and the default is not found or usable.

If you want to use it, you can either just add my fork as remote and merge or cherry-pick it in, or I can open a PR, especially if you want to request some changes.

2 Likes

And here the version with JGit as fallback if no executable is configured and default cannot be executed: https://github.com/Vampire/sonar-scm-git/commits/native-git-with-jgit-fallback

Same state basically, code should work properly, documentation is untouched.

It also includes a fix for the blaming in JGit part.
There were two tests that actually fail here (probably significant that this is on Windows) in master and also failed with native Git, because no -w was used for the diff command and I guess the test use-case has different line endings or something like that.

1 Like

Thanks! We’ll have our internal discussion beginning of next week, I’ll come back to you when we’ve made a decision!

1 Like

Just to add some context, long time ago we were using the native git cli. Then we moved to use JGit because Jenkins started to also use JGit, and consequently the git cli was not available on slaves.

There is no silver bullet, that’s why we have to discuss internally.

Food for though: https://wiki.jenkins.io/display/JENKINS/Git+Plugin#GitPlugin-WhyNotJGit

Well, there build slaves are usually under control of the one using them, so should be able to have any tool needed. Alternatively one could run builds in docker containers that have exactly the environment required. :slight_smile:

Also, that page mentions that any Git update can break the parsing.
This is basically mainly true when you use porcelain commands the output of which is not meant to be parsed but shown interactively and can change in any version.
For plumbing output it behaves differently, there Git usually tries not to make changes or make them stable and backwards compatible, as this output is meant for tool consumption.
So if you use plumbing commands or plumbing versions of commands (which I did in my changes as far as I’m aware), you should be relatively safe against Git updates.

1 Like

AFAIK the big players (GitHub, Microsoft, GitLab) rely on libgit2, which also comes with a business friendly license. I watched a GitHub talk a while ago where they mention digging into the performances of specific Git operations in libgit2 to find out that their complexity was quadratic whereas it should have been linear and fixing it: It’s probably a very well tested and maintained implementation by now.

1 Like

Well, I only find one pre-made Java binding for that and besides that using native libs from Java is almost always a PITA, especially if you want to be fully cross-platform compatible, it is - in their own words - very experimental and very mediocre and they suggest to use JGit instead. :smiley:

Hi,

Small update on the outcome of our internal discussion:

We had issues in the past using the native git commands so we don’t want to blindly switch back to them without understanding where exactly the performance issues come from.
So the plan is:

  • Try to reproduce on our side
  • Investigate were the issue come from
  • Try to fix the issue (either on our side or on JGit)
  • If performance issues can’t be fix without using native commands, we’ll consider using them, and will probably use some part of your implementation

Unfortunately, we are all very busy with the LTS coming soon, so it’s not on our top priority for the time being. In the mean time, I created a task so you can be notified when progress will be made: https://jira.sonarsource.com/browse/SONARSCGIT-39

Best

2 Likes

Hi,

the issue is most definitely in JGit (as it is also seen in Egit) and, according to the following bugtracker item:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=478868

it seems to be related to renaming/moving and at the same time editing files committed to git. In Java, move-and-edit is a very common operation (as you have to change the package line at least when you move), so this is crucial.

We are seeing >40 hours for the initial scan on a well aged (13 years) Git repo of >400kLOC with lots of renames and moves by a list of developers.

Local git is available on the system running the sonar scan, so this would help us. If you change this, please don’t forget the maven plugin :slight_smile:

Thanks :slight_smile:

Did you try to follow my recommendation in [1] ?

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=478868 ?

The recommendation by @msohn sounds interesting. Can someone at SonarSource have a look at this? (Yes, this is a bump. The original problem is still present, and at 40+ hours it is blocking workflows.)

Is there a workaround available to fix the performance issue? Initial load takes 5h to analyze 30k files.