Error: Locking FileBasedConfig[/root/.config/jgit/config] failed after 5 retries

SonarQube Cloud
,
1

Bitbucket Cloud with SonarCloud integration
Scanner command used: ./gradlew build jacocoTestReport sonarqube
Gradle with sonarqube plugin: id “org.sonarqube” version “2.7.1”
Languages of the repository: Java and Javascript
Error observed: At the end of the build, we see the following in the output causing a delay for every pipeline using sonar:

:jacocoTestReport
:sonarqube
locking FileBasedConfig[/root/.config/jgit/config] failed after 5 retries
BUILD SUCCESSFUL

Steps to reproduce: Run any Bitbucket pipeline using the Gradle tasks listed above

Also, a build has failed with the following (passes normally, no changes have been committed):

locking FileBasedConfig[/root/.config/jgit/config] failed after 5 retries
:sonarqube FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sonarqube'.
> The 'report' parameter is missing
2

Can you please run the Gradle command again with --stacktrace option and share the output?

3

I changed the pipeline to run ./gradlew --refresh-dependencies --stacktrace build publish jacocoTestReport sonarqube, however there was no change to the build output:

:assemble
:compileTestJava
Download <removed>
:processTestResources
:testClasses
:test
:check
:build
:publish UP-TO-DATE
:jacocoTestReport
:sonarqube
locking FileBasedConfig[/root/.config/jgit/config] failed after 5 retries
BUILD SUCCESSFUL
4

I’m seeing this too, after a recent SonarQube upgrade. I’ve also tried updating the Gradle plugin to 2.8, but the same warning still manifests.

6

Could you please post debug logs, by adding -d?
Is there any chance you have builds of the same project running in parallel?

7

pipelineLog-843.txt (80.0 KB)

Attached is the log output - I had to trim it before/after, as well as removed thousands of duplicate lines to make it small enough to upload.

We do run parallel builds in the same repos, but the error also occurs when only one build is running.

8

Any updates?

9

The library we use to interact with git, JGit, uses file locks when saving jgit’s config to prevent concurrent write accesses.
Looking at their code, it looks like the look is implemented by atomically creating a file with a different extension for /root/.config/jgit/config. That is in the project’s base directory, right?
It’s happening at a step where no multithreading is used, so it’s unlikely that it’s a concurrent access within the process. It seems that the project is being analyzed concurrently by different processes.

10

I have the same error in, albeit for a different path of the config file:

WARN: locking FileBasedConfig[/home/scanner-cli/.config/jgit/config] failed after 5 retries

But since this is run during GitLab CI with the sonarsource/sonar-scanner-cli:4.2 Docker image, I am pretty sure this is not a race condition between different processes.

11

Hi Teake, do you have the logs of it?

12

Sure thing, here’s a debug log.

Note that this job was run on a derived Docker image, namely omegacen/sonar-scanner-cli.

job_log.txt (2.4 MB)

13

Also just checking in to see if there’s been any progress; here’s my debug log if that helps.

debug.txt (2.2 MB)

14

Thanks everyone.

I found the root cause of the problem, and it’s a bug in JGit.
A deadlock happens when JGit’s configuration file and the git repository are located in different drives.
However, I believe it has no impact to the scanner (other than the warning in the logs).

I reported the bug to JGit. It should soon show up here: Eclipse Community Forums: EGit / JGit (see copy below)

There is a workaround that will also speedup the execution of JGit and that we plan to implement in the next release of the Git SCM plugin: Log in with Atlassian account.

Since that’s likely not the cause of the error originally reported, have you seen again the error with “> The 'report' parameter is missing”, @jseletz?
Any debug logs of that?

.
.
.

JGits bug report:
v5.6.1.202002131546-r

Description
If a git repository and JGit’s configuration file are located in different drives, JGit will fail to save the filesystem’s attributes when it runs for the first time.
The problem is that there is a deadlock involving the configuration file’s lock and the generation of both filesystem’s attributes.

Here is how the deadlock happens:

  1. JGit tries to create a FileSnapshot for the repository’s index
  2. Doesn’t have the FS attributes saved for that drive, so generates it
  3. Proceeds to save it in the configuration file located in the other drive
  4. Next it creates a lock file for the configuration file
  5. Tries to create a FileSnapshot for the lock file.
  6. Repeates steps 2-4 for the attributes of the other drive, and step 4 fails because the lock file already exists.

Impact
Note that it still manages to save the filesystem attributes for 1 of the drives, so next time it runs it will successfully save the attributes for the other drive. A warning is logged but it shouldn’t directly impact applications using the library.

Using 2 drives might sound like a corner case but turns out that CI pipelines with Atlassian and GitLab have this kind of setup in their worker nodes.

Reproducer
You’ll need 2 drives, let’s call them /d1 and /d2.

  1. Place a clone in /d1, let’s say /d1/clone.
  2. Set the environment variable XDG_CONFIG_HOME=/d2
  3. Run the following.
  @Test
>   public void reproduce() throws IOException {
>     try (Repository repo = new RepositoryBuilder()
>       .findGitDir(baseDir.toFile())
>       .setMustExist(true).build()) {
>     }
>   }

Workaround
Setting FS.setAsyncFileStoreAttributes(true) breaks the deadlock loop.

15

No, I have not seen those missing parameter errors again. Just the locking config error. Thanks!