Sonarqube 10.7-Rhel 8-Java 17-Stack Overflow error on scan-

SonarQube Server / Community Build
, ,
1
  • which versions are you using: SonarQube Server 10.7.0.96327/Enterprise - Java 17, Rhel 8, sonar-maven-plugin:5.0.0.4389
  • how is SonarQube deployed: from zip file on Rhel8 VM
  • what are you trying to achieve: Trying to complete a scan of a java project.
  • what have you tried so far to achieve this. (Verify environment, clean build area, rerun).

Main issue
We do the build in gitlab, when the build done Sonarqube begins the scan.

So when the build is complete, we start the last stage, the sonarscan. it says:

[Info] Load branch configuration (done) time-5ms
[ERROR] Error during sonarScanner Engine Execution
org.springframework.beans.factory.UnsatisifiedDependencyException on the jdk.internal.loader.ClassLoaders$AppsClassLoader.
.....  (excluding some lines, can be provided if requested)
at at org.sonar.scanner.bootstrap.ScannerMain.runScannerEngine(ScannerMain.java:149)
at org.sonar.scanner.bootstrap.ScannerMain.run(ScannerMain.java:66)
    at org.sonar.scanner.bootstrap.ScannerMain.main(ScannerMain.java:52)
.....
Caused by org.springframework.beans.factory.beancreationexception.  Error creating bean with name DefaultInputModuleHigherarchy defined in org.sonar.scanner.scan.InputModuleHierarchyProvieder: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiatiionException.
....
Caused by org.springframework.beans.BeansInstantiationException: Failed to instantiate [org.sonar.scanner.scanDefaultInputModuleHierarcyy]: Factory method 'provide' threw exceptiion; nexted exception is java.lang.StackoverflowError.
......
Caused by: Java.lang.stackoverlfowError:null
at java.base/sun.nio.fs.UnixNativeDispatcher.open(Unknown Source)
at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(Unknown Source)
at java.base/java.nio.file.Files.newDirectoryStream(Unknown Source)
at org.sonar.scanner.scan.WorkDirectoriesInitializer.list(WorkDirectoriesInitializer.java:96)
at org.sonar.scanner.scan.WorkDirectoriesInitializer.deleteAllRecursivelyExceptLockFile(WorkDirectoriesInitializer.java:84)
at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanWorkingDir(WorkDirectoriesInitializer.java:64)
at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:59)
at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
. . .
. . . and all the remaining of the total 1,024 stack trace entries repeat this last line at WorkDirectoriesInitializer.java:57.

More info can be shared as needed. The scan project does not create any entries in sonarqube, it just blows up in the build and never gets there. Other builds from the pipeline works.

Any ideas would be helpful. Full Errors from error file attatched.
java-error-stack overflow.txt (123.7 KB)

3

Hi,

I’ll be honest and say this is not the point at which we usually see errors in analysis.

This:

Makes me wonder about the location & structure of the working directory.

Do you have symlinks in your directory? In the path? In the project?

And if so, are they all valid and pointing to places the analysis user can go?

 
Thx,
Ann

5

Ann,

Andy tried to reply to you, but it came back to him, but here is his reply.

Ann: Your symbolic links question is valid, so I answered it by executing the following command for every one of our code repos:

dir /AL /S I:(repoName)

For every one of our code repos, this command came back with “File Not Found.”

The closest thing to symbolic links might actually be how all the other repos are pulled into our “ddp-sonar” repo as submodules. But this is not at all an issue with any of our other build jobs, or any of our other scans. The ddp-common repo is also a submodule, and it scans just fine. The ddp-client repo scan, which is the subject of our other open ticket, actually builds and scans much further and never aborts with a StackOverflowError, even though it does involve some thirteen submodules.

This is why I like your suggestion of using the “-Dsonar.verbose=true” option. I am rerunning the ddp-client scan with that option now, to see what else it can tell us.

But I also like the “-Dsonar.verbose=true” idea for this ddp-server scan, since the StackOverflowError tends to suppress other diagnostics, and maybe this can give us some new information before going down that road to consuming the entire call stack and aborting outright. So I reran this scan first, with the “-Dsonar.verbose=true” option, and its console outputs weren’t as different as I had hoped. But they did have one noticeable difference from before. This time it does not involve the WorkDirectoriesInitializer.list() call, at line 96.

What’s interesting is how these stack traces are really not the same. First for comparison, here is an earlier StackOverflowError stack trace I got, which I am typing in from the high side. This was one I got early on, before using the “-Dsonar.verbose=true” option. Interesting how this one is similar to, but not identical to another stack trace I typed in yesterday:

Caused by: java.lang.StackOverflowError: null
    at java.base/java.lang.ThreadLocal.nextHashCode(Unknown Source)
    at java.base/java.lang.ThreadLocal.<init>(Unknown Source)
    at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreatLocalHoldCounter.<init>(Unknown Source)
    at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync<init>(Unknown Source)
    at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$FairSync<init>(Unknown Source)
    at java.base/java.util.concurrent.locks.ReentrantReadWriteLock.<init>(Unknown Source)
    at java.base/sun.nio.fs.UnixDirectoryStream<init>(Unknown Source)
    at java.base/sun.nio.fs.UnixSecureDirectoryStream<init>(Unknown Source)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(Unknown Source)
    at java.base/java.nio.file.Files/newDirectoryStream(Unknown Source)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.list(WorkDirectoriesInitializer.java:96)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.deleteAllRecursivelyExceptLockFile(WorkDirectoriesInitializer.java:84)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanWorkingDir(WorkDirectoriesInitializer.java:64)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:59)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
. . .

Here is the most recent StackOverflowError stack trace I get today, this time when I run with the “-Dsonar.verbose=true” option:

Caused by: java.lang.StackOverflowError: null
    at java.base/java.util.Collections.$UnmodifiableMap.getOrDefault(Unknown Source)
    at org.sonar.scanner.scan.DefaultInputModuleHierarchy.children(DefaultInputModuleHierarchy.java:70)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:56)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)
    at org.sonar.scanner.scan.WorkDirectoriesInitializer.cleanAllWorkingDirs(WorkDirectoriesInitializer.java:57)

. . .

Hope this helps. @

6

Ann,

These are first time builds, and they do have a sonar yml file in them with a valid sonar login token in them. But since they don’t complete, how do we get to the sonar scan analysis log?

Can I pull it off the server?

Timothy.

7

Hi Timothy,

These aren’t just email threads. Reply notifications are sent automatically to your email if your account is configured for that (it’s the default) but this is primarily a forum. Andy needs to sign up for an account. Then his email replies will be posted to the forum (because the Discourse platform is nifty like that).

Now… the reply you’ve posted to this thread makes heavy reference to the other thread. Are these two things truly separate or should I combine the threads?

And can I please have the full, debug analysis log here?

 
Thx,
Ann

8

Hi again,

Also, please tell me more about this:

 
Thx,
Ann

9

Here is the reply from our end user today.

Ann: Your question about symbolic links is asking me to expound on this from last week:

The closest thing to symbolic links might actually be how all the other repos are pulled into our “ddp-sonar” repo as submodules. But this is not at all an issue with any of our other build jobs, or any of our other scans. The ddp-common repo is also a submodule, and it scans just fine. The ddp-client scan, which is the subject of our other open ticket, fails on an unrelated error. But it actually does build and scan much further than the ddp-server scan does, and never aborts with a StackOverflowError, even though it does involve some thirteen submodules.

When answering your question about symbolic links last week, the purpose of mentioning the GitLab submodule capability was to make the point that our baseline does not have anything close to symbolic links. If submodules were liable to compromise SonarQube scans, they would have done so in other places, not just for the Planner Server scan. That is the point I was making last week.

So why use these “submodules” in GitLab? For any project involving multiple code repos, GitLab offers the capability of cloning a set of other code repos into one project, as “submodules” to the current repo. So in the same way that Planner clones 23 other code repos in our “ddp-release” repo for the purposes of running delivery builds, Planner also clones those same 23 other repos into our “ddp-sonar” repo for the purpose of running SonarQube code scans. These submodules are “links,” only in the sense that they are clones of other repos, in the git sense, making each clone essentially a subdirectory of the ddp-sonar repo. This is how GitLab submodules work, and they are so commonly used that it’s hard to imagine how they could be the problem here.

So now we have a question for you, which expands on Timothy’s question to you from Thursday. Naturally we want more information on what the problem is, just as you do, and I liked where you were headed when you suggested adding in the “-Dsonar.verbose=true” option. You stated that this would make our logs very large, which is clearly what we need. But since using this option in a SonarQube scan job had very little effect on our GitLab scan job console outputs, those very large log outputs are being written somewhere else. Hence Tim’s question about where these go and how to retrieve them.

But I would take this another step further, to see if we could take the guesswork out of what the problem is. Any recursive call like this one will always have the potential for failing to exit and overflowing the stack. So can you find out from your engineering staff what are any other flags to set, switches to turn on, configuration settings to change, or even runtime debug arguments to assign, that would expose what exactly this recursive call is attempting to do? @

Thanks

10

Hi,

I asked for information about your submodules because I remembered we’ve had problems with Git submodules before. So I asked for more data from you while I tried to look up what that trouble was: it turns out it’s with blameing.

Ehm… No. Analysis logging goes to stdout.

Can you just post what you got, please?

 
Thx,
Ann

11

Ann,

We appreciate the hard work here. We are working on this. Here is what I can share.

We have run the scan with the following command:

$ mvn clean install sonar:sonar -Pcode-server-no-test,deploy-dev -B -DskipTest -e -fae -Dsonar.verbose=true -Dsonar.projectKey=[internal name] $SONAR_SCM $SONAR_URL $SONAR_LOGIN $SETTINGS
Picked up JAVA_TOOL_OPTIONS: Xmx8g -Xms512m
[INFO] Error stacktraces are turned on.
[INFO] scanning for projects…
.

.

Warnings about the pom stuff
.

.

[INFO] Reactor Build Order:
[INFO] Module 1 [pom]
…

.

.

.

[INFO] Module 22 [war]

Builds all 22 modules and installs them.

[INFO] Building DDP Top Level Builder Pom [Internal version] [23/22]
then the error after that is the exact same error we have already posted.

This is all I can safely share at this time. The -Dsonar.verbose=true did not offer any new details in the error messages already shared.

Hope this helps?

I think we need to focus on what is sonarqube attempting to do when it says…

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘DefaultInputModuleHierarchy’ defined in org.sonar.scanner.scan.InputModuleHierarchyProvider: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate

We need to ask, what file is it reading in from the build. Where is the input to this? If we figure this out or why we are not getting any extra details is our next step…

Timothy.

12

Hi Timothy,

Since this is a Maven project, can you add -X to the analysis command line? Feel free to break it into two separate commands:

mvn clean install  -Pcode-server-no-test,deploy-dev -B -DskipTest -e
mvn sonar:sonar -e -fae -X -Dsonar.projectKey=[internal name] $SONAR_SCM $SONAR_URL $SONAR_LOGIN $SETTINGS

It’s likely that -X will have the same impact as sonar.verbose, but let’s try the Maven-default.

Then I’d like to see the entire log starting from the analysis command.

Cool. I’d still like to see the rest of the log, which will help me understand the context in which the error occurs.

Yeah, good idea. That’s why I want to see all of the logging that comes before it.

 
Ann

13

The entire output of the analysis section has been shared in the file uploaded to this, stackoverlow.txt.

14

Hi,

Yep. I was looking for the debug-level logs.

 
Thx,
Ann