Sonar Scanner Parallel analysis fails - Monorepo Configuration

SonarQube Server / Community Build
, ,
1

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
  • how is SonarQube deployed: zip, Docker, Helm
  • what are you trying to achieve
  • what have you tried so far to achieve this

Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!
Trying to run parallel analyses inside an angular monorepo project, where analysis fails to run when running in parallel, here is the error logs for more information

[09:16:24] Trying to find a local install of the SonarScanner: /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner

/bin/sh: 1: /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner: not found

[09:16:24] Proceed with download of the platform binaries for SonarScanner...

[09:16:24] Creating /home/pwuser/.sonar/native-sonar-scanner



[09:16:24] decompressing /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-cli-5.0.1.3006-linux.zip into /home/pwuser/.sonar/native-sonar-scanner

[09:16:26] ERROR: impossible to download and extract binary: ETXTBSY: text file is busy, open '/home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/jre/bin/java'

[09:16:26]        SonarScanner binaries probably don't exist for your OS (linux).

[09:16:26]        In such situation, the best solution is to install the standard SonarScanner (requires a JVM).

[09:16:26]        Check it out at https://redirect.sonarsource.com/doc/install-configure-scanner.html

The SonarQube scan failed for project 'nx-sonar-p1'

Error: ETXTBSY: text file is busy, open '/home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/jre/bin/java'

    at Object.openSync (node:fs:582:18)

    at Utils.writeFileTo (/home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/adm-zip/util/utils.js:81:22)

    at /home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/adm-zip/adm-zip.js:618:27

    at Array.forEach (<anonymous>)

    at Object.extractAllTo (/home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/adm-zip/adm-zip.js:606:26)

    at getSonarScannerExecutable (/home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/sonarqube-scanner/src/sonar-scanner-executable.js:82:9)

    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

    at async scan (/home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/sonarqube-scanner/src/index.js:36:28)

    at async module.exports.async (/home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/sonarqube-scanner/src/index.js:63:3)

on the second task I see below error

[09:16:24] Starting analysis...

[09:16:24] Executable parameters built:

[09:16:24] {

  httpOptions: {},

  targetOS: 'linux',

  installFolder: '/home/pwuser/.sonar/native-sonar-scanner',

  platformExecutable: '/home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner',

  fileName: 'sonar-scanner-cli-5.0.1.3006-linux.zip',

  downloadUrl: 'https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-5.0.1.3006-linux.zip'

}

[09:16:24] Trying to find a local install of the SonarScanner: /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner

/bin/sh: 1: /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner: not found

[09:16:24] Proceed with download of the platform binaries for SonarScanner...

[09:16:24] Creating /home/pwuser/.sonar/native-sonar-scanner



[09:16:24] decompressing /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-cli-5.0.1.3006-linux.zip into /home/pwuser/.sonar/native-sonar-scanner

[09:16:26] decompressed /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  SIGBUS (0x7) at pc=0x00007fb748652540, pid=103, tid=112

#

# JRE version: OpenJDK Runtime Environment Temurin-17.0.7+7 (17.0.7+7) (build 17.0.7+7)

# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (17.0.7+7, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

# Problematic frame:

# V  [libjvm.so+0xa8b540]  LinkResolver::resolve_method(LinkInfo const&, Bytecodes::Code, JavaThread*)+0x240

#

# Core dump will be written. Default location: /home/jenkins/workspace/NEWS_test-pipeline_PR-1/core.103

#

# An error report file with more information is saved as:

# /home/jenkins/workspace/NEWS_test-pipeline_PR-1/hs_err_pid103.log

#

# If you would like to submit a bug report, please visit:

#   https://github.com/adoptium/adoptium-support/issues

#



[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x0000000000004206]



The SonarQube scan failed for project 'nx-sonar-p3'

Error: Command failed: /home/pwuser/.sonar/native-sonar-scanner/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner --from=ScannerNpm/3.1.0

    at genericNodeError (node:internal/errors:984:15)

    at wrappedFn (node:internal/errors:538:14)

    at checkExecSyncError (node:child_process:890:11)

    at execFileSync (node:child_process:926:15)

    at scan (/home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/sonarqube-scanner/src/index.js:43:3)

    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

    at async module.exports.async (/home/jenkins/workspace/NEWS_test-pipeline_PR-1/node_modules/sonarqube-scanner/src/index.js:63:3)

Can you please check and direct what needs to be done for a parallel analysis

2

Hi,

It’s not clear to me what exactly you’re trying to parallelize (two analyses running side by side in a race?).

While some work can be parallelized inside an analysis, that’s language-dependent and not available for Angular / JavaScript.

Running two analyses at the same time for the same project is just going to create a race condition in which you will see the first analysis to start error-out on the server side if it didn’t also finish first on the CI-side.

Now, if what you’re trying to do is analyze different projects within the monorepo at the same time, well, that should work since projects are independent of each other.

And in that context, perhaps you should just pre-install the scanner on the build agent and not make it part of the build script:

Either that, or make the job install the scanner in a unique location (maybe directly in the workspace?) for each job.

 
HTH,
Ann

3

Sorry for not giving full context before.

We have an angular monorepo and we are trying to have parallel sonar analysis for each of the project. We use sona qube DCEdition Version 9.9.2.

So far we are analysing entire project with modules and branch analysis takes 13+ mins, but with new monorepo support in sonarqube,

here is what we are trying to do parallel

  • enabled flag monorepo on the General Settings>DevOps Integration
  • updated sonar.properties file like below
sonar.projectKey=angular-test-monorepo:news
sonar.projectName=Angular Test Monorepo
sonar.projectVersion=1.0

sonar.exclusions=**/*mock.ts,**/*mocks.ts,**/mock*.ts,**/index.html,**/*descriptor.ts,**/mock/*.ts,**/auto-login.service.spec.ts,**/jsonpath.service.ts,**/travel-order/**/*.model.ts,**/generated/*.model.ts
sonar.coverage.exclusions=**/*spec.ts,**/index.html,**/index.ts,**/main.ts,**/jest-polyfills.ts

nx-sonar-p1.sonar.projectName=nx-sonar-p1
nx-sonar-p1.sonar.projectBaseDir=packages/nx-sonar-p1
nx-sonar-p1.sonar.typescript.lcov.reportPaths=../../coverage/nx-sonar-p1/lcov.info
nx-sonar-p1.sonar.typescript.tsconfigPath=./tsconfig.sonar.json
nx-sonar-p2.sonar.projectName=nx-sonar-p2
nx-sonar-p2.sonar.projectBaseDir=packages/nx-sonar-p2
nx-sonar-p2.sonar.typescript.lcov.reportPaths=../../coverage/nx-sonar-p2/lcov.info
nx-sonar-p2.sonar.typescript.tsconfigPath=./tsconfig.sonar.json
nx-sonar-p3.sonar.projectName=nx-sonar-p3
nx-sonar-p3.sonar.projectBaseDir=packages/nx-sonar-p3
nx-sonar-p3.sonar.typescript.lcov.reportPaths=../../coverage/nx-sonar-p3/lcov.info
nx-sonar-p3.sonar.typescript.tsconfigPath=./tsconfig.sonar.json
nx-sonar-p4.sonar.projectName=nx-sonar-p4
nx-sonar-p4.sonar.projectBaseDir=packages/nx-sonar-p4
nx-sonar-p4.sonar.typescript.lcov.reportPaths=../../coverage/nx-sonar-p4/lcov.info
nx-sonar-p4.sonar.typescript.tsconfigPath=./tsconfig.sonar.json

are these steps enough to have parallel analysis for the project?
we just have one project in sonar-qube, I understood I have to create one project ( in sonar-qube) for each of the subprojects, am I right?
Any reference configuration or an example project for the same?

4

Hi,

I’m still a bit confused and I believe this comes down to definitions. You’ve called this a “monorepo”, which means multiple - largely independent - projects in the same SCM repo.

You seem to have (partially) configured this as a multi-module project. Sorta.

So. If these projects are truly independent of each other, put a sonar-project.properties file in each project’s directory and analyze them independently, and in parallel if you like.

If these are modules of the same project, then no, you cannot parallelize analysis within a project except for some languages, as mentioned above.

The configuration you’ve provided is incorrect and will not work. You should just scrap it and answer the question: is this a single, multi-module project or a monorepo?

 
HTH,
Ann

5

Hi Ann,
thanks for the clarification, we have this project as a monorepo (built with Nx.dev), something similar to GitHub - nrwl/nx-examples: Example repo for Nx workspace, where apps folder will have our sub projects.

Currently we run sonar validation with each of the subprojects configured as modules inside a single sonar-project.properties file.

Now we are trying to make the analysis for each of the subprojects/modules and also in parallel as they can be run independently, so that we reduce the sonar validation time.

So far tried enabling monorepo flag in project settings ( in the existing repo) and with the above mentioned one repo in sonar and bitbucket as well (monorepo), yet to create sonar-qube projects for each module.

6

Can I get a sample configuration for project example show in above, it follows angular workspace like here GitHub - angular/angular: Deliver web apps with confidence 🚀, where all the subprojects are part of packages/ folder

7

Hi,

I’m still not clear on what you’re trying to accomplish. I can only point you to the docs.

 
HTH,
Ann