Stopping .NET scanner fails with "Error during SonarScanner execution: Failed to preprocess files"

mu88 · October 28, 2024, 8:15am

Versions:
- SonarQube: Community Edition v10.6
- .NET Sonar scanner (via NuGet package dotnet-sonarscanner, running as .NET local tool, i.e. via dotnet-tools.json): v9.0.1
how is SonarQube deployed: I don’t know (probably Docker)
what are you trying to achieve: finalize Sonar analysis via dotnet tool run dotnet-sonarscanner end
what have you tried so far to achieve this:
- Tested with v9.0.1 →
- Tested with v9.0.0 →
- Tested with v8.0.0 →
- Tested with v7.1.1 →

Logs

Following you find the logs when running dotnet tool run dotnet-sonarscanner end with v9.0.1:

08:58:08  + dotnet tool run dotnet-sonarscanner end
08:58:11  SonarScanner for MSBuild 9.0.1
08:58:11  Using the .NET Core version of the Scanner for MSBuild
08:58:11  Post-processing started.
08:58:11  07:58:08.536  Starting with Scanner for .NET v8 the way the `sonar.projectBaseDir` property is automatically detected has changed and this has an impact on the files that are analyzed and other properties that are resolved relative to it like `sonar.exclusions` and `sonar.test.exclusions`. If you would like to customize the behavior, please set the `sonar.projectBaseDir` property to point to a directory that contains all the source code you want to analyze. The path may be relative (to the directory from which the analysis was started) or absolute.
08:58:11  Calling the SonarScanner CLI...
08:58:11  INFO: Scanner configuration file: /home/jenkins/agent/.nuget/packages/dotnet-sonarscanner/9.0.1/tools/netcoreapp3.1/any/sonar-scanner-5.0.1.3006/conf/sonar-scanner.properties
08:58:11  INFO: Project root configuration file: /home/jenkins/agent/workspace/ate_major-all-dotnet-local-tools/.sonarqube/out/sonar-project.properties
08:58:11  INFO: SonarScanner 5.0.1.3006
08:58:11  INFO: Java 17.0.11 Eclipse Adoptium (64-bit)
08:58:11  INFO: Linux 5.10.226-214.879.amzn2.x86_64 amd64
08:58:11  INFO: User cache: ?/.sonar/cache
08:58:18  INFO: Analyzing on SonarQube server 10.6.0.92116
08:58:18  INFO: Default locale: "en_US", source code encoding: "US-ASCII" (analysis is platform dependent)
08:58:18  INFO: Load global settings
08:58:18  INFO: Load global settings (done) | time=226ms
08:58:18  INFO: Server id: 3A2253F6-AW0qupzI6yeggYxMEb6x
08:58:18  INFO: Loading required plugins
08:58:18  INFO: Load plugins index
08:58:18  INFO: Load plugins index (done) | time=103ms
08:58:18  INFO: Load/download plugins
08:58:29  INFO: Load/download plugins (done) | time=11720ms
08:58:29  INFO: Process project properties
08:58:29  INFO: Process project properties (done) | time=34ms
08:58:29  INFO: Project key: my-application
08:58:29  INFO: Base dir: /home/jenkins/agent/workspace/ate_major-all-dotnet-local-tools
08:58:29  INFO: Working dir: /home/jenkins/agent/workspace/ate_major-all-dotnet-local-tools/.sonarqube/out/.sonar
08:58:29  INFO: Load project settings for component key: 'my-application'
08:58:29  INFO: Load project settings for component key: 'my-application' (done) | time=86ms
08:58:29  INFO: Load project branches
08:58:29  INFO: Load project branches (done) | time=85ms
08:58:29  INFO: Load branch configuration
08:58:29  INFO: Load branch configuration (done) | time=2ms
08:58:29  INFO: Load quality profiles
08:58:29  INFO: Load quality profiles (done) | time=104ms
08:58:29  INFO: Auto-configuring with CI 'Jenkins'
08:58:29  INFO: Load active rules
08:58:32  INFO: Load active rules (done) | time=3651ms
08:58:32  INFO: Load analysis cache
08:58:32  INFO: Load analysis cache | time=99ms
08:58:33  INFO: Branch name: renovate/major-all-dotnet-local-tools
08:58:33  INFO: Preprocessing files...
08:58:33  INFO: ------------------------------------------------------------------------
08:58:33  INFO: EXECUTION FAILURE
08:58:33  INFO: ------------------------------------------------------------------------
08:58:33  INFO: Total time: 24.752s
08:58:33  INFO: Final Memory: 11M/44M
08:58:33  INFO: ------------------------------------------------------------------------
08:58:33  ERROR: Error during SonarScanner execution
08:58:33  java.lang.IllegalStateException: Failed to preprocess files
08:58:33  	at org.sonar.scanner.scan.filesystem.ProjectFilePreprocessor.processModuleSources(ProjectFilePreprocessor.java:164)
08:58:33  	at org.sonar.scanner.scan.filesystem.ProjectFilePreprocessor.processModule(ProjectFilePreprocessor.java:140)
08:58:33  	at org.sonar.scanner.scan.filesystem.ProjectFilePreprocessor.processModulesRecursively(ProjectFilePreprocessor.java:128)
08:58:33  	at org.sonar.scanner.scan.filesystem.ProjectFilePreprocessor.execute(ProjectFilePreprocessor.java:103)
08:58:33  	at org.sonar.scanner.bootstrap.SpringScannerContainer.doAfterStart(SpringScannerContainer.java:350)
08:58:33  	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:226)
08:58:33  	at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:205)
08:58:33  	at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:144)
08:58:33  	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:226)
08:58:33  	at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:205)
08:58:33  	at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:73)
08:58:33  	at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:67)
08:58:33  	at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
08:58:33  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
08:58:33  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
08:58:33  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
08:58:33  	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
08:58:33  	at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
08:58:33  	at jdk.proxy1/jdk.proxy1.$Proxy0.execute(Unknown Source)
08:58:33  	at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
08:58:33  	at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
08:58:33  	at org.sonarsource.scanner.cli.Main.execute(Main.java:126)
08:58:33  	at org.sonarsource.scanner.cli.Main.execute(Main.java:81)
08:58:33  	at org.sonarsource.scanner.cli.Main.main(Main.java:62)
08:58:33  Caused by: java.nio.file.NoSuchFileException: /home/jenkins/agent/workspace/ate_major-all-dotnet-local-tools/ws8308417f-37eb-4e6a-9791-fc17baa0c210/stylecop.json
08:58:33  	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
08:58:33  	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
08:58:33  	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
08:58:33  	at java.base/sun.nio.fs.UnixPath.toRealPath(Unknown Source)
08:58:33  	at org.sonar.scanner.scan.filesystem.FilePreprocessor.processFile(FilePreprocessor.java:61)
08:58:33  	at org.sonar.scanner.scan.filesystem.ProjectFilePreprocessor.processModuleSources(ProjectFilePreprocessor.java:159)
08:58:33  	... 23 more
08:58:33  ERROR: 
08:58:33  The SonarScanner did not complete successfully

mu88 · October 30, 2024, 10:55am

*push*

Colin · October 30, 2024, 3:51pm

It has been two days, @mu88. This is not commercial support.

From our FAQ:

I created a topic, when can I expect a response?

This is an open community with people volunteering their free time to provide assistance. We’re eager to contribute to the community, but you are not guaranteed a fast response.

Be patient

Wait a few days before bumping a topic that hasn’t received a response.

Do not @name mention individuals not involved in the topic.

Contribute as much as you expect to receive

Contribute to the community (helping others) as much as you expect to receive help.

It is not a replacement for more structured support

If you need SLAs, guaranteed response, privacy, SonarSource also offers Commercial Support.

Martin_Strecker · November 5, 2024, 9:22am

Hello @mu88

Given the logs, it seems the scanner is not able to locate the file /home/jenkins/agent/workspace/ate_major-all-dotnet-local-tools/ws8308417f-37eb-4e6a-9791-fc17baa0c210/stylecop.json.

This is probably due to a bug in the multi-file analysis. You can help us by providing detailed logs by following instructions below. We would need the logs for scanner version v9.0.1 (the failing latest version) and v7.1.1 (the last successful scan). It would also be interesting to know if you have any idea

What the ws8308417f-37eb-4e6a-9791-fc17baa0c210 directory is for? This looks like a generated directory to me. Is the ws prefix referring to WCF? If so, can you provide some more details about your solution and what technologies you are using?
Why is a stylecop.json file is present in that directory?

You can also disable multi-file analysis for your project by following the instructions given in the documentation.

Share the Scanner for .NET verbose logs

Add /d:"sonar.verbose=true" to the…
- SonarScanner.MSBuild.exe or dotnet sonarscanner begin command to get more detailed logs
  - For example: SonarScanner.MSBuild.exe begin /k:"MyProject" /d:"sonar.verbose=true"
- “SonarQubePrepare” or “SonarCloudPrepare” task’s extraProperties argument if you are using Azure DevOps
  - For example:
```
- task: SonarCloudPrepare@1
    inputs:
      SonarCloud: 'sonarcloud'
      organization: 'foo'
      scannerMode: 'MSBuild'
      projectKey: 'foo_sonar-scanning-someconsoleapp'
      projectName: 'sonar-scanning-someconsoleapp'
      extraProperties: |
        sonar.verbose=true
```
The important logs are in the END step (i.e. SonarQubeAnalyze / SonarCloudAnalyze / “Run Code Analysis”)

Share the msbuild detailed logs

MsBuild.exe /t:Rebuild /v:d

or

dotnet build -v:d

mu88 · November 5, 2024, 12:42pm

Hey @Martin_Strecker ,

First of all, thank you for your reply, and I’m sorry if I might have seemed impatient.

I’ve added the requested logs as attachments.
v711.log (914.0 KB)
v901.log (812.2 KB)

What the ws8308417f-37eb-4e6a-9791-fc17baa0c210 directory is for? This looks like a generated directory to me. Is the ws prefix referring to WCF? If so, can you provide some more details about your solution and what technologies you are using?

This is just a temporary directory created during the build to isolate some things.

Why is a stylecop.json file is present in that directory?

This file is used by all projects of the repo, be it testing or production code. Therefore, this file is always placed in the repo’s root. It’s a company-wide pattern across all our apps.

You can also disable multi-file analysis for your project by following the instructions given in the documentation.

I disabled MFA by adding sonar.scanner.scanAll=false and this indeed mitigates the issue. But well, not analyzing YAML and JSON files at all is not a way to go for us.

Martin_Strecker · November 5, 2024, 1:34pm

Thank you for the logs.
The problem is that for multi-file analysis, we collect all files in dotnet-sonarscanner begin. The reasoning is, that we do not want to analyse files that are artefacts of the build process. We later pass these collected files to dotnet-sonarscanner end. I assume, that the following is taking place in this order:
Update: This order is wrong. We collect the files in the end step.

Build preparation creates the ws8308417f-37eb-4e6a-9791-fc17baa0c210 and the artefacts inside of it.
dotnet-sonarscanner begin is executed and finds ws8308417f-37eb-4e6a-9791-fc17baa0c210/stylecop.json among other files
dotnet build is run. Also the temporary ws8308417f-37eb-4e6a-9791-fc17baa0c210 is deleted as part of some clean-up
dotnet-sonarscanner end is executed and because some of the passed files, like ws8308417f-37eb-4e6a-9791-fc17baa0c210/stylecop.json are no longer there, dotnet-sonarscanner end fails.

This is a bug in our workflow and I created a ticket for it.
In the meantime, you can try changing the flow described above. Run dotnet-sonarscanner begin before the temporary ws8308417f-37eb-4e6a-9791-fc17baa0c210 directory is created.

Best, Martin

mu88 · November 5, 2024, 1:43pm

That’s already the case:

The ws<GUID> directory is created to run the tests in isolation, i.e. in the Component Tests section. So it is created after the Sonar scan has been initiated.

Martin_Strecker · November 5, 2024, 2:31pm

You are right. We collect the files in the end step only. I was wrong about that.

I looked into the log and what seems odd to me is the following:
In the original report, the file not found error happens right in the beginning. The failure happens inside the pre-processing. The last INFO message before the execution summary is 08:58:33 INFO: Preprocessing files....
In the logs you provided, the file not found error happens much later. There are several INFO messages before the error occurs. Especially the pre-processing is successful. Several sensors (HTML, C#, javascript, and so on) run successfully until the error occurs:

12:07:10.889 INFO: Preprocessing files...
...
12:07:11.442 INFO: 5 languages detected in 828 preprocessed files
12:07:11.442 INFO: 169 files ignored because of inclusion/exclusion patterns
12:07:11.442 INFO: 30 files ignored because of scm ignore settings
12:07:11.444 INFO: Loading plugins for detected languages
12:07:11.444 DEBUG: Detected languages: [cs, web, json, yaml, docker]
12:07:11.444 INFO: Load/download plugins
..
12:07:36.836 ERROR: Unable to read file: ws0e6a27f7-c90f-4c2b-8c83-809f9adbf92e/src/Company.MyApp.AppStartup/appsettings.Development.json.
..
12:07:36.885 ERROR: Failure during analysis
java.lang.IllegalStateException: Unable to read file: file:///home/jenkins/agent/workspace/my-app_myuser-1730807873048/wse967a25a-1d16-4211-baa2-943f3060f4fd/src/Company.MyApp.DockerCompose/docker-compose.override.yml

It looks like, there is another CI tasks running in parallel which deletes the ws{guid} directory while dotnet-sonarscanner end is executing. Can you take a look, if there is some clean-up running in your pipeline?

mu88 · November 5, 2024, 3:24pm

The cleanup of the ws{guid} directory also happens during the Component Tests phase, i.e. Sonar starts, temp directory gets created, tests run, temp directory gets deleted, Sonar ends.
The cleanup does not happen in parallel while dotnet-sonarscanner end is executing.

mu88 · November 5, 2024, 4:13pm

FYI: I also tried to ignore the temp directory by specifying '/d:sonar.exclusions=\'ws*/**/*,coveragereport*/**/*\''. However, Sonar is still trying to find files in the temp directory for whatever reason and fails with Caused by: java.nio.file.NoSuchFileException: /home/jenkins/agent/workspace/my-app._myuser-1730807873048/wsb1b2a608-18a1-4fc8-b698-356d94ad8fcb/src/Company.MyApp.DockerCompose/docker-compose.override.yml.

Or is my syntax for excluding multiple directories wrong?

costin.zaharia · November 7, 2024, 4:48pm

I also tried to ignore the temp directory by specifying '/d:sonar.exclusions=\'ws*/**/*,coveragereport*/**/*\'' .

Could you please try again with the following?

/d:sonar.exclusions=ws*/**/*,coveragereport*/**/*

mu88 · November 7, 2024, 5:12pm

Sure, I will give it a try and report back next week after the weekend

mu88 · November 11, 2024, 7:42am

No, unfortunately, the error stays the same

mu88 · November 18, 2024, 11:57am

friendly ping

costin.zaharia · November 19, 2024, 1:35pm

Hi,

temp directory gets deleted, Sonar ends

Could it be that the delete operation is running async?

The scanner end command is responsible for running the analysis and this operation is executed in two steps:

create a virtual file system
execute the analysis

From what I can see, some files are available when the virtual system is created but not later. My understanding is that the files are getting deleted while the end command is running.

Based on this, could you please either:

move the deletion step after the scanner end command completes
or find a way to delete the files synchronously (wait for the delete operation to complete)?

mu88 · November 19, 2024, 2:08pm

It seems that your colleague has already asked this and I have answered it (see here). Or what am I missing?

costin.zaharia · November 19, 2024, 2:28pm

Yes, the quote I pasted was from the thread you linked I saw your message.

I’ve also saw this part:

The cleanup does not happen in parallel while dotnet-sonarscanner end is executing.

At the same time, if the files were deleted before the scanner’s end step was called, how would the end step know about them? I have to challenge that statement.

That’s why I ask at least give it a try and move the temporary folder deletion step after the scanner end step (or remove it completely just to test the behavior).

mu88 · November 19, 2024, 3:03pm

Okay, I just wanted to make sure that we don’t go round in circles

When disabling the deletion, the build passes. But I cannot move the temporary folder deletion after the scanner end step because it is encapsulated in a dedicated component, i.e. I’d have to split the component. This is our Jenkinsfile stripped to the bare minimum:

#!groovy

def call(Map params) {
    pipeline {
        stages {
            stage('Install tools from .NET manifest') {
                steps {
                    container("dotnet") {
                        script {
                            dotnetToolInstallAllFromManifest()
                        }
                    }
                }
            }
            stage('Start SonarQube') {
                steps {
                    container("dotnet") {
                        script {
                            def sonarParams = [
                                '/d:sonar.cs.vstest.reportsPaths=\'test-outputs/component/*.trx\'',
                                '/d:sonar.cs.opencover.reportsPaths=\'**/coverage.opencover*.xml\'',
                                '/d:sonar.coverage.exclusions=\'**/Migrations/**\'',
                                '/d:sonar.cpd.exclusions=\'**/Migrations/**\'' ]

                            dotnetSonarStart(key: SERVICE_NAME, installSonarTool: false, host: SONAR_URL, version: VERSION, sonarParameters: sonarParams, useToolFromManifest: USE_DOTNET_TOOL_MANIFEST)
                        }
                    }
                }
            }
            stage('Build') {
                steps {
                    container("dotnet") {
                        dotnetBuild(project: SOLUTION_PATH, dotnetParameters: ["'-p:RuntimeIdentifiers=\"linux-x64\"'"])
                    }
                }
            }
            stage('Component Tests') {
                parallel {
                    stage('Unit Tests') {
                        steps {
                            container("dotnet") {
                                script {
                                    isolationUtil.dotnetTest("ws${UUID.randomUUID()}", INTEGRATION_TEST_PROJECT_PATH, "Unit", "", "test-outputs/component")
                                }
                            }
                        }
                    }
                    stage('Integration Tests') {
                        steps {
                            container("dotnet") {
                                script {
                                    isolationUtil.dotnetTest("ws${UUID.randomUUID()}", INTEGRATION_TEST_PROJECT_PATH, "Integration", "", "test-outputs/component")
                                }
                            }
                        }
                    }
                }
            }
            stage('Parallelization Reports & AppStartup') {
                parallel {
                    stage('End SonarQube') {
                        tools { jdk 'jdk-21' } // JDK is used for Sonar
                        steps {
                            container("dotnet") {
                                script {
                                    dotnetSonarEndFromManifest()
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

The isolation happens in isolationUtil.dotnetTest():

#!groovy

def dotnetTest(workspace, project, category, filter, resultDir) {
    sh "mkdir $workspace"
    sh "mkdir -p $resultDir"
    sh "cp -R tests $workspace"

    def command = "dotnet test " +
            "--configuration Release " +
            "--no-build " +
            "--no-restore " +
            "--logger:trx " +
            "--verbosity minimal " +
            "--filter 'RunOnBuildServer!=false&Category=" + category + filter + "' " +
            "--results-directory " + resultDir + " " +
            "/p:CollectCoverage=true " +
            "'/p:ExcludeByAttribute=\"GeneratedCodeAttribute" + (category == "Unit" ? ",ExcludeFromUnitTestCoverageAttribute" : "") + "\"' " +
            "/p:Exclude=[*]*.Migrations.* " +
            "/p:CoverletOutputFormat=opencover " +
            "/p:CoverletOutput=coverage.opencover." + category + ".xml " +
            "$workspace/$project"

    def statusCode = steps.sh(script: command, returnStatus: true)

    echo "Completed dotnet test for ${project} with result code ${statusCode}"
    if (statusCode > 0) {
        // in case of error, ensure that results are uploaded
        steps.mstest testResultsFile: resultDir + "/*.trx", keepLongStdio: true
        error "Test run completed with errors"
    }

    sh "mv \$(find " + workspace + " -name 'coverage.opencover." + category + ".xml') tests/coverage.opencover." + category + ".xml"
    sh "rm -rf $workspace"
}

I temporarily removed the last line sh "rm -rf $workspace".

Do you see anything suspicious why this runs in parallel?

costin.zaharia · November 19, 2024, 4:33pm

Ok, that’s interesting, removing the delete operation solves the issue.

Unfortunately, I’m not familiar with Jenkins or Unix-based systems. From what I’ve found so far, both the sh and the rm commands should run synchronously and should return only after the operations are complete. I do not understand why the scanner file indexation still finds the files on disk, but I think this is a problem particular to your system, and I don’t consider it a scanner issue.

I’m sorry, but I cannot help more at this point. The solutions I see are to either:

reorganize your pipeline to delete the files after the scanner work is complete.
or, if the contents of the temporary project should not be analyzed at all, you might consider placing the folder outside the scanner base directory. That way the scanner will not see the folder (it’s outside the analysis scope) and the problem will be averted.

Topic		Replies	Views
SonarQube 6.7.4 Post-processing failed. Exit code: 1 SonarQube Server / Community Build csharp , sonarqube , dotnet	6	8190	June 18, 2020
I am getting this error while trying to execute SonarScanner SonarQube Cloud dotnet	1	29	February 4, 2025
Pre-processing Failure: Unable to Fetch SonarQube Version for SonarScanner SonarQube Server / Community Build csharp , scanner , coverage , dotnet	2	894	May 11, 2023
Dotnet sonarscanner 5.2.2 on ubuntu 20.04.3 with .net 5 show error on end post-processing about java SonarQube Server / Community Build dotnet	1	194	January 18, 2024
SonarScanner issue on AWS CodeBuild with .Net Core 2.1 Project SonarQube Server / Community Build scanner , dotnet	7	3502	February 26, 2019

Stopping .NET scanner fails with "Error during SonarScanner execution: Failed to preprocess files"

I created a topic, when can I expect a response?

Be patient

Contribute as much as you expect to receive

It is not a replacement for more structured support

Share the Scanner for .NET verbose logs

Share the msbuild detailed logs

Related topics