.Net CL compilable C/C++ program without .sln project file. SonarQube MSBuild integration failed: SonarQube was unable to collect the required information about your projects

SonarQube Cloud
,
1

Hi All,
I have a C/C++ program which can compilable with .NET’s “CL” command(which known as command line compiler) in developer command prompt.

However this program does not have any “.sln” file. It is not easy to create a “.sln” file because of there is may C/C++ files(nearly ~900) and some special command line parameters(they are not easy to configure those in solution as well). That is why I have to continue to use “CL” command with special parameters(I snippet this part from batch file, you will see them below).

(SNIPPET PART FROM BATCH FILE)
rem Compile files into .obj files in current directory
cl /I"…\testcasesupport" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo /c main.cpp CWE*.cpp CWE*.c …\testcasesupport\io.c …\testcasesupport\std_thread.c
rem Link all .obj file into a exe
cl /FeCWE416 *.obj /I"…\testcasesupport" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo

When I try to execute commands which are suggested by SonarCloud for “C/C++ language” and “Visual C++” compiler are shown below(with their outputs). Last and I guess the important information is that, I used CL command with parameters by replacing “MSBuild /t:Rebuil” part on build-wrapper step.

STEP 1
D:\Tolun\CWE416_Use_After_Free>SonarScanner.MSBuild.exe begin /k:“tolunardahanli_hello-suit” /d:sonar.organization=“tolunardahanli-github” /d:sonar.cfamily.build-wrapper-output=bw-output /d:sonar.host.url=“https://sonarcloud.io” /d:sonar.login=""
SonarScanner for MSBuild 4.4.2
Using the .NET Framework version of the Scanner for MSBuild
Default properties file was found at c:\sonarscanner-msbuild\SonarQube.Analysis.xml
Loading analysis properties from c:\sonarscanner-msbuild\SonarQube.Analysis.xml
Pre-processing started.
Preparing working directories…
05:29:07.905 Updating build integration targets…
05:29:07.925 Fetching analysis configuration settings…
05:29:09.255 Provisioning analyzer assemblies for cs…
05:29:09.265 Installing required Roslyn analyzers…
05:29:09.745 Provisioning analyzer assemblies for vbnet…
05:29:09.745 Installing required Roslyn analyzers…
05:29:09.775 Pre-processing succeeded.

STEP 2
D:\Tolun\CWE416_Use_After_Free>build-wrapper-win-x86-64.exe --out-dir bw-output cl /I"…\testcasesupport" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo /c main.cpp CWE.cpp CWE.c …\testcasesupport\io.c …\testcasesupport\std_thread.c**
main.cpp
CWE416_Use_After_Free__malloc_free_char_43.cpp
CWE416_Use_After_Free__malloc_free_char_62a.cpp
CWE416_Use_After_Free__malloc_free_char_62b.cpp
…
…
Generating Code…
Compiling…
…
…
CWE416_Use_After_Free__return_freed_ptr_16.c
CWE416_Use_After_Free__return_freed_ptr_17.c
CWE416_Use_After_Free__return_freed_ptr_18.c
io.c
std_thread.c
Generating Code…

STEP 3
D:\Tolun\CWE416_Use_After_Free>SonarScanner.MSBuild.exe end /d:sonar.login=""
SonarScanner for MSBuild 4.4.2
Using the .NET Framework version of the Scanner for MSBuild
Default properties file was found at c:\sonarscanner-msbuild\SonarQube.Analysis.xml
Loading analysis properties from c:\sonarscanner-msbuild\SonarQube.Analysis.xml
Post-processing started.
The SonarQube MSBuild integration failed: SonarQube was unable to collect the required information about your projects.
Possible causes:

  1. The project has not been built - the project must be built in between the begin and end steps
  2. An unsupported version of MSBuild has been used to build the project. Currently MSBuild 14.0 and 15.0 are supported
  3. The begin, build and end steps have not all been launched from the same folder
  4. None of the analyzed projects have a valid ProjectGuid and you have not used a solution (.sln)
    Generation of the sonar-properties file failed. Unable to complete SonarQube analysis.
    05:40:10.992 Creating a summary markdown file…
    05:40:10.992 Post-processing failed. Exit code: 1

My further investigation(of course not deep dive) shows that some files and folders which are special for Sonar are creted.

D:\Tolun\CWE416_Use_After_Free>tree /F
└───.sonarqube
│ ├───bin
│ │ │ SonarScanner.MSBuild.Common.dll
│ │ │ SonarScanner.MSBuild.Tasks.dll
│ │ │
│ │ └───targets
│ │ SonarQube.Integration.targets
│ │
│ ├───conf
│ │ │ SonarQubeAnalysisConfig.xml
│ │ │ SonarQubeRoslyn-cs.ruleset
│ │ │ SonarQubeRoslyn-vbnet.ruleset
│ │ │
│ │ ├───cs
│ │ │ SonarLint.xml
│ │ │
│ │ └───vbnet
│ │ SonarLint.xml
│ │
│ └───out
└───bw-output
build-wrapper-dump.json
build-wrapper.log

Any help and/or suggession will be appreciated.

Sincerely,

2

Hello Ardahanli,

First of all, since you are not using MsBuild, and you don’t seem to have any .NET code in your project, only C++ code, you don’t need to use SonarScanner.MSBuild.exe, you can simply use the regular SonarQube scanner (sonar-scanner).

Then, about the build wrapper, I’d like to highlight a point, because I’m not 100% sure that you you currently follow it: You need to run the build wrapper only once over a full compilation of your code. If you run it several times on different source files, it will only keep the last result. So if you manually launch several cl commands, you should put them in a script, and run the build wrapper around this script. If there is only one cl command, everything is fine.

With those changes, your analysis should be able to proceed correctly. If it does not, we will probably need the logs and the build wrapper output to investigate the issue.

A side comment (that is not related to SonarQube in any way) about your build process: even if you don’t like .vcxproj files (I’ve worked on projects with much more than 900 files with .vcxproj files, and it worked very nicely for me, but it’s a question of taste), there are many other build systems (for instance cmake, but once again, it’s a question of taste which one you prefer).

I would suggest you could look at some of them, because directly using cl, especially on many files and when the options differ, is really painful to maintain over time. Moreover, using build systems will also typically improve the experience in the code editors (they will help code completion for instance).

3

Hi @JolyLoic

Thank you for your quick and detailed response.

Development team suggested that I have to use “Visual C++ compiler and Microsoft Libraries” as environment only. That is why they have shared with me those “CL” commands in a batch file which can compile the project with files without problem. By the way, It is not my pference, it is my only choice :slight_smile:

When I try to make selections as below in SonarCloud it suggest me to use MSBuild;
-What is your project’s main language? C, C++ Objective-C
-Which compiler are you using? Microsoft Visual C++

Execute the Scanner for MSBuild from your computer…etc
-SonarQube.Scanner.MSBuild.exe begin /k:“tolunardahanli_hello-suit” /d:sonar.organization=“tolunardahanli-github” /d:sonar.cfamily.build-wrapper-output=bw-output /d:sonar.host.url=“https://sonarcloud.io” /d:sonar.login=""

-build-wrapper-win-x86-64.exe --out-dir bw-output MsBuild.exe /t:Rebuild

-SonarQube.Scanner.MSBuild.exe end /d:sonar.login=""

Project is openSource that is why, I can be able to prepare a sharing link which is including source files.
With that you can simulate same issue by yourself too(Please download “TolunTestcase.zip” from https://drive.google.com/open?id=1qAtHXXfFwH59Y_gP4SBISXgeLGDj8D7n ) -OR- rename attached file from “TolunTestcase.txt” to “TolunTestcase.zip”. (TolunTestcase.txt (2.0 MB))

I have prepared single line of CL command for compilation as you suggested for “build-wrapper” and “sonar-scanner”. However I am getting different errors for analysis stil. I only used those commands below;

STEP 1
-build-wrapper-win-x86-64.exe --out-dir bw-output cl /I"…\…\testcasesupport" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo /FeCWE416 main.cpp CWE*.cpp CWE*.c …\…\testcasesupport\io.c …\…\testcasesupport\std_thread.c
…
Generating Code…

STEP 2
-sonar-scanner.bat -Dsonar.projectKey=tolunardahanli_hello-suit -Dsonar.organization=tolunardahanli-github -Dsonar.sources=. -Dsonar.cfamily.build-wrapper-output=bw-output -Dsonar.host.url=https://sonarcloud.io -Dsonar.login=
WARN: SCM provider autodetection failed. Please use “sonar.scm.provider” to define SCM of your project, or disable the SCM Sensor in the project settings.
WARN: Property ‘sonar.abap.file.suffixes’ is not declared as multi-values/property set but was read using ‘getStringArray’ method. The SonarQube plugin declaring this property should be updated.
WARN: Metric ‘comment_lines_data’ is deprecated. Provided value is ignored.
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 20.514s
INFO: Final Memory: 31M/493M
INFO: ------------------------------------------------------------------------
ERROR: Error during SonarQube Scanner execution
ERROR: Illegal char <*> at index 3: CWE*.cpp
ERROR:
ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging
.
LAST TRY
sonar-scanner.bat -X -Dsonar.projectKey=tolunardahanli_hello-suit -Dsonar.organization=tolunardahanli-github -Dsonar.sources=. -Dsonar.cfamily.build-wrapper-output=bw-output -Dsonar.host.url=https://sonarcloud.io -Dsonar.login=

02:18:11.901 WARN: SCM provider autodetection failed. Please use “sonar.scm.provider” to define SCM of your project, or disable the SCM Sensor in the project settings.
02:18:13.929 WARN: Property ‘sonar.abap.file.suffixes’ is not declared as multi-values/property set but was read using ‘getStringArray’ method. The SonarQube plugin declaring this property should be updated.
02:18:18.406 WARN: Metric ‘comment_lines_data’ is deprecated. Provided value is ignored.
02:18:20.247 ERROR: Error during SonarQube Scanner execution
java.nio.file.InvalidPathException: Illegal char <> at index 3: CWE.cpp
at sun.nio.fs.WindowsPathParser.normalize(Unknown Source)
at sun.nio.fs.WindowsPathParser.parse(Unknown Source)
at sun.nio.fs.WindowsPathParser.parse(Unknown Source)
at sun.nio.fs.WindowsPath.parse(Unknown Source)
at sun.nio.fs.WindowsFileSystem.getPath(Unknown Source)
at java.nio.file.Paths.get(Unknown Source)
at com.sonar.cpp.N.H.A(na:1046)
at com.sonar.cpp.N.H.A(na:1366)
at com.sonar.cpp.plugin.R.A(na:1174)
at com.sonar.cpp.plugin.R.execute(na:1072)
at org.sonar.scanner.sensor.SensorWrapper.analyse(SensorWrapper.java:45)
at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:88)
at org.sonar.scanner.phases.SensorsExecutor.lambda$execute$1(SensorsExecutor.java:65)
at org.sonar.scanner.phases.SensorsExecutor.withGlobalStrategy(SensorsExecutor.java:80)
at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:65)
at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:74)
at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:164)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:319)
at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:314)
at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:288)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:48)
at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:82)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:131)
at org.sonar.batch.bootstrapper.Batch.doExecuteTask(Batch.java:116)
at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:71)
at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
at com.sun.proxy.$Proxy0.execute(Unknown Source)
at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:171)
at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:128)
at org.sonarsource.scanner.cli.Main.execute(Main.java:111)
at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
at org.sonarsource.scanner.cli.Main.main(Main.java:61)

Whole log file(tln.txt) is included in attached TolunTestcase.zip file. it is located under TolunTestcase.zip\testcases\CWE416_Use_After_Free

Hope that there will be any solution.

Sincerely,

4

Hello, I’ll look at your issue Monday, but could you please also attache the build wrappers files to this thread?

Thank you!

5

Hi @JolyLoic

Thank you for your support.

I have been attached your requested bw-output files also.

Note: Do not forget to change extention from “.txt” to “.zip” :slight_smile:

bw-output.txt (4.8 KB)

Sincerely,

6

Hello @Tolun,

We are now able to confirm that the issue you encounter is related to the way the compiler is invoked. You call it with wildcards (*.cpp), which is very unusual (in fact, I did not even know it was possible before I read your post). What happen most of the time is that build systems can use wildcards, but they resolved them, and then they invoke the compiler with an explicit list of files. This is why the use of wildcards is not supported by the build wrapper.

Therefore, there are 3 ways to solve this issue:

  • You can use a build system. From what I’ve seen in your code, I don’t see any reasons that would prevent it, and it would not be difficult to pass exactly the same build options as what you are currently sending.
  • You can update your build script so that it first resolves the wildcards, then runs cl.exe with the explicit list of files.
cl.exe {options} file1.cpp file2.cpp file3.cpp
  • You can update your build script so that it first resolves the wildcards, then runs cl.exe once for each file that needs to be compiled
cl.exe {options} file1.cpp 
cl.exe {options} file2.cpp 
cl.exe {options} file3.cpp

Best regards,

7

Hi @JolyLoic

Hope you will have a nice week…

First of all, I am so sorry for late response.

I have tried all three suggessions. However, I could not be able to solve “end” part’s sending problem to sonarCloud.

After short examining of my possibilities, I found that those commands which are shown below is suitable for uploading to sonarCloud the analysis of project which has not any .sln file(the file which is requiered for msbuild).

  • build-wrapper-win-x86-64.exe --out-dir bw-output cl /I"…\testcasesupport" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo /FeCWE561 main.cpp CWE561_Dead_Code__return_before_code_01.c CWE561_Dead_Code__unused_function_01.c …\testcasesupport\io.c …\testcasesupport\std_thread.c
    main.cpp
    Generating Code…
    Compiling…
    CWE561_Dead_Code__return_before_code_01.c
    CWE561_Dead_Code__unused_function_01.c
    io.c
    std_thread.c
    Generating Code…

  • sonar-scanner.bat -Dsonar.projectKey=tolunardahanli_hello-suit -Dsonar.organization=tolunardahanli-github -Dsonar.sources=. -Dsonar.cfamily.build-wrapper-output=bw-output -Dsonar.host.url=https://sonarcloud.io -Dsonar.login=
    …
    …
    INFO: Task total time: 11.080 s
    INFO: ----------------------------
    INFO: EXECUTION SUCCESS
    INFO: ----------------------------
    INFO: Total time: 14.189s
    INFO: Final Memory: 31M/427M
    INFO: ----------------------------

We have succeded :slight_smile:

May it is not relevant to this case, but my last and only problem is that I would like to take a customized and downloadable report from sonarCloud which should include only “Bugs + Issues”. Is it posssible to take something like that report? Because “OWASP Top 10” and “SANS top 25” reports does not include Bugs. Both of them has disabled “Show CWE distribution” choice. How can I enable it?

Sincerely,

8

So what you did is akin to option 2. I’m happy that it worked out for you.

For your new question, I’m not an expert, but I think that if you want the full power to create a customized report, you can use the web API (at the bottom of each web page in SonarQube, you have a link to the documentation of this API). If you have any issues with that, I suggest you open a new dedicated topic, that will have more visibility.

Thank you!