Using:
Sonarqube developer 8.6.0.39681 (trial)
Sonarqube extension 4.17.0
VS TFS 2018 Version 16.131.27701.1 (on-premises)
I guess I need more details on downloading, installing, and using the c wrapper. Maybe I also need to use another scanner? Not shure. What to do from here?
Hi @Zer0ne,
did you have a look at the documentation? You can have a look at Analyzing a C/C++/Obj-C project section at https://docs.sonarqube.org/latest/analysis/azuredevops-integration/.
Yes, that part of your docs indicate that you can’t mix and have to do either this or that.
This is what I see, what part is saying that and not clear?
Analyzing a C/C++/Obj-C project
In your build pipeline, insert the following steps in the order they appear here. These steps can be interleaved with other steps of your build as long as the following order is followed. All steps have to be executed on the same agent.
- Make Build Wrapper available on the build agent:
Download and unzip the Build Wrapper on the build agent (see Prerequisites section of the C/C++/Objective-C page). The archive to download and decompress depends on the platform of the host.
Please, note that:
- For the Microsoft-hosted build agent you will need to do it every time (as part of build pipeline), e.g. you can add PowerShell script task doing that. This can be done by inserting a Command Line task.
Example of PowerShell commands on a Windows host:Invoke-WebRequest -Uri '<sonarqube_url>/static/cpp/build-wrapper-win-x86.zip' -OutFile 'build-wrapper.zip' Expand-Archive -Path 'build-wrapper.zip' -DestinationPath '.'Example of bash commands on a linux host:
curl '<sonarqube_url>/static/cpp/build-wrapper-linux-x86.zip' --output build-wrapper.zip unzip build-wrapper.zipExample of bash commands on a macos host:
curl '<sonarqube_url>/static/cpp/build-wrapper-macosx-x86.zip' --output build-wrapper.zip unzip build-wrapper.zip
- For the self-hosted build agent you can either download it every time (using the same scripts) or only once (as part of manual setup of build agent).
- Add a Prepare analysis Configuration task and configure it as follow:
Click on the Prepare analysis on SonarQube task to configure it:
- Select the SonarQube Server
- In Choose the way to run the analysis , select standalone scanner (even if you build with Visual Studio / MSBuild )
- In Additional Properties in the Advanced section, add the property
sonar.cfamily.build-wrapper-outputwith, as its value, the output directory to which the Build Wrapper should write its results:sonar.cfamily.build-wrapper-output=<output directory>
- Add a Command Line task to run your build.
For the analysis to happen, your build has to be run through a command line so that it can be wrapped-up by the build-wrapper. To do so,
- Run Build Wrapper executable. Pass in as the arguments (1) the output directory configured in the previous task and (2) the command that runs a clean build of your project (not an incremental build).
Example of PowerShell commands on a Windows host with an MSBuild build:build-wrapper-win-x86/build-wrapper-win-x86-64.exe --out-dir <output directory> MSBuild.exe /t:RebuildExample of bash commands on a linux host with a make build:
build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir <output directory> make clean allExample of bash commands on a macos host with a xcodebuild build:
build-wrapper-macosx-x86/build-wrapper-macos-x86 --out-dir <output directory> xcodebuild -project myproject.xcodeproj -configuration Release clean build
- Add a Run Code Analysis task to run the code analysis and make the results available to SonarQube. Consider running this task right after the previous one as the build environment should not be significantly altered before running the analysis.
- Add a Publish Quality Gate Result task.
You show me the section regarding only c++, c# is another section and therefore indicate it is two different directions of stuff to setup. Just before, it says “Select your build technology below to expand the instructions …”
Another place https://docs.sonarqube.org/latest/analysis/languages/cfamily/ you have the docs for a mix, but that is just confusing. Seems like I would have to start all over with another scanner.
But nice if I can “just” add the wrapper on top of what I have done so far, from the c++ description?
Hi @Zer0ne in another thread I showed my setup.
@milbrandt So I’ve tried an exact copy of your build wrapper script but get:
Unexpected token ‘out-dir’ in expression or statement.
@Zer0ne It should be obvious that you need to define your variables in the yaml. It is not the whole azure-pipelines.yml which was posted - you will e.g. miss also die SonarAnalyze task.
'$(SrcDir), $(SonarOutDir), $(build_wrapper). $(msbuildPath), $(codeCoveragePath), $(BuildConfiguration), $(BuildPlatform) are all variables which are used and need to be defined in your yaml.
@milbrandt Well not using yaml, setting variables in TFS UI, exact copy of your script resolves to this:
New-Item -Path C:\TfsBuildAgent_work\25\wrapperoutput -ItemType directory -Force
Set-Location C:\TfsBuildAgent_work\25\s\solution
“C:\Program Files\SonarQubeBuildWrapperForCLang\build-wrapper-win-x86-64.exe” --out-dir C:\TfsBuildAgent_work\25\wrapperoutput “C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe” C:\TfsBuildAgent_work\25\s\solution\Solution.sln /t:Rebuild /nologo /nr:false /t:“Clean” /p:FullRelease=true /p:platform=ARM /p:configuration=debug /p:VisualStudioVersion=“16.0” /m’
Found a fix for the error though, needs and ampersand in front of the build command like so:
New-Item -Path (SonarOutDir) -ItemType directory -Force
Set-Location (SrcDir)
& "(build_wrapper)" --out-dir (SonarOutDir) "(msbuildPath)\MSBuild.exe" (SrcDir)\Solution.sln /t:Rebuild /nologo /nr:false /t:“Clean” /p:FullRelease=true /p:platform=(BuildPlatform) /p:configuration=(BuildConfiguration) /p:VisualStudioVersion=“16.0” /m
But in the end it fails anyway because now project references is not found.
Finally!
I started out with a functional c# setup.
Sonarqube supply the c-family build wrapper exe by a link to your sonarqube server:
{SonarQube URL}/static/cpp/build-wrapper-win-x86.zip - unzipped it and put the exe in program files on build server.
Added the line:
sonar.cfamily.build-wrapper-output=$(SonarOutDir)
to Build process → Prepare analysis on SonarQube → Advanced → Additional Properties
Added an inline powershell script (thanks to @milbrandt ) to build process instead of the Visual Studio Build plugin, notice I used extra msbuild arguments after VisualStudioVersion, that you might not, and remember to get the version right, I used VS 2017 which is v.15, on my build server:
New-Item -Path $(SonarOutDir) -ItemType directory -Force Set-Location $(SrcDir) & "$(build_wrapper)" --out-dir $(SonarOutDir) "$(msbuildPath)\MSBuild.exe" $(SrcDir)\Solution.sln /t:Rebuild /nologo /nr:false /t:"Clean" /p:FullRelease=true /p:platform=$(BuildPlatform) /p:configuration=$(BuildConfiguration) /p:VisualStudioVersion="15.0" /p:AppxBundlePlatforms="$(BuildPlatform)" /p:AppxPackageDir="$(Build.ArtifactStagingDirectory)\AppxPackages\\" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=StoreUpload /m
All the variables used above needs to be set in TFS 2018 UI as so. Make sure build_wrapper path matches where you put it, and find the path to your version of msbuild.exe:
Apparently not the solution anyway, it scans the whole repo from root. I would want it to scan all and only the files used in the vs solution I compile or at least just files within project directory with possibility to include other directories in levels above. Exclusions will not work in my case.
Why is it even able to scan cpp files in other directories not included in the build, I thought the c-wrapper was needed for that?
Also tried to setup another project only to realise that it scans the same files and fails to acknowledge it’s the same source code. Should I pay for the same lines twice?
Maybe Sonarqube is just not any help
Hello @Zer0ne,
Let me follow up on this.
As far as I understand you resolved the problem of analyzing a project that’s a mix of C# and C++.
I will not get into the intricacies of TFS (because I am no expert in TFS) but in a nutshell the process is:
scanner begin /d:sonar.cfamily.build-wrapper-output=<someTempLocalDir> ...
build-wrapper --out-dir <someTempLocalDir> <yourC#C++MixBuildCommand>
scanner end ...
The build wrapper executable can either be installed statically on the build agents or downloaded dynamically from the SonarQube server as @mpaladin explained. The latter is better because it guarantees that you’ll always used the latest version of the build wrapper. (Every time you upgrade the SonarQube server you are likely to have a new version of the wrapper)
<yourC#C++MixBuildCommand> can be any command that would build the code, but typically some MSBuild type of command in the .NET world.
That was for my understanding of the current status.
I also understand that you have a remaining problem: How can you analyze only a subset of your entire monorepo, especially in the context of C++ analysis ?
The problem is not easy, but there are usually some solutions. However it is difficult to give guidance on this without a clear understanding of the structure of your monorepo and of the pieces that you want to analyze separately.
If you give some visibility on this structure (mostly the directory structure AND the root location of each .sln .xxproj you would like to analyze separately) I will be glad to offer you some options.
Olivier
When I use visual studio plugin for TFS (and exclude c-family) I get a result that contains only the files in paths used by the project, but when I switch to msbuild command with the c-wrapper I get a result that includes all of my repo (above 2 mill. lines), even cpp files not used in compilation. Should not be standard behaviour, but you say its fixable.
Following is a little explanation of the complexity of the repo:
repo\dirA\dirI
Several additional solutions in repo\DirA also with dependancy to projects E, F, and G
repo\dirD\dirK
SolutionB.sln
SolutionB.sln
repo\dirA\dirU
Here we have something similar to the above just with a different cpp lib, that by the way includes a lot of identical code (work in progress).
Following solutions use in total 64 cs and vcx projects whereof 5 is shared. And they are part of the same build process. And a lot of code is identical to code in the cpp lib from ProjectA and the other one from within path repo\dirA\dirU, both copied code and code from the same paths.
repo\dirR\dirS
SolutionE.sln
repo\dirR\dirT
SolutionF.sln
…and many more…
I’ve tried to build the small SolutionA and a similar one with and without c-family and the big SolutionE with c-family, all c-family builds resulted in the same above 2 mill. lines analysed, but still somewhat different results. So you might be able to tell me how to exclude some paths from analysis, but notice, doing that with all our CI build processes every time we add a new path, is not a solution.
Is Sonarqube able to automatically not count identical code twice?
Hello,
Let’s start by answering your final question
Is Sonarqube able to automatically not count identical code twice?
The answer is no. If you have some same code that’s included in several analysis (eg one of you shared projects that you don’t exclude explicitly from all but one scan) this code will be analyzed and included in the SonarQube report as many times as it appears. This may be acceptable… or not.
Customers having this code structure and that want the code reported only once in SonarQube typically build (and analyze) the shared projects separately from the rest and exclude it from all the other projects using them (using sonar.exclusions)
Now coming to your main problem. Your repo is definitely complicated. Your description gives me a good first overview but I still have a couple of questions:
The general rules that you should respect:
sonar.exclusions or sonar.inclusions propertiesSo let’s say that you want to analyze SolutionA.sln located in repo\dirA\dirI.
I understand that the underlying projects are under different directory locations (repo\dirC repo\dirA)
So indeed the root dir that include all the code is repo, and you may get too much analyzed if you don’t exclude files.
Can you check if passing the property sonar.inclusions=dirC/**/*,dirA/dirB/dirG/**/* does not achieve what you expect (the setting is only to cover files of ProjectA, you may have to add the directories for the files of projectB, E D and G if you want the whole scope of the SolutionA to be analyzed
cd repo
SonarScanner.MSBuild.exe begin /d:sonar.login=... /k:... /d:"sonar.inclusions=dirC/**/*,dirA/dirB/dirG/**/*,..."
build-wrapper-win-x86-64.exe --out-dir bw-out MSBuild /t:rebuild dirA\dirI\SolutionA.sln
SonarScanner.MSBuild.exe end /d:sonar.login=...
Let me know.
In this topic I already typed in a powershell script, but now I am trying following command:
"$(build_wrapper)" --out-dir $(SonarOutDir) "$(msbuildPath)\MSBuild.exe" $(SrcDir)\SolutionA.sln /t:Rebuild /nologo /nr:false /t:"Clean" /p:FullRelease=true /p:platform=$(BuildPlatform) /p:configuration=$(BuildConfiguration) /p:VisualStudioVersion="15.0" /p:AppxBundlePlatforms="$(BuildPlatform)" /p:AppxPackageDir="$(Build.ArtifactStagingDirectory)\AppxPackages\\" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=StoreUpload /m
Let’s focus on changing root dir first.
I have tried changing the commands working directory in TFS UI, but that didn’t change anything other than the path to .sln needs to change accordingly, if defined relatively. TFS runs the command from repo as standard. Changing the “Sources root directory” in “Prepare analysis on Sonarqube” from . to anything else gives following error:
java.lang.IllegalStateException: The "build-wrapper-dump.json" file was found but 0 C/C++/Objective-C files were analyzed. Please make sure that:
* you are using the latest version of the build-wrapper and the CFamily analyzer
* you are correctly invoking the scanner with correct configuration
* your compiler is supported
* you are wrapping your build correctly
* you are wrapping a full/clean build
* you are providing the path to the correct build-wrapper output directory
* you are building and analyzing the same source checkout, absolute paths must be identical in build and analysis stepsAdding sonar.inclusions=dirC/**/* results in the samme error.
Hello,
Can you send:
Olivier
Yes. I did an exclusion test today and that worked, switched back to run with only inclusion and failure, then grabbed all the logs from today and the failed build process. How to send you file privately?
Hello @Zer0ne ,
I wonder why you did not stick to your exclusions setting if you achieved what you wanted ?
In case there’s some good reasons you may send me your logs privately through direct message.
Click on my avatar and then message.
Olivier
Just saying that typing in an exclusion removed that directory from my analysis. That should at least give some thought about how inclusions may be different, if a bug is to be found.
Why it is not what I wanted, is because adding new exclusions to several individual projects whenever a developer, decides too add a directory, is just not the way to go. I guess yet another complication with monorepos that needs focus.
In my line of thought it is a bug that c-wrapper analysis reads files all the way from root that is not included in build process. Does it even make sense to include a cpp file that is not compiled?
If it can’t be better set up than now I can see only two usage approaches for us:
We choose a single but big cpp project. It includes all of our repo, so I guess it will not work that well with CI and Quality Gate. Only compiled cpp files gets a wrapper analysis? Well I expect a lot of deficiencies so I also expect a discount. In return I will include as many LOC as possible.
We exclude cpp completely which is our main language and only setup a few of the small projects.
Or did I misunderstand something?
Cannot see a pm feature from clicking on your avatar.