Csharp files of subprojects are not scanned by SonarQube

Adrie · February 19, 2020, 4:32pm

How is our software special project structure:

main (contains the solution main.sln)
main\Shared.ToBasis (contains the shared CSharp sources)
main\ToBasis\R231 (contains the project to build ToBasis for AutoCAD 2019)
the subproject file ToBasis_R231.csproj contains the following line:

<Import Project="..\..\Shared.ToBasis\Shared.ToBasis.projitems" Label="Shared" />

as a reference to the location of the sources files.

The main reason for this split-up is building modules for different AutoCAD versions with a shared code base.

Our problem is that the csharp files of these subprojectes are not included anymore in a SonarQube scan.

In our TFS build log i found the following lines:

2020-02-18T15:50:59.2644907Z INFO: Indexing files of module ‘ToBasis_R230’
2020-02-18T15:50:59.2646163Z INFO: Base dir: D:\Agent.2\110\s\ToBasis\R230
2020-02-18T15:50:59.2646542Z INFO: Source paths: packages.config
2020-02-18T15:50:59.2646896Z INFO: Excluded sources for coverage: **/*.cs

this is diffentent from a subproject thats not in a submap:

2020-02-18T15:50:59.2051436Z INFO: Indexing files of module ‘XmlCommon’
2020-02-18T15:50:59.2051759Z INFO: Base dir: D:\Agent.2\110\s\XmlCommon
2020-02-18T15:50:59.2052105Z INFO: Source paths: Properties/AssemblyInfo.cs, CommonXmlException.cs, Convert.cs…
2020-02-18T15:50:59.2052467Z INFO: Excluded sources for coverage: **/*.cs

This subproject contains sources paths with files included and this subproject works fine with SOnarQube.

Then i looked in the logging of SonarQube on our build server

I found the following files FilesToAnalyze.txt

D:\Agent.2\110\s\ToBasis\Properties\AssemblyInfo.cs
D:\Agent.2\110\s\Shared.ToBasis\ArcToLine.cs
D:\Agent.2\110\s\Shared.ToBasis\CommandHandler.cs
D:\Agent.2\110\s\Shared.ToBasis\Common.cs
D:\Agent.2\110\s\Shared.ToBasis\DI\Container.cs

and this file contains all the files of the subproject. But still all of these files are not scanned by SonarQube.

I hope somebody understands what i am doing.

And hopefully somebody knows a solution.

Finally: We are working with SonarQube 7.9 and the latest versions of all scanners and we are using the SonarQube plugin on TFS version 2015.

czurn · February 25, 2020, 10:03am

Hello @Adrie,

Thanks for the detailed information you provided, it helps a lot !

A little bit of context here: the scanner for MSBuild currently sets the base directory to the project it is building. In your case, when building ToBasis_R231.csproj, the base directory will be main\ToBasis\R231 (as shown in the logs: 2020-02-18T15:50:59.2646163Z INFO: Base dir: D:\Agent.2\110\s\ToBasis\R230). This is where the problem comes from, since the shared files are outside this base directory, even if they were detected and added to the FilesToAnalyze.txt list, they will not be resolved properly.

To avoid this issue, the base directory should manually be set to the common parent directory, so that all source files that needs to be analyzed will be found when running the analysis.
Can you try to add the following property /d:sonar.projectBaseDir={pathToMainFolder} (this property can also be set relatively using /d:sonar.projectBaseDir=../.. in your case, since the common parent directory is 2 level higher) to the begin step of the sonarscanner, this should allow for the shared files to be found by the analyzers.
Ex: dotnet sonarscanner begin /k:myProjectSharedTest /d:sonar.projectBaseDir=.. {other_scanner_properties}

Let me know if this fixes the issue for you.

Best,
-Chris

Adrie · March 12, 2020, 4:59pm

Hi Chris,

Finally i had time to try your solution.

I’ve tried to add the following parameters to the “Prepare the SonarQube analysis” task in TFS.

sonar.projectBaseDir=$(Build.SourcesDirectory)

and the second option:

sonar.projectBaseDir=…/…

both parameters doesn’t change annything and the csharp files in the subfolders are not included in the sonarqube scan. I tried to find out if i can check somewhere if this parameter is used. Do you know where i can find this information.

Regards,
Adrie

czurn · March 27, 2020, 1:44pm

Hello Adrie,

Just to be sure, when you say you tried the second option sonar.projectBaseDir=…/…, can you confirm it’s a typo in the posted message and that the actual parameter you tried was: sonar.projectBaseDir=../.. ?

Ok, if this did not work for you, could you try to add the /d:sonar.verbose=true parameter (See https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/) to the begin step of the analysis, and then share the full log of the end step of the analysis. These debug logs should display more information and would help to find the root of the issue.

Thanks.

Best,
-Chris

Adrie · March 27, 2020, 3:26pm

Hi Chris,

I can confirm that I use two dots instead three in my post.

sonar.sourceEncoding=windows-1252
sonar.projectBaseDir=…/…
sonar.verbose=true

I’ve added now verbose parameter.

I’am working with the integrated with SonarQube scanner within TFS. Thats way i cannot use the commandline parameter as you mentioned, but within TFS there is a possibility to enter additional parameters.

In the log I found that the parameter sonar.projectBaseDir still point to the folder where the solution resides and not to the folder where the source code can be found. In the following (sub)project this should be D:\Agent.2\110\s\Shared.ToObject. This means the shares sources are still not found of used by SonarQube.

2020-03-27T14:56:22.4509405Z 5264DF0F-9145-4D96-92F1-5F999B48447F.sonar.projectName=ToObject_R231
2020-03-27T14:56:22.4509605Z 5264DF0F-9145-4D96-92F1-5F999B48447F**.sonar.projectBaseDir**=D:\Agent.2\110\s\ToObject\R231
2020-03-27T14:56:22.4509818Z 5264DF0F-9145-4D96-92F1-5F999B48447F.sonar.sourceEncoding=windows-1252
2020-03-27T14:56:22.4510015Z 5264DF0F-9145-4D96-92F1-5F999B48447F.sonar.sources=\

If necessary I can upload the full logging.

Kind Regards,
Adrie

Pavel_Mikula · April 14, 2020, 10:58am

Hi Andrie,

I’ve tried to reproduced your scenario and this is what happened: First run of scanner end step starts like this, with warning about what’s going to happen.

SonarScanner for MSBuild 4.7.1
Using the .NET Framework version of the Scanner for MSBuild
Post-processing started.
WARNING: File 'C:\Projects\WithShared\Shared\SharedClass.cs' is not located under the root directory 'C:\Projects\WithShared\SmazatWithShared' and will not be analyzed.
Calling the SonarQube Scanner...

This file was not indexed as warned. Then I’ve changed the relative dir by adding projectBaseDir argument (it’s only one dir up in my case, you should keep …/…)

SonarScanner.MSBuild.exe begin /k:"WithShared" /d:sonar.projectBaseDir=..

And the warning disappeared. Shared file was indexed.

So far it looks that there’s something wrong with passing the argument to begin step. Do you have the /d: prefix there?

Adrie · April 14, 2020, 12:01pm

Hi Pavel,

In you’re example you are using the command-line version of the SonarScanner.

In my situation i am using the SonarQube plugin on TFS (Team Foundation server).

In this plugin there’s no need to enter parameters in the same syntax as the commandline interface. For the plugin you set parameters in the key=value format.

Like:
sonar.verbose=true
sonar.sourceEncoding=windows-1252
sonar.projectBaseDir=…/…

But still this has no effect. I tried it with a relative path and with a absolute pad, both without any success. Probably this doesn’t work with the SonarQube plugin on TFS.

But last Friday i found a work-around for this problem. In this work around i moved all the shared folders back to a map below one of the projects, when i am getting all the sources for the automatic build in TFS. With a search and replace action (a task in TFS), i replaced the location of the project files in the solution and other project files. It was a little work to setup a new build procedure under TFS, but now i have SonarQube results again.

So probably you’re solution works for command-line scanners, but it isn’t a solution in our environment with TFS.

For now i can continue.

Thanks for your reaction.

Kind regard,
Adrie

Pavel_Mikula · May 22, 2020, 1:40pm

Hi Adrie,

Sorry I missed that point. Scanner for TFS embeds Scanner for MSBuild under the hood so the behaviour should be the same.

Can you please try to:

share full logs,
clarify what exact version of scanner do you have (you’ve mentioned “latest” at the beginning),
find some other place in your SQ project/config files/environment that is already setting this variable and overriding it? Although explicit scanner parameter should have highest priority.

Pavel_Mikula · May 22, 2020, 3:20pm

I’ve installed TFS 2015 and SQ extension. There’s Advanced/Additional settings field for additional command line arguments. Can you try to use /d:sonar.projectBaseDir=..\.. there?

Adrie · June 10, 2020, 9:44am

Hi Pavel, finally i had some time to look to his problem.

I am using the following parameters in the “Additional Properties” part of the “Prepare the SonarQube analysis” task of TFS. The version of the task is 4.

sonar.verbose=true
sonar.sourceEncoding=windows-1252
sonar.projectBaseDir=…/…

You asked also for full logs, but can you be more specific which logs and of extra information you are looking for?

Pavel_Mikula · June 16, 2020, 8:31am

Hi Adrie,

There’s a sonar.verbose=true parameter that you can set. See this page for more details. And then just send the build log as before. Please review the log before posting to remove any confidential information.

You’ve mentioned

Finally: We are working with SonarQube 7.9 and the latest versions of all scanners and we are using the SonarQube plugin on TFS version 2015.

I’d like to verify what are the exact versions. Latest version of SonarScanner for Azure DevOps is 4.10, but that’s compatible with TFS 2017 Update 2+. So you should use sonar-scanner 3.0.3.778 that is shipped with sonarsource.sonarqube-3.1.0.vsix TFS plugin with TFS 2015.

Adrie · June 23, 2020, 3:38pm

Hi Pavel,

The version of our scanner
D:\Agent.2_tasks\SonarQubePrepare_15b84ca1-b62f-4a2a-a403-89b77a063157\4.10.0\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe is 4.9.0.17385 used for a Visual Studio 2017 build on TFS 2015. This scanner works fine for all other project we are using.

If also use used the following parameters (see my previous posts):

sonar.verbose=true
sonar.sourceEncoding=windows-1252
**sonar.projectBaseDir=…/…

If have added a zip with the logging. Maybe this information helps…

.sonarqube tbv TopoCAD build en forum SQ.zip (1.5 MB)

Pavel_Mikula · July 21, 2020, 1:33pm

Hi Adrie,

unfortunately, the ZIP file doesn’t contain the verbose log - similar to the one you’ve provided in your first post.

Also as mentioned in previous post, can you try

/d:sonar.projectBaseDir=…..
instead of
sonar.projectBaseDir=…..
?

I’ve tried to install sonar scanner 4.9 extension to VSTS 2015, but I can’t configure the SonarQube Endpoint property of the begin step. The option list is just empty even when I’ve created the endpoint. Do you have any tip how to workaround this? I’m not so good at VSTS 2015.

The setup worked with sonarscanner 3.1. Can you try to download and install v3.1? The vsix can be downloaded from here.

Adrie · July 23, 2020, 8:16am

Hi Pavel,

As we are using TFS as front-end, i have no possibility to set the parameter with the /d: option. See screenshot below.

The only possibility is to put one parameter (key=value) on a seperate line.

You suggestion about using another version of the Sonar-scanner s something i don-not know what to do with it. Because i think the TFS plugin is doing all the work. So i didn’t need to install a command-line scanner before i ran into this problem.

Kind regards,
Adrie

Pavel_Mikula · July 24, 2020, 11:52am

Ok, we’re getting lost here.

You have this

sonar.verbose=true
sonar.sourceEncoding=windows-1252
sonar.projectBaseDir=../..

Why cannot you change it to this?

/d:sonar.verbose=true
/d:sonar.sourceEncoding=windows-1252
/d:sonar.projectBaseDir=../..

Where /d:sonar.projectBaseDir is the key and ../.. is the value of key=value pair?

You don’t need to install command line tool. I’ve suggested to try to replace SonarQube Scanner for Azure DevOps v4.9 extension with older SonarQube Scanner for Azure DevOps v3.1 extension.

But that step might not be needed, because your screenshot looks way too different from my TFS 2015. This is what I see in my local instance:

Can you please try to describe your environment in more detail and verify what exact version are you using?

I have this in my TFS admin console:
Edition: Team Foundation Server 2015
Product Version: 14.114.28805.0 (Tfs2015.Update4.2)
Your screenshot looks like much more modern version.

Adrie · July 24, 2020, 1:37pm

Hi Pavel,

Is this the information you are looking for?

About Microsoft Visual Studio Team Foundation Server

Microsoft Visual Studio Team Foundation Server

and we are using the following Team Foundation Server Extensions van SonarQube

||Version|4.11.0|
|Last updated|‎25‎-‎6‎-‎2020‎ ‎17‎:‎27‎:‎05|
|Publisher|SonarSource||

or

2020-07-24T12:58:23.2936967Z Task : Prepare Analysis Configuration
2020-07-24T12:58:23.2937083Z Description : Prepare SonarQube analysis configuration
2020-07-24T12:58:23.2937176Z Version : 4.11.0
2020-07-24T12:58:23.2937272Z Author : sonarsource
2020-07-24T12:58:23.2937364Z Help : Version: 4.11.0. More Information

Finaly use /d: options

does not make any difference:

Still no issues found on the shared projects.

Pavel_Mikula · July 24, 2020, 2:25pm

Thanks for the update. Now it’s clear where the difference comes from:

Version 15.117.26714.0 is actually Team Foundation Server 2017 Update 2 RTM and not TFS 2015.

I’ll reinstall my test environment and take a look at it next week.

And there’s no reason to play with version 3.1 of the extension as well.

Pavel_Mikula · July 29, 2020, 3:39pm

Hi @Adrie,

Good news is, that I have working environment and I’m able to build it with SQ.
Bad news is, that it worked as expected without any explicit settings.

So questions are:

What is your ‘Project.Solution’ settings? I mean what do you build in your Build step. main\ToBasis\R231\ToBasis_R231.csproj ?
What is your VSTS setting that made the SQ scanner to run from the main\ToBasis\R231\ path?
Can you please keep the sonar.verbose=true parameter and share the full build logs?

Adrie · July 30, 2020, 8:43am

Hi Pavel,

Remember that we are using shared projects. For example

ToBasis_R231.csproj you can find the following line

In the file Shared.ToBasis.projitems all the cs files of this project collected.

I think this is the case that SonarQube skips these cs files.

I will send you the logs in a separate mail.

About youre next question:

What is your VSTS setting that made the SQ scanner to run from the main\ToBasis\R231\ path?

We do nothing special, only a few settings are made at the SonaQube task of the TFS build

Thats all, and of course this works fine for all other project. Only within this project (working with shared projectitems) the most cs files are not scanned. In fact all files collected in Shares…projtitems are skipped. And we have 26 different shared folders with projitems. You must be able to reproduce the same problem if you work in the same way.

Adrie · July 30, 2020, 9:11am

Here you can see that for all the shared projects there are no lines of code