SonarCloud tasks in Azure DevOps build pipeline do not work with C# projects behind the corporate proxy

  • ALM used: Azure DevOps
  • CI system used: Azure DevOps
  • Languages of the repository: C#

Hello

We have been using free SonarQube on our local server for a while and we really like it. So we would like to “move further” and want to switch to paid (for now just for 100 000 LOC) SonarCloud.

  • We want SonarCloud because we prefer service instead of our own SonarQube server, which we need to care about.
  • We want paid version, because we want to use Pull Request scanning, not just master branch after PR was merged.

The setup of SonarQube was not hard and we managed to use it quite quickly. No we are in the middle of 14-days free trial period of SonarCloud and we are not able to manage it to work. :unamused:

We are testing it on small solution which contains several .NET Core projects. This solution has around 5000 LOC, but in SonarCloud it show that it has only 269 LOC. If I browse source code there, the only files with LOC counted are some SQL scripts and HTML resource file. So no C# files were scanned. I suspect, that it is because of the tasks do not know to use build agent’s proxy settings.

Most of our build run on our own build server, which is behind the corporate proxy. (But we use hosted DevOps agents as well.) SonarCloud DevOps tasks does not support proxy “the right way”, which means, they should ask the build agent for proxy settings and just use it. I found several questions here about proxy and there is also an issue filed a year ago. But nothing helped us.

Setting proxy using extraProperties as mentioned somewhere did not help.We managed to “persuade” the analyze task to use proxy by setting the environment variable SONAR_SCANNER_OPTS=-Dhttps.proxyHost=xxx -Dhttps.proxyPort=xxxx. But it only works when the analyzer task’s scanner mode is configured as CLI. When the scanner mode is MSBuild, it will not use proxy and so it will not reach https://sonarcloud.io. And I think (but not sure of course) that the CLI mode is responsible for not scanning C# files. On our local SonarQube, we use MSBuild scanner mode and everything works - though the SonarQube is on the same server as build agents, so it is not using proxy at all.

So is there some way how to fix this and make it to work? The best would be to implement using build agent’s proxy right into the tasks, because that way the Azure DevOps users can just use those task without any extra work. And just to note, even if it would work setting the proxy using extraProperties and have the scanner mode as MSBuild, it is not an option. Because even if we have our own build server, we also use hosted DevOps build machines and so we cannot hardcode proxy settings into the build pipeline. The same build runs one day on local server behind the proxy and the other day on hosted DevOps server without the proxy.

Thank you.

1 Like

Hi @satano and welcome to our community !

Thank you for you feedback.

Let me answer some of your questions :

We want paid version, because we want to use Pull Request scanning, not just master branch after PR was merged.

Pricing mode is slightly different between SonarQube and SonarCloud. On SonarQube, you are right, from paid developer edition, you get branch and PR analysis, whereas on SonarCloud, this feature is included from the free version.
SonarCloud paid version is for both LOCs analyzed and to get benefit of private organization and projects.

And I think (but not sure of course) that the CLI mode is responsible for not scanning C# files.

You are partly right. Scanner CLI can analyze C#, but you have to manually provide path to sources using sonar.sources property. We then fully recommand to use the Scanner for MSBuild for .NET projects, so you can benefit from the full support for that.

So is there some way how to fix this and make it to work?

I looked a bit and, given this documentation, you can let the scanner itself inherits from system proxy setting by using the following argument:

-Djava.net.useSystemProxies=true

Please note that the SONAR_SCANNER_OPTS should be an environment variable accessible by the task, otherwise it won’t be taken in account. You can for example set it as a pipeline variable, in the “Variables” tab in the “Edit” mode of the pipeline (if in classic mode, for YAML, you have to define a variable in the top of your pipeline so that it’s globally accessible :

variables:
  SONAR_SCANNER_OPTS: -Djava.net.useSystemProxies=true

To make sure that this is correctly taken, you’ll find this kind of log in the Run Code Analysis task :

Using the supplied value for SONAR_SCANNER_OPTS. Value: -Djava.net.useSystemProxies=true

Hope that it will help, please let us know.

Thank you.

Mickaël

1 Like

Hello @mickaelcaro

Thank you for quick response (and sorry for my late).

Your suggested settings does not work. Setting

SONAR_SCANNER_OPTS=-Djava.net.useSystemProxies=true

does not work and I am not the only one. What works is setting explicit proxy values into that environment variable:

SONAR_SCANNER_OPTS=-Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=yyyy -Dhttp.nonProxyHosts="localhost|..."

But this works only for CLI scanner, not for MSBuild.

The thing is, that when we use CLI scanner, it uses proxy settings set by SONAR_SCANNER_OPTS, environment variable, but it does not scan/report C# (.cs) files, when we try to use MSBuild scanner, it is not working at all, because it does not use proxy and cannot connect to https://sonarcloud.io.

So here is our setup

We have set proxy in SONAR_SCANNER_OPTS environment variable explicitly as mentioned above. Just note, we also have set environment variables HTTP_PROXY and HTTPS_PROXY.

When I try CLI scanner, our step in pipeline is:

- task: SonarCloudPrepare@1
  displayName: 'SonarCloud Prepare'
  inputs:
    SonarCloud: 'SonarCloud'
    organization: 'krossk'
    cliProjectKey: 'esw.invoicing'
    scannerMode: 'CLI'
    configMode: 'manual'
    cliSources: '$(Build.SourcesDirectory)'

When I try MSBuild scanner, pipeline step is:

- task: SonarCloudPrepare@1
  displayName: 'SonarCloud Prepare'
  inputs:
    SonarCloud: 'SonarCloud'
    organization: 'krossk'
    projectKey: 'esw.invoicing'
    scannerMode: 'MSBuild'

Our project is a .NET C# solution with several projects. So most of the files are C# (.cs), but there are some SQL scripts (.sql) and HTML (.html) files as well.

For testing purposes, I connected the build machine to internet directly, without proxy. MSBuild scanner worked as expected. It found all the issues in all the files - we have some issues in all three file types. So I returned the machine back behind the proxy.

MSBuild scanner behind proxy

This is not working working at all. It does not matter if SONAR_SCANNER_OPTS is set to use system proxy, set to explicit proxy, or is not set. This always fails during SonarCloud Prepare step. The log is:

SYSTEMVSSCONNECTION exists true
c:\BuildAgents\agent-2\_work\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.9.0\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe begin /k:esw.invoicing /o:krossk
SonarScanner for MSBuild 4.7.1
Using the .NET Framework version of the Scanner for MSBuild
Pre-processing started.
Preparing working directories...
14:51:45.696  Updating build integration targets...
14:51:45.724  Fetching analysis configuration settings...
##[error]14:52:48.799  Failed to request and parse 'https://sonarcloud.io/api/server/version': Unable to connect to the remote server
14:52:48.799  Failed to request and parse 'https://sonarcloud.io/api/server/version': Unable to connect to the remote server
##[error]14:52:48.799  Could not connect to the SonarQube server. Check that the URL is correct and that the server is available. URL: https://sonarcloud.io/
14:52:48.799  Could not connect to the SonarQube server. Check that the URL is correct and that the server is available. URL: https://sonarcloud.io/
##[error]14:52:48.8  Pre-processing failed. Exit code: 1
14:52:48.8  Pre-processing failed. Exit code: 1
##[error]The process 'c:\BuildAgents\agent-2\_work\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.9.0\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe' failed with exit code 1
Finishing: SonarCloud Prepare

CLI scanner behind proxy

SonarCloud Prepare step is working in both cases: if SONAR_SCANNER_OPTS is set to use system proxy and also if it is set to explicit proxy. The log in this step is very simple:

SYSTEMVSSCONNECTION exists true
Finishing: SonarCloud Prepare

The result of SonarCloud Analyze step depends on SONAR_SCANNER_OPTS variable. If it is set to use system proxy, this step fails:

C:\WINDOWS\system32\cmd.exe /D /S /C "c:\BuildAgents\agent-2\_work\_tasks\SonarCloudAnalyze_ce096e50-6155-4de8-8800-4221aaeed4a1\1.9.0\sonar-scanner\bin\sonar-scanner.bat"
INFO: Scanner configuration file: c:\BuildAgents\agent-2\_work\_tasks\SonarCloudAnalyze_ce096e50-6155-4de8-8800-4221aaeed4a1\1.9.0\sonar-scanner\bin\..\conf\sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 4.1.0.1829
INFO: Java 12.0.2 Oracle Corporation (64-bit)
INFO: Windows 10 10.0 amd64
INFO: SONAR_SCANNER_OPTS=-Djava.net.useSystemProxies=true
INFO: User cache: C:\WINDOWS\ServiceProfiles\NetworkService\.sonar\cache
##[error]ERROR: SonarQube server [https://sonarcloud.io] can not be reached
ERROR: SonarQube server [https://sonarcloud.io] can not be reached
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 15.426s
INFO: Final Memory: 3M/56M
##[error]ERROR: Error during SonarQube Scanner execution
ERROR: Error during SonarQube Scanner execution
INFO: ------------------------------------------------------------------------
##[error]org.sonarsource.scanner.api.internal.ScannerException: Unable to execute SonarQube

... here is very long stack trace log ...

ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.
##[error]The process 'c:\BuildAgents\agent-2\_work\_tasks\SonarCloudAnalyze_ce096e50-6155-4de8-8800-4221aaeed4a1\1.9.0\sonar-scanner\bin\sonar-scanner.bat' failed with exit code 1
Finishing: SonarCloud Analyze

If the SONAR_SCANNER_OPTS is set to explicit proxy, SonarCloud Analyze does not fail. But I would not say it is working, because it does not find issues in C# files. It just reports issues in SQL and HTML files and does not report any C# issue. Even when I browsed source code in project in the sonarcloud.io, there were only .sql and .html files, no .cs files. I double checked this with adding two new files with issues in them. One is .html and the other one is .cs. After the build, we see new issues in new .html file, but no C# issues.

Just last note is, I do not change settings in project in sonarcloud.io. I am just playing with those two scanner modes nad proxy settings in our pipeline. And as I mentioned at the beginning, when the machine is not behind proxy, MSBuild scanner really works.

So any other ideas what is wrong and if there is some solution for us?

Thanks a lot, Stano

1 Like

Hi @satano,

If you run the analysis with the Scanner for MSBuild and in debug mode (either with the global system.debug=true or sonar.verbose), do you see this kind of message in the logs ? (should be in the Run Code Analysis task)

“Using the supplied value for SONAR_SCANNER_OPTS”

Thank you.

Mickaël

Hello @mickaelcaro

The message you mention is not in the logs anywhere, regardless which scanner mode is used.

MSBuild scanner

The SONAR_SCANNER_OPTS is not in the log file of prepare task. The log files for analyze and publish tasks are empty, because the prepare task fails and so the build does not continue.

CLI scanner

The SONAR_SCANNER_OPTS is mentioned only in the log of analyze task (so not in prepare, nor publish). The log message is:

INFO: SONAR_SCANNER_OPTS=-Dhttps.proxyHost=192.168.1.3 -Dhttps.proxyPort=3128 -Dhttp.nonProxyHosts="localhost|127.0.0.1|mlearning"

Full logs from SonarCloud steps:

To be sure, I started my test/investigation from the very beginning. I deleted the project at sonarcloud.io and created a new one. Then I run my build pipeline on master branch, using MSBuild scanner first, then the CLI scanner.

The MSBuild scanner still fails during prepare task.

The CLI scanner does not fail and shows the result at sonarcloud.io. But there are only two issues, 1 in html file and 1 in sql file. The projects has only 346 line of codes, because only html and sql files are counted. C-sharp (cs) files are not scanned and reported.

Stano

Hi @satano

Ok i saw what it is about !

We have a open issue on GitHub for this (pretty old i concur) here https://github.com/SonarSource/sonar-scanner-msbuild/issues/654

We might have a look soon on that.

In the mean time, can you try setting this kind of environment variable (be careful to make them accessible in your task, by restarting the agent if you are running it as self-hosted)

  • HTTP_PROXY: the hostname or IP address of the proxy server used on HTTP requests.
  • HTTPS_PROXY: the hostname or IP address of the proxy server used on HTTPS requests.
  • ALL_PROXY: the hostname or IP address of the proxy server used on HTTP and/or HTTPS requests in case HTTP_PROXY and/or HTTPS_PROXY are not defined.

Thank you.

Mickaël

Hello @mickaelcaro

Those environment variables did not help. The results are still the same.

Just note, that our proxy is not authenticated, so the issue you have mentioned is probably not the real one we are facing. I think our issue may be this one: Use agent proxy configuration when doing node request and during analysis. Although it is not filed on GitHub, so maybe nobody knows about it anymore.

Thanks, Stano

I confirm these are not node related requests, as they are located in the Scanner for MSBuild. We had a long time ago opened a discussion on that particular topic, i’ll try to see if we can do anything more with it.

Mickaël

We are experiencing the exact same issues here. Our proxy does require authentication, but Azure devOps agent is taking care of that (normally). So on-prem builds can access the GIT repos on Azure DevOps, and build, but fail (##[error]ERROR: SonarQube server [https://sonarcloud.io] can not be reached) at the Run Code Analysis task.

So I am getting the feeling this task is not using the correct proxy settings, or tries to setup its own internet connection.

Hello @mickaelcaro and the others

Since this is a real problem for us, I spent couple of days trying to find some workaround at least and this is what I found.

The SonarCloudPrepare task in our build machine already has two versions: 1.7.0 and 1.9.0. So it updates automatically. Looking into detailed logs, ´1.9.0´ is used in pipeline (and it is not working). I tried to run it manually, not by running the task itself, but running

dotnet <path to SonarScanner.MSBuild.dll> begin ...
... build step is in between ...
dotnet <path to SonarScanner.MSBuild.dll> end

as documented here. Is obviously still did not work.

So I downloaded Sonar Scanner for MSBuild, and built it myself. Copied the netcoreapp3.0 build to the build machine. I had also to:

  • Copy there SonarScanner.MSBuild.Tasks.dll which is build in the one of the projects in that repository.
  • Download latest sonar-scanner, so I used version 4.2.0.1873 and also copied it to the dotnet-sonar-scanner-msbuild folder.

This works. :tada:

But is is just a proof of concept that the latest version works. Because using it this way, it lacks most of the functionality. Right now, we are analyzing just master branch, when it is updated.

  • I can not run Pull Request analysis.
  • I can not run SonarCloudPublish task, so the result is not visible directly in the build.

Although probably I would manage to make these work. It seems they need correct settings (some environment variables) before the run. But it would be a lot of additional work to investigating what/how to set up.

So what I would like, is you to publish the latest version of your SonarCloud tasks for Azure Pipelines as soon as possible. I hope they will work.

And please let us know here when you will publish them. Because for now, we will stick to this manual master branch analysis. So we want to know when the new tasks are available, to try them.

Thanks a lot, Stano

1 Like

For us it is an issue as well, as teams need to use the on premise pipeline to run their tests on target machines, and they want to use the same build to run the code analysis, to get 1 complete picture.
We have a proxy with authentication switched on. If I use the SONAR_SCANNER_OPTS with the option mentioned above, the prepare task goes well, but the Run code Analysis task fails, saying it cannot reach sonarcloud (it is mentioning the use of the SONAR_SCANNER_OPTS). Also tried supplying the proxy user/pwd, but no success either, Run Code Analysis step still fails to get to sonarcloud.

So makes me wonder if the Run Code Analysis step is always following the proxy network connection (?)

Hi all

I have created an DevOps build task UseNetCoreSonarCloud@1, which simply replaces SonarCloudPrepare task’s DLLs with .NET Core ones. The taks is the automatic solution of what I described in another thread. It works only for MSBuild scanner of SonarCloudPrepare.

Hello,

Is there planned fix for this issue? It is also pain for us, since we also have whole environment behind proxy and I cannot make sonarcloud work. Or the only way how to make this work is the workaround made by satano?

Hi,

we also have problems with the Msbuild scanner if a proxy with authentication is required. Seams the C# part of the scanner is not reading SONAR_SCANNER_OPTS?

Regards.
Günter

Hi @guwirth

It is not reading that variable. Plan are to move to another Http Client, so that it should read HTTP_PROXY kind of variable, but i don’t have any ETA to give you for the moment.