Running azure-devops extension against docker build instead of msbuild

Hi,

using azuredevops to build / release a dotnet core project.

THe project was updated to 3.1 which has caused the builds to break as the sdk on the build servers (ubuntu or windows) only goes to 3.0

Is it possibe to run the scans against a docker image? I do our builds via a multistage dockerfile.

S

Hi,

Yes you can ! Simply manage to have a dotnet docker image, then you can install the Scanner for MSBuild as a dotnet global tool, and you should be good to go ! Don’t forget to copy all your files inside your docker image, including .git to get SCM information.

Mickaël

1 Like

That’s sounds awesome… is there a guide I can follow? As I’m rather green on this stuff…

Thanks
S

For the whole experience, well, not yet unfortunately.

I can give you some pieces of documentation however, which hopefully can help you

Dotnet core sdk docker image : https://hub.docker.com/_/microsoft-dotnet-core-sdk
From that image, you can create a dockerfile file, which will help you build the image :https://docs.microsoft.com/en-us/dotnet/core/docker/build-container#create-the-dockerfile
And then add to that dockerfile the installation of the Scanner for MSBuild as a dotnet global tool : https://sonarcloud.io/documentation/analysis/scan/sonarscanner-for-msbuild/

As i said, don’t forget to copy your whole repo inside, with the .git folder; so SonarCloud will be able to get SCM information for git blame essentially.

HTH
Mickaël

1 Like

Hi Mickael,

Trying to add it to the dockerfile but getting an error when running the scanner.

DOCKERFILE:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build

WORKDIR /app

copy csproj and restore as distinct layers

COPY *.sln .

COPY *.csproj ./

RUN dotnet restore

copy everything else and build app

COPY ./ ./

WORKDIR /app

RUN dotnet publish -c Release -o out

sonarscanner test

RUN dotnet tool install --global dotnet-sonarscanner

RUN dotnet dotnet-sonarscanner begin /k:"staggerlee011-github " /o:“Staggerlee011” /d:sonar.login=“xxxxx”

RUN dotnet build

RUN dotnet sonarscanner end /d:sonar.login=“xxxx”

I get an error:

l
You can invoke the tool using the following command: dotnet-sonarscanner
Tool ‘dotnet-sonarscanner’ (version ‘4.8.0’) was successfully installed.
Removing intermediate container 6ec6f41c0a91
—> 69baa04c083e
Step 10/17 : RUN dotnet dotnet-sonarscanner begin /k:"staggerlee011-github " /o:“Staggerlee011” /d:sonar.login=“xxx”
—> Running in e7a4631afc05
Could not execute because the specified command or file was not found.
Possible reasons for this include:

  • You misspelled a built-in dotnet command.
  • You intended to execute a .NET Core program, but dotnet-dotnet-sonarscanner does not exist.
  • You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.

IVe tried changing the line:

RUN dotnet dotnet-sonarscanner begin

RUN dotnet sonarscanner begin

RUN dotnet-sonarscanner begin

Always getting the same error.

Can you offer any help?

Hi,

This is due to the global installation of dotnet that installs it in a path and add this path to the PATH env variable, so then you need to restart the shell so the command will be recognized, and for some reason, it doesn’t work on Docker.

One workaround : Specify an install path while installing, with the --tool-path flag, then invoke the command from that path :

RUN dotnet tool install --tool-path msbuildscanner dotnet-sonarscanner
RUN msbuildscanner\dotnet-sonarscanner(.exe) begin /k:"staggerlee011-github" /o:"Staggerlee011" /d:sonar.login="xxxxx"

Let me know if it’s working on your side.

Thanks !

hi,

getting the same error running the container in --interactive i cant see a folder called msbuildscanner being created in the app directory, where should it be installed?

It should be created relatively to the folder you are currently in : This is defined by, for instance, the command

WORKDIR /app

So the msbuildscanner will be in /app/

hi, output of app and its not listed :confused: but i do the get the message to say its installed…

l
root@7e7b88623733:/app# ls -l
total 216
drwxr-xr-x 2 root root 4096 Dec 9 14:37 Controllers
-rwxr-xr-x 1 root root 843 Dec 6 22:05 Gulpfile.js
drwxr-xr-x 2 root root 4096 Dec 6 21:43 Models
-rwxr-xr-x 1 root root 712 Dec 6 21:41 Program.cs
drwxr-xr-x 2 root root 4096 Dec 6 21:41 Properties
-rwxr-xr-x 1 root root 507 Dec 6 21:41 SecurityHeadersAttribute.cs
-rwxr-xr-x 1 root root 1118 Dec 6 21:41 SecurityHeadersMiddleware.cs
-rwxr-xr-x 1 root root 2082 Dec 9 14:37 Startup.cs
drwxr-xr-x 4 root root 4096 Dec 6 21:41 Views
-rwxr-xr-x 1 root root 234 Dec 6 21:43 appsettings.Development.json
-rwxr-xr-x 1 root root 271 Dec 6 21:41 appsettings.json
drwxr-xr-x 1 root root 4096 Dec 27 09:35 bin
-rwxr-xr-x 1 root root 949 Dec 27 08:43 dockerfile
-rwxr-xr-x 1 root root 347 Dec 27 08:59 dockerfile.build
drwxr-xr-x 276 root root 12288 Dec 7 20:08 node_modules
drwxr-xr-x 1 root root 4096 Dec 27 09:35 obj
drwxr-xr-x 3 root root 4096 Dec 27 09:35 out
-rwxr-xr-x 1 root root 113290 Dec 7 20:08 package-lock.json
-rwxr-xr-x 1 root root 343 Dec 6 22:05 package.json
-rwxr-xr-x 1 root root 773 Dec 9 14:43 site.csproj
-rwxr-xr-x 1 root root 1112 Dec 6 21:41 site.sln
-rwxr-xr-x 1 root root 8564 Dec 9 14:37 wss-unified-agent.config
drwxr-xr-x 5 root root 4096 Dec 6 22:05 wwwroot

And if you do a recursive search (even on /), do you find it somewhere else ?

Just tested on my side, the folder is created where in my $pwd , so it should be somewhere !

Hi soo…

root@7e7b88623733:/# find . -name “sonar
./root/.dotnet/tools/dotnet-sonarscanner
./root/.dotnet/tools/.store/dotnet-sonarscanner
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner.4.8.0.nupkg
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner.4.8.0.nupkg.sha512
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner.nuspec
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp3.0/any/sonar-scanner-4.2.0.1873
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp3.0/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner-debug
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp3.0/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp3.0/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner.bat
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp3.0/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner-debug.bat
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp3.0/any/sonar-scanner-4.2.0.1873/conf/sonar-scanner.properties
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp3.0/any/sonar-scanner-4.2.0.1873/lib/sonar-scanner-cli-4.2.0.1873.jar
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp2.1/any/sonar-scanner-4.2.0.1873
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp2.1/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner-debug
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp2.1/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp2.1/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner.bat
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp2.1/any/sonar-scanner-4.2.0.1873/bin/sonar-scanner-debug.bat
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp2.1/any/sonar-scanner-4.2.0.1873/conf/sonar-scanner.properties
./root/.dotnet/tools/.store/dotnet-sonarscanner/4.8.0/dotnet-sonarscanner/4.8.0/tools/netcoreapp2.1/any/sonar-scanner-4.2.0.1873/lib/sonar-scanner-cli-4.2.0.1873.jar
./root/.local/share/NuGet/v3-cache/1ca707a4d90792ce8e42453d4e350886a0fdaa4d$ps:_api.nuget.org_v3_index.json/nupkg_dotnet-sonarscanner.4.8.0.dat
./root/.local/share/NuGet/v3-cache/1ca707a4d90792ce8e42453d4e350886a0fdaa4d$ps:_api.nuget.org_v3_index.json/list_dotnet-sonarscanner.dat

So if i then try and run the scanner using the top path.

root@7e7b88623733:/# /root/.dotnet/tools/dotnet-sonarscanner
SonarScanner for MSBuild 4.8
Using the .NET Core version of the Scanner for MSBuild
WARNING: Please specify the command ‘begin’ or ‘end’ to indicate whether pre- or post-processing is required. These parameters will become mandatory in a later release.
Post-processing started.
19:36:34.338 Temporary analysis directory (usually .sonarqube) doesn’t exist. The “begin” step was probably not executed.
19:36:34.338 Post-processing failed. Exit code: 1

Which is starting to look good! i then tried putting in the info from the example code using a newly generated token from my profile:

root@7e7b88623733:/# ./root/.dotnet/tools/dotnet-sonarscanner /k:"staggerlee011-github " /o:“Staggerlee011” /d:sonar.login=“xxx”
SonarScanner for MSBuild 4.8
Using the .NET Core version of the Scanner for MSBuild
WARNING: Please specify the command ‘begin’ or ‘end’ to indicate whether pre- or post-processing is required. These parameters will become mandatory in a later release.
Pre-processing started.
Preparing working directories…
19:42:17.859 19:42:17.859 Invalid project key. Allowed characters are alphanumeric, ‘-’, ‘_’, ‘.’ and ‘:’, with at least one non-digit.
19:42:17.859 Expecting at least the following command line argument:

  • SonarQube project key
    When connecting to a SonarQube server earlier than version 6.1, the following command line arguments are also required:
  • SonarQube project name
  • SonarQube project version
    The full path to a settings file can also be supplied. If it is not supplied, the exe will attempt to locate a default settings file in the same directory as the SonarQube Scanner fo
    r MSBuild.
    Use ‘/?’ or ‘/h’ to see the help message.
    19:42:17.859 Pre-processing failed. Exit code: 1

Can you confrim how i get the right info for the /k /p and /d

and is it because im using a diff image to you? slightly worried that im having to hardcode a path to the executable which is diff to the one in the documentation and to yours…

Hi,

Thanks !

So the Scanner for MSBuild expects at least some mandatory flags to work properly :
1- “begin” or “end” : This defines the step to execute, begin should be executed before any msbuild command, so that the targets that will be written by this step will be well taken in account when building your project / solution (Please note that for the end step, you will only need the /d:sonar.login to be passed, not all described below)
2- -k: or /k: : the project key, should reflect what is written in either the id of the dashboard in the url (i.e https://sonarcloud.io/dashboard?id=myprojectkey) or in the right pane of the home of a project, under Project Key
3 - -o or /o: :the organization key
4: The /d:sonar.host.url flag : as it is used manually, it should be provided with https://sonarcloud.io value
5: the /d:sonar.login , which is the token that you generate on your SonarCloud’s account.

In you case, be careful, your project key has a trailing space (at least in the log you showed)

For the path, i’ll have a look deeper for that, the path that i provided was a local one, because i didn’t find a suitable solution to call directly the dotnet-sonarscanner command in a docker env, the tool installation log said that it’s added to the path, but it seems that the shell is not well refreshed between 2 RUN command, so it’s not recognized.

Mickaël

Hey,

So still getting issues (gettng ever closer!):

dockerfile:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build

WORKDIR /app

copy csproj and restore as distinct layers

COPY *.sln .

COPY *.csproj ./

RUN dotnet restore

sonarscanner test

COPY ./ ./

RUN dotnet tool install --global dotnet-sonarscanner

RUN /root/.dotnet/tools/dotnet-sonarscanner begin /k:“staggerlee011-github” /o:“Staggerlee011” /d:sonar.login=“xxx” /d:sonar.host.url=“https://sonarcloud.io

RUN dotnet build

RUN /root/.dotnet/tools/dotnet-sonarscanner end /d:sonar.login=“xxx”

Output:

14:33:30.884 Fetching analysis configuration settings…
14:33:31.817 Failed to request and parse ‘https://sonarcloud.io/api/qualityprofiles/search?projectKey=staggerlee011-github&organization=Staggerlee011’: The remote server returned an error: (404) .
14:33:31.818 Could not find a file on the SonarQube server. URL: https://sonarcloud.io/api/qualityprofiles/search?defaults=true&organization=Staggerlee011
14:33:31.819 Pre-processing failed. Exit code: 1

Screenshot of my account on sonarcloud.io

oh! it seems my org is called staggerlee011-github as well… pouting it in both gets it to run! and error on java instead :slight_smile: nearly there!

1 Like

can you confirm i need to install the full openjdk? build time has gone to seconds to nearly 10 mins…

Im currently running it in --it and get it to work. but just hoping im doing a massive overkill with it…

I think so yes.

We haven’t done that much benchmark on Docker, but i suggest that you have a look at the dedicated resource on your image itself, by doing a docker stats , you should be able to see if it’s stuck at 100% either in CPU / RAM, which can lead to such behavior.

Mickaël

no its not stuck

its the time it takes to install the openjdk constat wall of text flowing. just takes a while. I take it, its the openjdk or nothing then?

The web page doesnt say about needing node either. but when i ran it I get a bunch of warnings of it being missing and lacking results because it…

Ok, my bad, i thought this was the analysis that took amount of time.

I think that for getting better result, you should build a docker image with all the stuff you want, but don’t run the analysis inside the docker file, just run it everytime you need it, with a docker run command (by indeed replacing the RUN dotnet-sonarscanner by a EXEC or a CMD)

By doing that, the openjdk and node installation will be done only once, and you should be “good to go”

Hey,

Sorry not sure what you mean, do you mean install them but only call them when i needed (No idea how you do that! if it is possible please let me know how!?) i dont think that would work as idea in this case as this is just the first stage of a multi stage docker file

S

The idea is to build a base image upon dotnet sdk, with all the stuff you want installed on, then use this image as the base of the other.

For example, here’s the dockerfile of the prime base image (note that there’s no copy of your code base or something else, just installing the tools you’ll need after)

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build

#Install open jdk 11
RUN apt-get update && \
	apt-get install apt-utils -y && \
    apt-get install -y openjdk-11-jdk && \
    apt-get install -y ant && \
    apt-get clean;
	
RUN apt-get update && \
    apt-get install ca-certificates-java && \
    apt-get clean && \
    update-ca-certificates -f;

ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/
RUN export JAVA_HOME


#Install node.js 12
RUN apt-get update -yq \
    && apt-get install curl gnupg -yq \
    && curl -sL https://deb.nodesource.com/setup_12.x | bash \
    && apt-get install nodejs -yq

#check node installed
RUN node --version

#check java installed
RUN java --version

RUN dotnet tool install --global dotnet-sonarscanner

ENV PATH=${PATH}:/root/.dotnet/tools

With this, execute a docker build, tag your image propertly, and push it into a registry (local or remote).

docker build -t dotnetsonarbaseimage:tag .

Then this base will not change unless you have to rebuild something inside it (changing versions of one of the components for instance).

You can then use this image for your following docker file with :

FROM dotnetsonarbaseimage:tag AS dotnetscannerbase

WORKDIR /app

COPY *.sln /

COPY ./ ./

RUN dotnet restore

RUN dotnet-sonarscanner begin {args}

RUN dotnet build

RUN dotnet test

RUN dotnet-sonarscanner end {args}

Mickaël

1 Like