Hi all
Some context:
I am trying to solve an interesting problem but at the moment I am struggling. We have an on premises SonarQube server and a set of agents that can communicate with both Azure DevOps and the SonarQube server. The SonarQube server itself sits behind a company firewall that prevents the Microsoft Azure Pipelines agents from communicating with the server.
As more teams have been adopting SonarQube, the load on the agent pool built up dramatically. Many teams have multiple jobs running per build pipeline and they all run directly on the agent pool. The agent pool technically only consists of 3 VMs. So to handle the load, I installed 3 instances of the Azure Pipelines Agent service on each VM, as you can see in the screenshot below.
The problem:
The problem we are having is that when there are multiple builds running on a single VM, all using the MSBuild Sonar Scanner, we get unexpected and inconsistent build errors. Initially my assumption was that one of the security tools running on our servers was the problem but after further investigation, it seems to be related to the fact that the SonarQubePrepare@4 task appears to install a global dotnet tool, or something to that affect. It seems that when there are multiple builds that one can influence the state of the Sonar Scanner for the other build. We end up having a variety of errors, but the most common is the following:
##[error]15:30:11.322 The SonarScanner for MSBuild integration failed: SonarQube was unable to collect the required information about your projects.
Possible causes:
1. The project has not been built - the project must be built in between the begin and end steps
2. An unsupported version of MSBuild has been used to build the project. Currently MSBuild 14.0.25420.1 and higher are supported.
3. The begin, build and end steps have not all been launched from the same folder
4. None of the analyzed projects have a valid ProjectGuid and you have not used a solution (.sln)
I searched and searched for solutions to this error until I eventually found this post this morning: SonarQube impacting seperate agent job?
This post seems to deal with the issue but doesn’t resolve it for me, and here’s why:
In my case, I am managing 3 VMs, 9 agents that are available to all teams within my organization and they can and do use them as they need. It is not completely feasible for all teams to add config to their .csproj or .sqlproj files to tell the scanner whether or not to run. What we need for our use case is that the SonarQubePrepare@4, SonarQubeAnalyze@4 and SonarQubePublish@4 tasks are totally isolated and atomic to their $(Pipeline.Workspace).
Example of the scenario that’s causing the race condition:
Team A has a build running on Agent1_1 which has passed the SonarQubePrepare@4 and is currently running a dotnet command, say, dotnet build. During this time, Team B triggers a build on Agent1_2. Now there are two builds running on the Agent1 VM. Their first task is to run SonarQubePrepare@4. This now updates the current running directory of the dotnet SDK. A little time passes and Team A’s build on Agent1_1 is finishing and starts running the SonarQubeAnalyze@4 task. Their build fails with the following error message detailed above. Not long after, Team B’s build on Agent1_2 passes.
Configuration:
3 VMs - Agent1, Agent2 and Agent3 with 3 instances of the Azure Pipelines service running under the following directories:
C:\agent\1\...
C:\agent\2\...
C:\agent\3\...
These VMs are running Windows Server 2019 Standard with 6 cores, 12 threads and 12GB of RAM and a 10GB network to each.
The desired state:
3 VMs with 3 services, all able to run the SonarQube tasks without influencing each other during the process without having to update the configuration of every single team’s project that is using the SonarQube scan in their pipelines.
Is this currently possible? My research tells me no - hence 2/3 agents have been disabled in the screenshot above - as does the article I link above.
Thanks & looking forward to hearing back!
