We’ve been encountering an issue with one of our build pipelines in Azure DevOps.
The pipeline has 2 Agent Jobs, the first builds our web application and the second builds our Database Red Gate Change Automation 3 project. They jobs are configured to not use parallelism.
The first job executes successfully, this pipeline has SonarQube configured for analysis and by the end of this job, the analysis report is uploaded to our locally hosted SonarQube install.
The second job intermittently fails and we’re noticing when it does fail a SonarQube occurs in the log which appears to be causing the database project build to fail.
Here’s a screenshot (I’ve temporarily disabled the SonarQube steps to bypass the issue):
When the build fails:
We see the following in the error log for the second agent job:
##[debug] Message: Current MSBuildToolsVersion: 4.0
##[debug] Message: Current MSBuildAssemblyVersion: 15.0
##[debug] MSBuild: Using “Error” task from assembly “Microsoft.Build.Tasks.v4.0, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
##[warning] Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.
##[debug]Processed: ##vso[task.logissue type=warning] Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.
##[debug] : Target “_CleanGetCurrentAndPriorFileWrites” skipped. Previously built successfully.
##[debug]Done building project “Intelligence.sqlproj” – FAILED.
Should SonarQube be analyzing during the build phase of the 2nd agent job?
According to the logs it’s using SonarScanner for MSBuild 4.6.2.
That is weird. Presumably you are running self-hosted build agents, and the same agent is being used for both stages?
My guess is that the scanner hooks into the MSBuild process aren’t being removed correctly in the Run Code Analysis step.
Could you trying running the build with system.debug=true set for the pipeline?
When the build fails, does the log for the Run Code Analysis step in the first stage contain a warning like the following?
Could not delete SonarQube.Integration.ImportBefore.targets from
I’ve been running the builds with debug=true because I’ve been having SonarQube/Change Automation 3 issues for a little while now. However, the plot thickens. SonarQube seems to be doing running even when the 3 SonarQube build tasks are disabled in the first pipeline as I’ve now seen the exact same problem occur in our last 2 builds.
In the scenario where the SonarQube steps are disabled, there is no “Run Code Analysis” step to review the log for; but I still have this in the log for the 2nd stage:
##[warning]|Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.|
##[debug]Processed: ##vso[task.logissue type=warning]|Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.|
My gut is telling me it’s either a race condition somewhere in the build pipeline or other build agents on the same build server (we have multiple agents per self-posted build box) is somehow interfering, e.g. if any of the files used by SonarQube or Red Gate are stored in shared locations rather than the build agent’s working directory.
If I remember correctly, an Azure DevOp build agent will only run one build at a time, so unless you have multiple build agents installed on the same machine it shouldn’t be a race condition.
Yes, the scanner’s ImportBefore targets file that hook into the build are stored in a shared location so will interfere with other builds unless they are deleted correctly, which should happen in the end step. That’s why I suggested checking the Run Code Analysis step logs for a specific error message.
The verbose log output for the Prepare code analysis step lists the shared locations that the ImportBefore targets files are being copied to, so you could manually check the build agent to see if they are still present. If so, it’s safe to delete them.
Yes, we have multiple build agents on the same physical box (each agent has it’s own working directory). The issue is intermitent (suggesting the race condition). And, as mentioned, I’ve seen sonarqube errors fail in my build now, even when my build has all sonarqube build tasks disabled.
Is it possible that it’s using this shared location that’s causing the issue if 2 builds are running simultaneously - and can the agent specific working directory be used instead of the shared directory? Is it something I can control/configure?
The scanner uses two targets files to hook into the build:
SonarQube.Integration.ImportBefore.targets is copied to a shared location on the machine. This location is specified by MSBuild and is per-user.
SonarQube.Integration.targets is copied to a build-specific location (namely, the .sonarqube folder that is created under the agent build directory.
The target in the shared location will be picked up by all builds, regardless of the agent. However, that file doesn’t do much, other than check for and import the second targets file in the per-build location if it exists.
Given the following scenario:
build A with Sonar analysis enabled,
build B (no analysis)
both builds run simultaneously on the same box by different agents;
Both builds will pick up the first targets file, which will look for the second targets file.
Build A should find the second targets file, and run the analysis.
Build B should not find the second targets file, and do nothing.
However, if your case, Build B seems to be finding the second set of targets files which fail the build because of the version of MSBuild being used.
The only way I can think of that this might happen is if the second build definition previously had analysis enabled at some point, and the build failed before the second targets file was deleted. The targets will still be on the machine in the work folder for that build/agent job combination, so any time there is a second build with analysis enabled running at the same time Build B will fail.
That’s the theory. Some practical things you can try:
check what’s in the build-specific directory for your database build (i.e. the build agent …\work\xxx folder for that build definition). If there is a .sonarqube folder there, delete it.
If that folder exists then it would explain the problem, and deleting it should fix it.
If not, then I don’t what the issue is, but there is another workaround you could try to stop the second build from failing:
pass the argument SonarQubeTargetsImported=true to the database build you don’t want to be analysed. You could do this in several ways:
by manually editing the project file to set it as an MSBuild property i.e. <SonarQubeTargetsImported>true</SonarQubeTargetsImported>
by passing it as an argument to the build using /p:SonarQubeTargetsImported=true
finally, adding SonarQubeTargetsImported as a variable in the build definition might work to (since all non-secret build variables are passed as environment variables to all tasks in the pipeline, and MSBuild turns all environment variables into build properties).
The second workaround tells the first set of targets not to import the second set of targets, even if they exist.
Apologies, but I’ve been side-swiped by another project so we’ve just been living with this issue (kicking off the build again and again until it eventually works).
Before I try your recommendations, I’m just checking the build directory and have found this:
/agent/_work/67/.sonarqube/bin/targets/SonarQube.Integration.targets
It looks like it already has the property workaround you describe?
As mentioned previously, this is one “Build” with 2 agent phases within it; the first to compile the web project (for which I do want the SonarQube analysis) and the second to perform the database project build (which I’d like to be ignored by SonarQube entirely).
There appears to be a .targets file being created in the service account’s user directory:
C:\Users\{build service account}\AppData\Local\Microsoft\MSBuild\4.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.target
If I delete this file manually, the SonarQube analysis will not cause the the Change Automation 3 project to fail! However, it keeps cropping back up.
Is the SonarQube process somehow using the service account’s AppData directories (which definitely span build agents on the same build server) to store the .targets file?
Yes, the scanner begin step will copy the targets to several locations under AppData:
SonarScanner for MSBuild 4.7.1
Using the .NET Framework version of the Scanner for MSBuild
...
Pre-processing started.
Preparing working directories...
Using environment variables to determine the download directory...
Using environment variable 'AGENT_BUILDDIRECTORY', value 'd:\a\1'
13:14:15.933 13:14:15.933 Loading analysis properties from d:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.9.0\classic-sonar-scanner-msbuild\SonarQube.Analysis.xml
13:14:15.933 13:14:15.933 sonar.verbose=true was specified - setting the log verbosity to 'Debug'
13:14:16.167 Updating build integration targets...
13:14:16.183 Installed SonarQube.Integration.ImportBefore.targets to C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\4.0\Microsoft.Common.targets\ImportBefore
13:14:16.183 Installed SonarQube.Integration.ImportBefore.targets to C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\10.0\Microsoft.Common.targets\ImportBefore
13:14:16.183 Installed SonarQube.Integration.ImportBefore.targets to C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\11.0\Microsoft.Common.targets\ImportBefore
13:14:16.183 Installed SonarQube.Integration.ImportBefore.targets to C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\12.0\Microsoft.Common.targets\ImportBefore
13:14:16.183 Installed SonarQube.Integration.ImportBefore.targets to C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore
13:14:16.183 Installed SonarQube.Integration.ImportBefore.targets to C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\15.0\Microsoft.Common.targets\ImportBefore
13:14:16.183 Installed SonarQube.Integration.ImportBefore.targets to C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\Current\Microsoft.Common.targets\ImportBefore
13:14:16.183 Installed SonarQube.Integration.targets to d:\a\1\.sonarqube\bin\targets
The targets are removed in the end step:
[command]d:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.9.0\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe end
SonarScanner for MSBuild 4.7.1
Using the .NET Framework version of the Scanner for MSBuild
Post-processing started.
13:19:21.084 13:19:21.021 Uninstalling target: C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\4.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
13:19:21.084 13:19:21.021 Uninstalling target: C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\10.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
13:19:21.084 13:19:21.021 Uninstalling target: C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\11.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
13:19:21.084 13:19:21.021 Uninstalling target: C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\12.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
13:19:21.084 13:19:21.021 Uninstalling target: C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
13:19:21.084 13:19:21.021 Uninstalling target: C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\15.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
13:19:21.084 13:19:21.021 Uninstalling target: C:\Users\VssAdministrator\AppData\Local\Microsoft\MSBuild\Current\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
13:19:21.084 13:19:21.084 sonar.verbose=true was specified - setting the log verbosity to 'Debug'
The <SonarQubeTargetsImported>true</SonarQubeTargetsImported> setting is expected to be in the file .sonarqube/bin/targets/SonarQube.Integration.targets.
Try setting it in your database .csproj, or (better) passing it as an additional MSBuild command line argument to the database build step i.e /p:SonarQube.Integration.targets=true
Thanks for the reply! Really appreciate your help.
Is it possible that, if I have a second build agent on the same build server, that places this file in these shared locations, that my build running on the first build agent could pick it up and be affected by it’s presence?
Currently, all 4 build agents I have on my build server are using the same user credentials so will effectively be using the same C:\Users… directory.
As you can see from my build screenshot in the first post, there are no SonarQube activities wrapping around the database build. It’s also using a Redgate SQL Change Automation: Build task so I can’t pass any arguments to it; which probably outlines the issue since it’s not using a SonarQube compatible versoin of MSBuild - which leads to the error:
##[warning] Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.
Also, if I set the Database project to be the first build in the pipeline, it’s still affected; suggesting it’s not caused by the presence of the SonarQube tasks wrapping the build of my web project, if they tidy up after themselves.
The scanner begin step copies files to two locations:
SonarQube.Integration.ImportBefore.targets is copied to machine-wide, user-specific locations, and it will be picked up by all builds run on the machine for all agents.
SonarQube.Integration.targets and its related assemblies are copied to a pipeline-specific location ([build pipeline folder]\.sonarqube\bin\targets)
Both sets of files need to be present for the Sonar analysis tasks and targets to be picked up and run as part of an MSBuild run. The error message you are seeing is coming from the SonarQube.Integration.targets file, which means that both the shared-location and pipeline-specific files are present at the point your database project is being built.
On a machine with a single build agent installed the sequence of steps would be as follows:
(1) Agent 1 starts Analysis Build X -> installs the shared targets and targets under PipelineX\.sonarqube\...
(2) Agent 1 runs MSBuild -> Sonar targets are executed
(3) Agent 1 runs analysis end step -> the targets in the shared location are deleted
(4) Agent 1 runs the Redgate DB step -> the shared targets are not on the machine -> Sonar targets are not executed, so no error.
However, if the machine has multiple build agents installed then it’s possible for the following to happen:
(3a) Agent 2 starts Analysis Build Y -> installs the shared targets and targets under PipelineY\.sonarqube\...
Then at stage (4) for Agent 1 both shared targets and those under PipelineX\.sonarqube\... are present -> Sonar targets are picked up the the Redgate DB build -> error.
Try the following: add a script step in the Build Query Intelligence Database job immediately before the Build SQL Change Automation Project Intelligence step that deletes the pipeline-specific targets in [AGENT_BUILDDIRECTORY]\.sonarqube\bin\targets.
2019-10-17T09:23:42.4931398Z ##[warning]|Error: ERROR C:\Users\svc_devbuildfiles\AppData\Local\Microsoft\MSBuild\4.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets(62,5): The build is configured to run SonarQube analysis but the SonarQube analysis targets could not be located. Project: Intelligence.sqlproj
2019-10-17T09:23:42.4932860Z ##[debug]Processed: ##vso[task.logissue type=warning]|Error: ERROR C:\Users\{svc_account}\AppData\Local\Microsoft\MSBuild\4.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets(62,5): The build is configured to run SonarQube analysis but the SonarQube analysis targets could not be located. Project: Intelligence.sqlproj
Thanks again for your help, I feel we’re really close and I’m desperately trying to avoid needing to run the agents as different users as this will require some extensive re-setup!
I have a similar issue in which the sonarcloud integration tasks seem to “leak” into other jobs on the same agent. We are using private agents, and every once in a while a job that doesn’t have sonarcloud integration suddenly starts scanning with sonarcloud. So it seems that the sonarcloud integration sometimes isn’t cleaned up properly on an agent.
Is the only way around this to not integrate sonarcloud with msbuild and run it manually? We cannot end up in a situation where “sometimes” random builds fail because of unwanted sonarcloud integration.
Let me stress we are NOT running parallel jobs on the same agent machine, so it really seems to be a cleanup issue
