SonarQube impacting seperate agent job?

Hi there!

We’ve been encountering an issue with one of our build pipelines in Azure DevOps.

The pipeline has 2 Agent Jobs, the first builds our web application and the second builds our Database Red Gate Change Automation 3 project. They jobs are configured to not use parallelism.

The first job executes successfully, this pipeline has SonarQube configured for analysis and by the end of this job, the analysis report is uploaded to our locally hosted SonarQube install.

The second job intermittently fails and we’re noticing when it does fail a SonarQube occurs in the log which appears to be causing the database project build to fail.

Here’s a screenshot (I’ve temporarily disabled the SonarQube steps to bypass the issue):

When the build fails:
image
We see the following in the error log for the second agent job:

##[debug] Message: Current MSBuildToolsVersion: 4.0
##[debug] Message: Current MSBuildAssemblyVersion: 15.0
##[debug] MSBuild: Using “Error” task from assembly “Microsoft.Build.Tasks.v4.0, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
##[warning] Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.
##[debug]Processed: ##vso[task.logissue type=warning] Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.
##[debug] : Target “_CleanGetCurrentAndPriorFileWrites” skipped. Previously built successfully.
##[debug]Done building project “Intelligence.sqlproj” – FAILED.

Should SonarQube be analyzing during the build phase of the 2nd agent job?

According to the logs it’s using SonarScanner for MSBuild 4.6.2.

Many thanks,

Jody

Hi @Jody_Lester,

That is weird. Presumably you are running self-hosted build agents, and the same agent is being used for both stages?

My guess is that the scanner hooks into the MSBuild process aren’t being removed correctly in the Run Code Analysis step.

Could you trying running the build with system.debug=true set for the pipeline?
When the build fails, does the log for the Run Code Analysis step in the first stage contain a warning like the following?

Could not delete SonarQube.Integration.ImportBefore.targets from

Thanks.
Duncan

Hi Duncan, thanks for the reply.

I’ve been running the builds with debug=true because I’ve been having SonarQube/Change Automation 3 issues for a little while now. However, the plot thickens. SonarQube seems to be doing running even when the 3 SonarQube build tasks are disabled in the first pipeline as I’ve now seen the exact same problem occur in our last 2 builds.

In the scenario where the SonarQube steps are disabled, there is no “Run Code Analysis” step to review the log for; but I still have this in the log for the 2nd stage:

##[warning]|Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.|

##[debug]Processed: ##vso[task.logissue type=warning]|Error: ERROR C:\agent_work\67.sonarqube\bin\targets\SonarQube.Integration.targets(148,5): SonarQube analysis is only supported with MSBuild 14 or later.|

My gut is telling me it’s either a race condition somewhere in the build pipeline or other build agents on the same build server (we have multiple agents per self-posted build box) is somehow interfering, e.g. if any of the files used by SonarQube or Red Gate are stored in shared locations rather than the build agent’s working directory.

If I remember correctly, an Azure DevOp build agent will only run one build at a time, so unless you have multiple build agents installed on the same machine it shouldn’t be a race condition.

Yes, the scanner’s ImportBefore targets file that hook into the build are stored in a shared location so will interfere with other builds unless they are deleted correctly, which should happen in the end step. That’s why I suggested checking the Run Code Analysis step logs for a specific error message.

The verbose log output for the Prepare code analysis step lists the shared locations that the ImportBefore targets files are being copied to, so you could manually check the build agent to see if they are still present. If so, it’s safe to delete them.

Duncan,

Thanks for the reply.

Yes, we have multiple build agents on the same physical box (each agent has it’s own working directory). The issue is intermitent (suggesting the race condition). And, as mentioned, I’ve seen sonarqube errors fail in my build now, even when my build has all sonarqube build tasks disabled.

Is it possible that it’s using this shared location that’s causing the issue if 2 builds are running simultaneously - and can the agent specific working directory be used instead of the shared directory? Is it something I can control/configure?

Many thanks,

Jody,

The scanner uses two targets files to hook into the build:

  • SonarQube.Integration.ImportBefore.targets is copied to a shared location on the machine. This location is specified by MSBuild and is per-user.
  • SonarQube.Integration.targets is copied to a build-specific location (namely, the .sonarqube folder that is created under the agent build directory.

The target in the shared location will be picked up by all builds, regardless of the agent. However, that file doesn’t do much, other than check for and import the second targets file in the per-build location if it exists.

Given the following scenario:

  • build A with Sonar analysis enabled,
  • build B (no analysis)
  • both builds run simultaneously on the same box by different agents;

Both builds will pick up the first targets file, which will look for the second targets file.
Build A should find the second targets file, and run the analysis.
Build B should not find the second targets file, and do nothing.

However, if your case, Build B seems to be finding the second set of targets files which fail the build because of the version of MSBuild being used.

The only way I can think of that this might happen is if the second build definition previously had analysis enabled at some point, and the build failed before the second targets file was deleted. The targets will still be on the machine in the work folder for that build/agent job combination, so any time there is a second build with analysis enabled running at the same time Build B will fail.

That’s the theory. Some practical things you can try:

  1. check what’s in the build-specific directory for your database build (i.e. the build agent …\work\xxx folder for that build definition). If there is a .sonarqube folder there, delete it.

If that folder exists then it would explain the problem, and deleting it should fix it.
If not, then I don’t what the issue is, but there is another workaround you could try to stop the second build from failing:

  1. pass the argument SonarQubeTargetsImported=true to the database build you don’t want to be analysed. You could do this in several ways:
  • by manually editing the project file to set it as an MSBuild property i.e. <SonarQubeTargetsImported>true</SonarQubeTargetsImported>
  • by passing it as an argument to the build using /p:SonarQubeTargetsImported=true
  • finally, adding SonarQubeTargetsImported as a variable in the build definition might work to (since all non-secret build variables are passed as environment variables to all tasks in the pipeline, and MSBuild turns all environment variables into build properties).

The second workaround tells the first set of targets not to import the second set of targets, even if they exist.

Hi Duncan, thanks for the reply!

Apologies, but I’ve been side-swiped by another project so we’ve just been living with this issue (kicking off the build again and again until it eventually works).

Before I try your recommendations, I’m just checking the build directory and have found this:
/agent/_work/67/.sonarqube/bin/targets/SonarQube.Integration.targets

If I open this file, I see:

true

It looks like it already has the property workaround you describe?

As mentioned previously, this is one “Build” with 2 agent phases within it; the first to compile the web project (for which I do want the SonarQube analysis) and the second to perform the database project build (which I’d like to be ignored by SonarQube entirely).

Thanks again

I have an update to this issue.

There appears to be a .targets file being created in the service account’s user directory:
C:\Users\{build service account}\AppData\Local\Microsoft\MSBuild\4.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.target

If I delete this file manually, the SonarQube analysis will not cause the the Change Automation 3 project to fail! However, it keeps cropping back up.

Is the SonarQube process somehow using the service account’s AppData directories (which definitely span build agents on the same build server) to store the .targets file?

Many thanks,