Azure DevOps build stuck after migrating from SQ 9.1 to SQ 9.2

Build stuck with an error
Azure build SQ Developer version:
SonarQube ID information
Version: 9.2.4.50792
Date: 2022-01-21

5>CoreCompile:
         CompilerServer: server failed - server rejected the request due to analyzer / generator issues 'analyzer assembly 'C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\.sonarqube\resources\2\System.Buffers.dll' has MVID '36e84b0d-9d74-4086-a062-54e1963f24d5' but loaded assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' has MVID 'f389ca43-32bf-4e24-ad8c-a6ed5efddff3', analyzer assembly 'C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\.sonarqube\resources\2\System.Numerics.Vectors.dll' has MVID '95de52ab-0179-450a-9f6f-08d224d60b15' but loaded assembly 'System.Numerics.Vectors, Version=4.1.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' has MVID '34905ed1-db9d-4250-b31b-b80e1ff70ff5', analyzer assembly 'C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\.sonarqube\resources\2\System.Runtime.CompilerServices.Unsafe.dll' has MVID 'bd600ba8-23b5-4d45-ba63-f24457fa3be3' but loaded assembly 'System.Runtime.CompilerServices.Unsaf...
##[warning]OnlineForms\SeleniumTest.cs(57,33): Warning CS0168: The variable 'ex' is declared but never used

Have successful build at:
Docker Community version:
SonarQube ID information
Version: 9.1.0.47736
Date: 2022-01-20

what is the workaround?

& if i remove SQ tasks from the build, the build is successful

& successful & not are on a diff pc`s

any thoughts?

it just blocks us to use what we payed for

Hello @Aleksandr_Kovalev

We really need the full verbose logs. The log lines you gave tell us nothing about the real problem of the failure.

Please give us the following version information:

And we’ll need the logs:

  • please give us the verbose output of the scanner commands (please run SonarScanner.MSBuild.exe begin /k:“MyProject” /d:sonar.verbose=true as the begin step, and please attach the output of the BEGIN and END steps)
  • please give the output of running MSBuild in verbose mode (/v:d)

Also, please tell us:

  • what are the commands you are running to do the analysis? (sharing the pipeline yaml would help)

Thanks

@Andrei_Epure

yaml

#Multi-configuration and multi-agent job options are not exported to YAML. Configure these options using documentation guidance: https://docs.microsoft.com/vsts/pipelines/process/phases

pool:
  name: AWS
  demands:
  - msbuild
  - visualstudio

variables:
  BuildPlatform: 'Mixed Platforms'
  BuildConfiguration: 'Release'

steps:
- task: NuGetToolInstaller@1
  displayName: 'Use NuGet '

- task: NuGetCommand@2
  displayName: 'NuGet restore'
  inputs:
    restoreSolution: 08Stuff.sln

- task: SonarSource.sonarqube.15B84CA1-B62F-4A2A-A403-89B77A063157.SonarQubePrepare@5
  displayName: 'Prepare analysis on SonarQube'
  inputs:
    SonarQube: sonarqube
    projectKey: Personal
    projectName: Personal

- task: VSBuild@1
  displayName: 'Build solution 08Stuff.sln'
  inputs:
    solution: 08Stuff.sln
    msbuildArgs: '/v:d'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    clean: true
    maximumCpuCount: true

- task: SonarSource.sonarqube.6D01813A-9589-4B15-8491-8164AEB38055.SonarQubeAnalyze@5
  displayName: 'Run Code Analysis'
  enabled: false

- task: SonarSource.sonarqube.291ed61f-1ee4-45d3-b1b0-bf822d9095ef.SonarQubePublish@5
  displayName: 'Publish Quality Gate Result'
  enabled: false

build logs

sq_logs.txt (1.0 MB)

sq_scanner_verbose_logs.txt (1.1 MB)

@Andrei_Epure I also see that the caption of the topic was changed, basically I did not do any migrations, I just used the latest dev edition version on the server & when ran into the issue I installed the latest docker community version on my local pc to test, so as a conclusion - no migrations, just build with the version I could get

Hi @Aleksandr_Kovalev,

Did you cancel/stoped the build? I’ve noticed this in the logs and I would to know if this is to be expected:

warning MSB5021: Terminating the task executable "csc" and its child processes because the build was canceled.

As I’ve noticed that the build was stopped after 10 minutes, can I ask what is the exact problem? How long did the build take when analyzing with SQ 9.1 and how long does it take now?

Thank you,
Costin

yes, I cancelled the build, no reason to wait, it will never end (waited about 1 hour), without SQ it takes < 1 min

SQ 9.1 build takes: Time Elapsed 00:02:33.10

@Andrei_Epure @costin.zaharia i have updated my local docker version to sonarqube:9.2.4-community & the build stuck the same as on the server

Unfortunately with an unfinished build we don’t have enough information to figure out where the problem might be.

Visual studio outputs a nice report when /p:reportanalyzer=true is set when MsBuild.exe or dotnet is called but only if the build finishes.

  • SonarQube 9.1 uses the 8.29 version of the analyzer and SonarQube 9.2.1 uses the 8.32 so the problem might be between one of these releases.

  • Looking at the logs, I see that indeed nothing happens for ~10 minutes (from 2022-01-26T12:22:41.0313210Z to 2022-01-26T12:32:29.2581288Z) which is not expected.

My first attempt would be to exclude from analysis these 5 projects which were built when the cancel was triggered and see if the problem disappears:

  • CatalogGenerator\CatalogGenerator.csproj
  • 08Stuff\08Stuff.csproj
  • Core\Core.csproj
  • 08pdfGen\08pdfGen.csproj
  • 08StuffHelpers\08StuffHelpers.csproj

You can do that by adding the following in the csproj files:

<!-- in .csproj -->
<PropertyGroup>
  <!-- Exclude the project from analysis -->
  <SonarQubeExclude>true</SonarQubeExclude>
</PropertyGroup>

If the problem disappears, you can enable the analysis of the projects one by one to narrow down the problem.

I also have a few questions:

  • are the projects open source?
  • are code generators used in these projects or is there any other kind of generated code?
  • is there anything else particular to these projects? (e.g. very large files, entity framework database migrations)

@costin.zaharia

  • are the projects open source? - NO
  • are code generators used in these projects or is there any other kind of generated code? - NO (EF maybe)
  • is there anything else particular to these projects? (e.g. very large files, entity framework database migrations) - i see edmx there, but not sure about migrations, the project has ref to f# project with a lot of small f# files (but i have similar solution which builds without any problems)

@Aleksandr_Kovalev : thanks a lot for your prompt response.

Could you please try to narrow down the problem by excluding the mentioned projects from the analysis? This might also be a work-around until we manage to find and fix the problem.

the problem in these two

reviewed quickly, nothing special, just cs files

Based on what you found, would it be possible to create a reproducer for us?