Massive performance issues in azure integration

SonarQube Server / Community Build
, ,
1

Hi,
I am using SonarQube version 9.2.3. I am currently integrating our projects one by one into SonarQube analysis. Some of the projects are using Azure DevOps with the pipeline SonarQube plugin, others use different Build Systems.
Now I ran into massive performance issues with at least 2 different projects from our Azure projects (others seem to run OK, or at least not causing this huge issues). Both are .NET Projects.

Project 1:
Before SonarQube integration:
Complete build time: 6m44s
Build time of Build step “API Build”: 27s
Result: Always success

After SonarQube integration:
Complete build time (after increasing default 60m timeout): 1h14m (BUT: Build could not complete!)
Build time of Build step “API Build”: 1h10m (not finished, crashed during this step)
Result: Microsoft hosted build agent seems to crash (likely memory exhausted). Error:

,##[error]We stopped hearing from agent Hosted Agent. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610

Even though the message talks about CPU and network, some googling shows that mostly memory exhaustion seems to cause this.

The build step in question is just a simple:

- task: DotNetCoreCLI@2
    displayName: Build Api
    inputs:
      command: 'build'
      projects: 'xxxApi.csproj'
      arguments: '--output $(Build.ArtifactStagingDirectory) --configuration Release'

Project 2:
Before SonarQube integration:
Total Build time: 12-13 minutes
Build step “Build Debug Implementation Client”: 2m12s
Build step “Build Debug Implementation Server”:2m8s

After SonarQube integration:
Complete Build time: 1h28m (had to increase default 60m timeout)
Build step “Build Debug Implementation Client”: 27m58s
Build step “Build Debug Implementation Server”:23m46s

The build steps in question here are simple VSBuild commands.

Those build times are unacceptable for a “per commit and branch” CI/CD pipeline, even though, in Project 2 at least I can finish the analysis… For Project 1 i have no idea how to fix it. I have no control over the microsoft hosted agents and normally they should work fine for such cases. It works in other projects.

How can I figure out what the issue is here? The log shows nothing suspicous…

2

A post was split to a new topic: Analysis runs even though there is no Prepare task or any other sonarqube task in the pipeline

3

Hi,

Are you running Developer Edition? We found an issue with C# slowness related to the taint analysis rules that will be fixed in 9.4 (E.T.A. 4 April).

 
Ann

4

No, we run the enterprise edition.

5

Hi,

This would apply to Enterprise Edition too (all commercial editions).

I know 6 days is a long time in this situation, but can I ask you to try 9.4 when it comes out on Monday?

 
Ann

6

Ok thank you. Yes I can wait for Monday. Will the DevOps Plugins be automatically updated then too or do I need extra steps to use latest version on DevOps?

7

Hi,

In SonarQube? Or do you mean a plugin within your DevOps Platform? If the former, yes.

 
Ann

8

The Plugin within DevOps platform. Does it need an update too, since the scanning is done there?

9

Hi,

Fair question, but the scanners download the rules from SonarQube & the problem is rule-side. So simply upgrading SonarQube should do it.

 
Ann

1 Like
10

Thank you for your support.
I did an upgrade yesterday and can now report following feedback:

Project 1 (the one that never could finish an analysis before since Agent crashed before it ended after >1h10m):
The build finished now without issues in 23 minutes, of which only 11min are the actual build (which was the step running endlessly before). This is an acceptable result for me.

Project 2 (before upgrade had ~1h30m build times, without SonarQube had ~13min build time):
No difference here, build times are still somewhere between 1h20 and 1h50.

Any ideas how to improve the second projects build or what to look at? From 2 to 23 or 27 minutes build time (per solution build step) seems a bit strange to me, or is this expected behaviour?

11

Ok unfortunately it looks like Project 1 issue isn’t resolved. On next run the agent died again at 44 minutes runtime… Any further ideas?

13

Hi,

I’m sorry to hear the upgrade didn’t solve it. It’s odd to me that your first post-upgrade run of Project1 worked and the second run timed out.

I’m wondering if this could be relevant:

Since your jobs keep timing out, I’m guessing that none of that end-step cleanup ever happens. Do you see .sonarqube directories on your build agent?

Also, could you take a look at this & see if it helps?

And finally, I’m pretty close to being over my head here (glub, glub), so I’m going to flag this for more expert attention.

 
Ann

15

Thank you.
The issue with the breaking project seems more strange. The build times toggle between short (20-25 minutes without problem) and long (>45-60min, with agent dying). Without SonarQube, build times are stable on few minutes.
The build agents are MS hosted Azure standard agents - I don’t have control over them. In other projects, even with long builds and sonarqube included, I don’t have issues with these.

I already went before through the other threat you mentioned, I don’t see anything in the logs that could help.

1 Like
17

Hello @rkg

Totally agree with you.

Could you please share:

  • the verbose logs (msbuild and END step) with us?
  • the azure pipelines yaml file to see the configuration

I can send a private message if you prefer to avoid sending them publicly.