Error observed: Long build times and increased space usage
Good Morning,
I would like to start by saying that I have looked at some of the similar topics relating to build times but our situation is a tad different. We have a project that has 19 projects, a few of them are pretty large and the total amount of lines in the project is probably around 700k. Building without SonarCloud takes about 15 minutes, but building with SonarCloud takes over an hour and a half, though I’m not sure how much longer because we use hosted Azure-Pipelines agents and it also adds too much space and inevitably puts the build over the 10 gb limit for a hosted agent. The slow build times make it difficult to coordinate with PRs (a feature we use in other projects) but more than that the increase in used space makes it impossible for us to use the feature because the build will fail when it reaches the limit.
This is just a thought and I don’t know how your system works but for large projects like that does the analysis need to be done right at that moment that the build is run? Aren’t the files uploaded to the endpoint anyway, and if so could there be an extra option or premium feature that allows the build to upload the files and the analysis to be performed at the endpoint so it doesn’t disrupt the builds as much?
I wrote a brief overview of how our .NET analysis works. Feel free to ask more questions if you want.
The analysis is (mostly) done during the build because for .NET, our analyzers are Roslyn analyzers (.NET Compiler Platform). They are invoked by MSBuild during compilation. Our advanced vulnerability analysis is done separately, after the build (but most of the analysis is done during the build).
We do have Automatic Analysis for some languages (check the list here), but not for .NET.
I suggest that we focus on this part, and see if there’s any improvement that you can make on your side, or if we can learn about specific performance bottle-necks that manifest on your code base.
To see which rules take the most time during your build, it would help to run the build in verbose mode.
First run the scanner begin command (to reproduce locally) - check the scanner for .net docs.
Then run the build in verbose mode with reportanalyzer set to true
select-string -path "build_logs.txt" -pattern "NOTE: Elapsed time may be less than analyzer execution time because analyzers can run concurrently." -Context 1,100 > analyzer_times.txt
And share with us the analyzer_times.txt . If you don’t to share them publicly here, I can send you a private message.
Thank you for the very detailed post, I’ve been working on this issue a bit more since submitting this question and I may have actually found a solution that works for us. Most of our logic that we would like analyzed for this project is actually in about 4 projects, the others are libraries that rarely change, so we decided to analyze the projects with the standalone analyzer instead of using the msbuild integration, doing that we were able to specify the projects we wanted analyzed and leave out the projects that don’t change. This sped up the process significantly and we’re also using less space. Thank you for your time but it looks like this can be closed.
Regarding this, I’d like to better understand what you did - do you mean you added the SonarAnalyzer.CSharp nuget directly in the project files?
If you don’t use the scanner integration, you will be missing syntax highlighting and metrics in SonarCloud… And actually, without using the scanner, it wouldn’t really push the results from the Roslyn analyzers, either, right? Are you still integrating with ?
Please note that you can use the sonar.exclusions parameter to Narrow the Focus and exclude files based on patterns.
Also, if you add
<!-- in .csproj -->
<PropertyGroup>
<!-- Exclude the project from analysis -->
<SonarQubeExclude>true</SonarQubeExclude>
</PropertyGroup>
in your projects, they will be excluded from analysis (see SonarScanner for .NET docs).
?