ALM used (GitHub, Bitbucket Cloud, Azure DevOps) Azure DevOps
CI system used (Bitbucket Cloud, Azure DevOps, Travis CI, Circle CI Azure DevOps
Scanner command used when applicable (private details masked) Azure DevOps tasks: SonarCloudPrepare@1, SonarCloudAnalyze@1, SonarCloudPublish@1
Languages of the repository C#
My team uses SonarCloud integrated with Azure DevOps pipelines and SonarLint for Visual Studio. Both point at the same project(s) with the same underlying Quality Profile. However, we often notice that the code analysis results are different between SonarLint analysis in VS and pipeline analysis in DevOps.
A typical workflow is that developers make their changes in Visual Studio, run code analysis on the solution, fix detected issues, and create a PR in DevOps that triggers the validation build, including the SonarCloud analyzer. The problem is that SonarLint doesnāt flag various issues, but they are flagged on PR builds. Please see the below examples.
Example #1 Note: This is the biggest problem as it generates for many files, sometimes resulting in dozens of auto-generated comments on the PR that have to be resolved manually.
SonarCloud analysis:
Code Smell: Missing XML comment for publicly visible type or member āREDACTEDā (external_roslyn:CS1591)
SonarLint analysis:
No issue flagged
Example #2
SonarCloud analysis:
Code Smell: Remove this empty statement. (csharpsquid:S1116)
SonarLint analysis:
No issue flagged
Example #3
SonarCloud analysis:
Code Smell: Complete the task associated to this āTODOā comment. (csharpsquid:S1135)
Itās expected that this one isnāt raised by SonarLint. The external_roslyn part of the rule key indicates that itās not a Sonar rule. Itās raised by some other part of your build and automatically imported into SonarCloud. (See the bottom of this page.)
Thx,
Ann
P.S. Can you provide the version details of SonarLint and of Visual Studio, please?
Thanks for your response! That is correct, SonarLint is in connected mode bound to the same project that our pipeline in VisualStudio uses for the SonarCloud tasks. Thank you for the hint about external_roslyn. We will investigate this further. Of course the issues with other warnings are still outstanding as they seem to originate directly from Sonar.
Here are the software versions:
SonarLint 6.12.0.59751
Visual Studio Professional 2022 (64-bit) 17.4.2
@ernest_sioa are any Sonar/MS C# issues being reported in the IDE?
The Sonar C# and VB.NET rules are implemented as Roslyn analyzers, the same as the built-in Microsoft analysis rules. That means that Visual Studio controls when analysis is run and on which files.
By default, Visual Studio will not run analysis against the entire solution for performance reasons so you only see issues for a subset of files. See the Microsoft docs for more information.
Hi duncanp, thanks for your response! Indeed, we get most of the issues shown in DevOps reported in IDE as well. In Visual Studio, we manually run code analysis on a solution, and thatās how we get the issues to appear.
However, as stated earlier, some of the issues (such as issues #2 and #3) donāt appear in IDE, but they do appear in DevOps.
just to check, by āwe manually run code analysis on a solutionā, you mean by selecting Analyze, Run Code Analysis, On Solution?
do the expected errors appear if you open the containing file?
The MS docs suggest that the menu command should run Roslyn analyzers (like those for Sonar C# and Sonar VB.NET).
However, I increased the MSBuild logging output in VS to detailed, clicked on the menu command, and looked at the output in the Build pane in the Output window. The /analyzer: arguments passed to csc.exe do not include any of the Sonar analyzers: it looks like they only contain the first-party Microsoft .NET analyzers.
So, it looks like Run Code Analysis does not run third-party analyzers by default, although the documentation does not say this. I havenāt found a way of including the Sonar analysers in the set of analyzers to be executed.
Third-party analyzers like the Sonar ones are run live in the IDE when editing (depending on how you have VS configured - see custom analysis scope for more information).
So developers should be able to see (most*) Sonar issues while they are developing. Viewing all of the issues in the solution is overwhelming, so I normally have the Error List filter set to show only the issues for the current document:
*most: I say most Sonar issues should appear in the IDE because it appears there are certain kinds of Roslyn rule that are not executed in the IDE unless the analysis scope is set to the whole solution. Again, I havenāt found this documented anywhere in the MS docs; itās based on observation and experimentation. This limitation applies to rules that depend on the whole "compilation. However, most of our Roslyn rules donāt require the whole compilation.
When you mentioned the Error List filter, I played about with it a bit, and it looks like the issue is not with the analyzer but instead the refreshing of that list. When I ran the code analysis on the solution (and yes, as per your question #1 this is indeed what we do: RMB click solution ā Analyze and Code Cleanup ā Run Code Analysis on Solution), the issues didnāt initially appear. However, when I switched the filter to anything else, and then back āEntire Solutionā I can see all issues properly listed.
I donāt know if itās an issue with the IDE, or the analyzer, but clearly, itās a list refreshing problem. I checked on another developerās machine and can confirm that we could reproduce the problem.
Thank you for that hint as it will help us detect all issues before we create PRs. Itās still a bit inconvenient but it sounds like a Visual Studio problem that need to be flagged with Microsoft, unless you think that itās related to Sonar analysis?
Just to comment on this one- thatās not what we observe. We actually do get Sonar analysis results when running code analysis on a solution.
I donāt think itās related to the Sonar analyzers. The Sonar analyzers are normal Roslyn analyzers. Visual Studio is responsible for running them, displaying the results, and for filtering the Error List.
