Inconsistent scan results for new code (PR) analysis

Template for a good new topic, formatted with Markdown:

  • ALM used: Azure DevOps

  • CI system used: Azure DevOps

  • Scanner command used when applicable (private details masked)

  • Languages of the repository: C#

  • Error observed:
    We constantly notice inconsistent sonarcloud reports:
    Our current process is: feature branch (FB) ->(merges into) long-lived feature branch (LLB) → main branch (MB)
    each merge (PR) is gated with sonarcloud analysis and quality gate (same CI pipeline runs for all PRs)
    during pull request from FB to LLB, if bugs/smells are found - they are resolved.
    LLB ends up containing multiple features, but it SHOULD NOT have any new bugs/code smells, because all of them should be caught by FB PR gate.
    What we see is that when we create a PR from LLB to main branch MB - we end up with gate failing because new bugs/smells are detected.
    We can trace back to specific sonar scan where the change (and therefore bug) was added to solution, but it was not detected by the gate.

  • Steps to reproduce
    Don’t know how to reproduce, new bugs/smells vary in type but we constantly see the behavior described above.

how can this be resolved? solution can only have one quality profile, so same rules should be applied to all scans

Hi @aba , welcome to the community forum.

The issues displayed on PR/feature branches are filtered to show only the issues detected in the changed files.
It can happen that a change on a given file generates an issue on another file not changed in the PR.
For example, if you change a method that can now return a null value that was not possible before we may found risks of null pointer errors where this method is called, on some other files. (I don’t know C#, I’m sorry if that doesn’t make sense for that language). Those issues won’t be detected on the PR, but only on the full branch analysis.

Without more concrete example of the unexpected issues you get on your project, I can just say that this can be a completely normal situation.
Feel free to provide more precise information, and an example on a public project, if this is not what happens on your project.

HTH,
Claire

Hi Claire,

what we see is the opposite - we tracked down a PR which contained a bug (Duplicate key ‘Components’) but the bug WAS NOT detected by sonar


this PR was successfully merged into long lived release branch and eventually from that release branch - a new PR was created to main branch. now in the second PR, which contained more features, this bug was detected for this file, for the newly added code block (line 85 from above) (Don’t have a full report anymore, sorry):

this example is for a bug in .vue code, but we see that in C# code as well

So we end up with a situation when bugs/smells are not detected when features are being merged in, but when we merge full release branch (witch should contain features with NO bugs/smells (new code bug gate is 0, that cannot be the cause)) - we suddenly see issues we need to fix.

I’ve also created another topic where I tried to rephrase this around our assumption that not all rules are being applied on every scan - could that be the case? Does Sonar run all rules for new code? - Get help / SonarCloud - SonarSource Community

Thanks for clarifying. I couldn’t reproduce the issue, the duplication was detected on the PR as expected.
Could you please share your build pipeline file or the sonar-scanner command you use to analyze the pull requests, with the project name/key and token masked?

build tasks we use are pretty straight forward

          - task: SonarCloudPrepare@1
            inputs:
              SonarCloud: 'connection'
              organization: '...'
              scannerMode: 'MSBuild'
              projectKey: '...'
              projectName: '...'
              extraProperties: |
                sonar.cs.opencover.reportsPaths=$(Pipeline.Workspace)\TestResults\coverage.opencover.xml
                sonar.javascript.lcov.reportPaths=$(build.SourcesDirectory)\src\test_reports\lcov.info
                sonar.exclusions=**/.nuget/**/*,...

then later MSBuild, Run tests, generate report, Sonar analyze and publish

since the configuration is minimal and either way the same one is applied every time we run a scan - we suspect that it’s sonar internal actions that cause this. bugs smells are found in different places (c#, vue mainly) and it constantly force us to fix code that already passed sonar gate

Thanks for the details.

Could you please build a public repository, as small as possible, with 2 branches that reproduce the error, and share the link here? I wasn’t able to reproduce it myself.
Also, could you please share the full pipeline file?

Hi Claire,

I could not reproduce this is smaller solution either, even when using these specific files files in same build/scan flow. this issue pops up from various different rules and we continue to see them.
one recent example is pretty basic c# rule:
csharpsquid:S1128 Unused “using” should be removed
it was not picked up when feature branch was merged to release and later main branches but it then detected the smell when merging main → develop.
It’s easy to create a small example and conclude that it cannot be replicated but we do see these problems and we are failing to understand how they are not being detected in the first place
our codebase size: ~155k lines
PR size: 10 - 3k lines

Also, can branching strategies play a role in this? for example, when merges are done from short lived branch (SLB) to long lived branch (LLB)? LLB to main branch (MB)? MB to LLB ?

Hi,

Issues not being shown in PRs often relate to branching strategy, to branches being incorrectly marked as short-living or long-living branch in SonarCloud, or to a too small clone depth before analysis.
The full pipeline definition, for both long branches and PRs would help.
I would also need the full logs, in DEBUG mode, produced on SonarScanner, on a build where an issue should have been detected first (for example, the commit that introduced the unused “using” statement you mentioned, by removing the code using it before, or introducing the statement without using it).

Thanks