Inconsistent scan results for new code (PR) analysis

Template for a good new topic, formatted with Markdown:

  • ALM used: Azure DevOps

  • CI system used: Azure DevOps

  • Scanner command used when applicable (private details masked)

  • Languages of the repository: C#

  • Error observed:
    We constantly notice inconsistent sonarcloud reports:
    Our current process is: feature branch (FB) ->(merges into) long-lived feature branch (LLB) → main branch (MB)
    each merge (PR) is gated with sonarcloud analysis and quality gate (same CI pipeline runs for all PRs)
    during pull request from FB to LLB, if bugs/smells are found - they are resolved.
    LLB ends up containing multiple features, but it SHOULD NOT have any new bugs/code smells, because all of them should be caught by FB PR gate.
    What we see is that when we create a PR from LLB to main branch MB - we end up with gate failing because new bugs/smells are detected.
    We can trace back to specific sonar scan where the change (and therefore bug) was added to solution, but it was not detected by the gate.

  • Steps to reproduce
    Don’t know how to reproduce, new bugs/smells vary in type but we constantly see the behavior described above.

how can this be resolved? solution can only have one quality profile, so same rules should be applied to all scans

Hi @aba , welcome to the community forum.

The issues displayed on PR/feature branches are filtered to show only the issues detected in the changed files.
It can happen that a change on a given file generates an issue on another file not changed in the PR.
For example, if you change a method that can now return a null value that was not possible before we may found risks of null pointer errors where this method is called, on some other files. (I don’t know C#, I’m sorry if that doesn’t make sense for that language). Those issues won’t be detected on the PR, but only on the full branch analysis.

Without more concrete example of the unexpected issues you get on your project, I can just say that this can be a completely normal situation.
Feel free to provide more precise information, and an example on a public project, if this is not what happens on your project.

HTH,
Claire

Hi Claire,

what we see is the opposite - we tracked down a PR which contained a bug (Duplicate key ‘Components’) but the bug WAS NOT detected by sonar


this PR was successfully merged into long lived release branch and eventually from that release branch - a new PR was created to main branch. now in the second PR, which contained more features, this bug was detected for this file, for the newly added code block (line 85 from above) (Don’t have a full report anymore, sorry):

this example is for a bug in .vue code, but we see that in C# code as well

So we end up with a situation when bugs/smells are not detected when features are being merged in, but when we merge full release branch (witch should contain features with NO bugs/smells (new code bug gate is 0, that cannot be the cause)) - we suddenly see issues we need to fix.

I’ve also created another topic where I tried to rephrase this around our assumption that not all rules are being applied on every scan - could that be the case? Does Sonar run all rules for new code? - Get help / SonarCloud - SonarSource Community

Thanks for clarifying. I couldn’t reproduce the issue, the duplication was detected on the PR as expected.
Could you please share your build pipeline file or the sonar-scanner command you use to analyze the pull requests, with the project name/key and token masked?

build tasks we use are pretty straight forward

          - task: SonarCloudPrepare@1
            inputs:
              SonarCloud: 'connection'
              organization: '...'
              scannerMode: 'MSBuild'
              projectKey: '...'
              projectName: '...'
              extraProperties: |
                sonar.cs.opencover.reportsPaths=$(Pipeline.Workspace)\TestResults\coverage.opencover.xml
                sonar.javascript.lcov.reportPaths=$(build.SourcesDirectory)\src\test_reports\lcov.info
                sonar.exclusions=**/.nuget/**/*,...

then later MSBuild, Run tests, generate report, Sonar analyze and publish

since the configuration is minimal and either way the same one is applied every time we run a scan - we suspect that it’s sonar internal actions that cause this. bugs smells are found in different places (c#, vue mainly) and it constantly force us to fix code that already passed sonar gate

Thanks for the details.

Could you please build a public repository, as small as possible, with 2 branches that reproduce the error, and share the link here? I wasn’t able to reproduce it myself.
Also, could you please share the full pipeline file?