Inaccurate Sonar scanner report with respect to changed files and lines after merge events

SonarQube Cloud
,
1
  • ALM used: GitHub
  • CI system used: Github Actions
  • Languages of the repository: C++, Objective C
  • Error observed : the Sonar scanner reports a number of code smells and uncovered lines which are not related to the actual pull-request changes after a merge event (main branch towards the pull-request’s feature branch). It seems that the report does not distinguish anymore the changes which are part of the pull-request and those that were merged from the main branch. Irrelevant files and lines are displayed in the web interface regarding code smells and coverage.
  • Steps to reproduce: Let the CI Analysis run after the first changes included in a new Github pull-request. The Sonar Analysis is then accurate. Code smells and uncovered code lines are reported appropriately. Merge then the main branch into the pull-request’s feature branch and push these new changes to Github. The CI Analysis reports code smells and uncovered code lines which are not related to the initial changes of the pull-request’s feature branch.

More on my setup:

  • I am using an Analysis of type CI for my C++/Objective C project (automatic analysis disabled from the SonarCloud web interface)
  • I am using a Github workflow run by a self-hosted Window 11 CI runner.
  • The Windows OS version is: Microsoft Windows [version 10.0.22631.4249].
  • Shallow git cloning is disabled (fetch-depth: 0, see Github workflow excerpt below).
  • The New Code definition in my Sonar Cloud web interface is set to “1 day”. (I have only two options, version versus number of days). I believe this definition should be overridden by the scanner configuration parameters used when running the action SonarSource/sonarqube-scan-action@v4, see Github workflow excerpt below.

Here is an excerpt of my Github workflow containing the relevant Sonar scanner action:

name: Windows CI

on:
  pull_request:
    branches-ignore:
        - 'release/**'
    types: [synchronize, review_requested]

concurrency:
  group: windows-${{ github.head_ref }}
  cancel-in-progress: true

env:
<some secrets and env vars>

jobs:
  build-kDrive:
    runs-on: [ self-hosted, Windows, desktop-app ]
    env:
      PR_SOURCE_BRANCH: ${{ github.head_ref }}
      PR_TARGET_BRANCH: ${{ github.base_ref }}
      COVFILE: ${{ github.workspace }}\\src\\test.cov
    steps:
      - name: Load system Path into Github environment
        run : echo Path=%Path%>> %GITHUB_ENV%
        shell: cmd

      - name: Checkout the PR
        uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref }}
          submodules: recursive
          fetch-depth: 0
        
      - name: Clean the log directory
        run : rm -r -force C:/Windows/Temp/kDrive-logdir/*
      
       <some irrelevant tasks>

      - name: Build application
        run: |
               <build instructions>
        shell: cmd

      - name: Execute tests
        run: |
         <tests instructions>

      - name: Compute code coverage
        if: true
        run : |          
          <coverage computation via BullseyeCoverage>

      - name: Run sonar-scanner
        if: true
        env:
          GITHUB_TOKEN: ${{ github.token }}
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: https://sonarcloud.io
          INCLUDE: "C:\\"
        uses: SonarSource/sonarqube-scan-action@v4
        with:
          args: >
            --define sonar.projectKey="<project-name>"
            --define sonar.organization="<my-organisation>"
            --define sonar.host.url="https://sonarcloud.io"
            --define sonar.sources="src,extensions"
            --define sonar.tests="test"
            --define sonar.exclusions="src/3rdparty/**,src/gui/**"
            --define sonar.cfamily.compile-commands="build-windows/build/compile_commands.json"
            --define sonar.cfamily.bullseye.reportPath="coverage.xml"
            --define sonar.scm.exclusions.disabled=true
            --define sonar.verbose=true
            --define sonar.newCode.referenceBranch
            --define sonar.branch.name="${{ env.PR_SOURCE_BRANCH }}"
            --define sonar.branch.target="${{ env.PR_TARGET_BRANCH }}"

<some post-build steps and cleanup steps>

Thanks in advance for your guidance.

Best regards,
Luc

2

Hey @luc.guyot

Based on your post, I think what you’re after is a PR analysis, which should only show the changes that would be merged into the target branch.

Instead, you’re configuring branch analysis with sonar.branch.* and setting sonar.newCode.referenceBranch (which does not work on SonarQube Cloud).

Why are you setting these parameters? What happens if you remove them?

Am I missing something about what you’re trying to do?

3

Hi @Colin ,

That’s very nice of you to step in.
You are absolutely right: I am after PR analysis.

The three branch-related parameters that I have used in my Github workflow for the Sonar scanner were part of an unsuccessful attempt to activate this type of analysis.

So, with or without these parameters, the issue I reported is the same.
We can ignore them for the rest of the discussion.

We read in the online documentation that

In the SonarQube Cloud Free plan, pull request analysis is available only when the pull request is merged into the main branch. Please see the Comparison table on the Subscription plans for plan details.

As I am only targeting my main branch (unless I incorrectly read my Sonar Cloud configuration), I suppose I am eligible for PR Analysis.

Note. The Automatic analysis of Sonar Cloud is working properly for me, in the sense
that it works as a PR Analysis: it only shows the changes that would be merged into the target branch. It is only when I switch to CI Analysis (see above Github workflow, with branch parameters removed) that the issue happens. I need a CI Analysis because I want to include code coverage information from BullseyeCoverage.

Best regards,
Luc

4

Hey again @luc.guyot

Thanks for the info. In the absence of sonar.branch.* parameters, the scanner (the GitHub Action) should automatically configure everything necessary for PR analysis.

There should be some logs like this:

10:37:30.464 INFO  Load branch configuration
10:37:30.465 INFO  Github event: pull_request
10:37:30.475 INFO  Auto-configuring pull request 56159

Is that what you see?

5

Dear @Colin ,

Yes, this is what I see:

14:41:38.658 INFO  Load project pull requests
14:41:38.658 DEBUG --> GET https://sonarcloud.io/api/project_pull_requests/list?project=<my-project-name>
14:41:38.944 DEBUG <-- 200 https://sonarcloud.io/api/project_pull_requests/list?project=<my-project-name> (282ms, unknown-length body)
14:41:38.969 INFO  Load project pull requests (done) | time=311ms
14:41:38.975 INFO  Load branch configuration
14:41:38.975 INFO  Github event: pull_request
14:41:38.975 INFO  Auto-configuring pull request 610

Should I display more information?

Best regards,
Luc.

6

It would be great if you could share the full analysis logs at DEBUG level (they appear to already be at DEBUG level).

7

Hi @Colin,

I attached to this message the full log of the Github CI build of a build for which the issue arises.

Most likely, the relevant information can be found below the line containing the comment ##[group]Run SonarSource/sonarqube-scan-action@v4.

As the top of the logs refers to some configurations and operations of git , I have included them as well.

Note. The issue I reported at the top of this thread can also happen while there is no merge event from the main branch into the feature branch. I cannot exclude that I misconfigured something at the level project of the SonarCloud web interface. We may check this too.
logs.txt.zip (6.6 MB)

8

Thanks @luc.guyot!

I think the problem comes from here:

Instead of running on the pull request merge branch, SonarQube Cloud is detecting lines from the branch itself. As such, you get the behavior you experience:

Do you know why you have your build configured this way? It’s not the default checkout strategy.

9

Dear @Colin,

Thanks for your head ups and your careful inspection of the git configuration.

The reason why we opted for the non-default option ref: ${{ github.head_ref }} is still unclear to me, but I’ll figure it out.

Note. This configuration of the git checkout with non-default ref has not prevented so far the SonarCloud Automatic Analysis from being fully accurate. The issue is visible with CI Analysis.

Following your lead, I removed this non-default option from our Github workflow’s checkout action.

Unfortunately, the issue persists. At least to some extent.

In my latest trial, the reported issues and the associated files and lines are correct, but Duplication on New Code isn’t. The SonarCloud report mentions indeed a file, with some changed lines, that is not modified by the pull-request. (The aforementioned changes had been merged into the main branch in another PR, before the creation of the PR under scrutiny).

Best regards,
Luc

11

@luc.guyot I’m really sorry this thread sled off my plate. I plan to come back to it next week.

12

Hey Luc!

It sounds like everything is working as expected. This is indeed still code duplication, but the duplicated code exists outside of your PR changes. You probably still want to clean up that duplication (otherwise it will just be duplication that gets added once you merge).

Does that make sense?