Different scan result between Bamboo and GitHub

SonarQube Server / Community Build
,
1

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    SonarQube: 9.9.1 Enterprise LTS
    GitHub:
    sonarsource/sonarqube-scan-action@master
    sonarsource/sonarqube-quality-gate-action@master
    Bamboo:
    Sonar plugin 2.10.1
    SonarScanner 4.8

  • how is SonarQube deployed: zip

  • what are you trying to achieve: I want to get the same scan result in GitHub as we did before in Bamboo

  • what have you tried so far to achieve this: I tried playing around with the sonar.branch.name and sonar.projectVersion properties in the sonar-project.properties file.

The problem:
Scanning the develop branch (main branch) in Bamboo returns the expected result. I assume it is a full scan and not a diff scan. It takes about an hour.
In SonarQube I see on the main project page: New Code since 9/11/2022, started one year ago.
I see also an activity graph with issues. This graphs remains steady and predictable when scanning with Bamboo

We are migrating to GitHub and when I scan the develop branch, the nr of issues goes to 0. Also the acticvity graph goes to 0 for that analysis. So there is a difference between a scan in Bamboo and GitHub. We can to keep track of our issues.
My guess is in GitHub we somehow scan as a PR and only check the diffs? I tried multiple things, but cant get it to work as we would like.

I already cleaned up alot of deprecated/legacy properties in our sonar-project.properties file. But maybe there is more legacy causing this issue.

Any help is appreciated. If you need more information or details, please ask.

2

Hey there.

A good starting point would be to produce DEBUG logs (sonar-scanner -X) of the runs in Bamboo and GitHub and start looking for obvious differences. You can also share them here.

3

scanner_log.zip (22.1 KB)
I tried to cleanup the logs and remove the sensitive data. See attached zip which contains the logs from GitHub and Bamboo

4

The logs basically look the same (files indexed) except that your Bamboo analysis wasn’t able to make use of an analysis cache.

Bamboo:

|build|08-Nov-2024 09:08:40|INFO: Starting batch processing.|
|build|08-Nov-2024 09:08:41|INFO: The Java analyzer cannot skip unchanged files in this context. A full analysis is performed for all files.|

GitHub

2024-11-08T08:58:29.3534434Z 08:58:29.353 INFO  Server-side caching is enabled. The Java analyzer was able to leverage cached data from previous analyses for 17854 out of 17858 files. These files will not be parsed.`

Can you share a screenshot of what you’re seeing in SonarQube after performing the analysis of the develop branch in GitHub?

5

“xxx-core” is from Bamboo
“xxx-core / develop” is from GitHub

image
image1323×1035 84.3 KB

Basically this graph drops to 0 when having scanned via GitHub. I removed most (incorrect) scans from the past. Leaving only a few for illustration. One on the right is from today, but hard to see on the graph.

image
image1313×1248 82.1 KB

And the project dashboard after the scan from GitHub. Normally there are more (historical) issues reported here. Now everything is basically 0.

image
image1356×1236 69.8 KB

6

The first thing I’d like to determine is if it’s related to the analysis cache.

Can you disable the caching mechanisms as documented here when using GitHub and report whether or not the issue count changes?

7

Hi Colin,
Thank you for all your (quick) feedback up until now!

New results are more or less the same.
GitHub workflow log indicating the settings:

Run sonarsource/sonarqube-scan-action@master
  with:
    projectBaseDir: core
    args: -Dsonar.projectBaseDir=/github/workspace/core -Dsonar.working.directory=/github/workspace/core/.scannerwork -Dsonar.java.skipUnchanged=true -Dsonar.verbose=true -Dsonar.analysisCache.enabled=false
  
  env:
    RUN_SonarQube: true
    DEV_CI_SCRIPTS_BRANCH: develop
    JAVA_HOME: /opt/github-runner/action-runner/_work/_tool/Java_Temurin-Hotspot_jdk/17.0.13-11/x64
    JAVA_HOME_17_X64: /opt/github-runner/action-runner/_work/_tool/Java_Temurin-Hotspot_jdk/17.0.13-11/x64
    SONAR_TOKEN: ***
    SONAR_HOST_URL: https://sonarqube.company.be
    SONAR_SCANNER_OPTS: -Xmx6500m

I tried something else. This time with sonar.java.skipUnchanged=false

Run sonarsource/sonarqube-scan-action@master
  with:
    projectBaseDir: core
    args: -Dsonar.projectBaseDir=/github/workspace/core -Dsonar.working.directory=/github/workspace/core/.scannerwork -Dsonar.java.skipUnchanged=false -Dsonar.verbose=true -Dsonar.analysisCache.enabled=false
  
  env:
    RUN_SonarQube: true
    DEV_CI_SCRIPTS_BRANCH: develop
    JAVA_HOME: /opt/github-runner/action-runner/_work/_tool/Java_Temurin-Hotspot_jdk/17.0.13-11/x64
    JAVA_HOME_17_X64: /opt/github-runner/action-runner/_work/_tool/Java_Temurin-Hotspot_jdk/17.0.13-11/x64
    SONAR_TOKEN: ***
    SONAR_HOST_URL: https://sonarqube.company.be
    SONAR_SCANNER_OPTS: -Xmx6500m

This yielded the expected result.

Still some questions and concerns:

  • sonar.analysisCache.enabled=false increases the scan time considerably, even for PR’s. I know this is normal. Yes, I can work around it by making my workflow action dynamic (which I have done already). But I would expect that the scanner detects branch vs PR scan and by default sets the correct values in the background, no?
  • Is there no option to only scan for the differences on a branch scan AND keep the issue history? I know we set a project version in the past and it is something I try to remove without losing history. But might the project version from 2 years ago be the culprit here?
  • Any other things I should do to make this just work as expected without all the config properties?
8

Hey @Dennis_DECA

Just to confirm – does setting sonar.java.skipUnchanged=false and -Dsonar.analysisCache.enabled=true (alternatively, not setting sonar.analysisCache.enabled) a working combination?

If so, next we’ll dig into why sonar.java.skipUnchanged is affecting the analysis results. It should only speed it up, not remove issues.

This is the goal of sonar.java.skipUnchanged and the analysis cache.

9

Hi Colin,

New test run with the suggested settings has also a positive result. All issues are reported as expected.

Run sonarsource/sonarqube-scan-action@master
  with:
    projectBaseDir: core
    args: -Dsonar.projectBaseDir=/github/workspace/core -Dsonar.working.directory=/github/workspace/core/.scannerwork -Dsonar.java.skipUnchanged=false -Dsonar.verbose=true -Dsonar.analysisCache.enabled=true

Based on the tests we ran, it indeed seems like sonar.java.skipUnchanged is the root cause.
My guess it only takes into account the issues from the scanned files. If there are just few files with no (new) issues it will be reported as such.
It feels like it is treated as a PR diff scan, which also only reports new issues for the changed files, not the complete history of the complete project.
Where as what I try to do here is a full branch scan. In this case I expect to keep the history and add/remove issues. Likely these issues already updated in the project by the PR scan that came before.

If you like to see more of the log, Im happy to provide it. Or other SQ server/project settings.

10

Thanks.

That’s really weird behavior.

My understanding of sonar.java.skipUnchanged is that it should only be applied in a PR context.

  • How’s your build time looking now?
  • Can you share your full GitHub Actions YML file?

I’d like to pass this on to some expert eyes who understand the mechanism better – however, be aware they have quite a backlog at the moment. I’m hoping your build time is back as expected and at least for now, you can live with the setting (even though it should be unnecessary).

12

No worries. Indeed I get the expected results now. Speed for the main branches scan is less of a problem.

PR scan about 15 min (incl. build)
Branch scan about 1 hour (incl. build)

Is this what you are looking for? GitHub Workflow snippet:

      - name: SonarQube Scan
        if: env.RUN_SonarQube == 'true'
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONARQUBE_TOKEN }}
          SONAR_HOST_URL: ${{ vars.SONARQUBE_HOST_URL }}
          SONAR_SCANNER_OPTS: -Xmx6500m
        with:
          projectBaseDir: ria
          args: >
            -Dsonar.projectBaseDir=/github/workspace/xx
            -Dsonar.working.directory=/github/workspace/xx/.scannerwork
            -Dsonar.java.skipUnchanged=${{ github.event_name == 'pull_request' }}
            -Dsonar.verbose=${{ github.event_name != 'pull_request' }}
            -Dsonar.analysisCache.enabled=${{ github.event_name == 'pull_request' }}
            
      - name: SonarQube Quality Gate check
        if: env.RUN_SonarQube == 'true'
        uses: sonarsource/sonarqube-quality-gate-action@master
        # Force to fail step after specific time.
        timeout-minutes: 30
        env:
          SONAR_TOKEN: ${{ secrets.SONARQUBE_TOKEN }}
          SONAR_HOST_URL: ${{ vars.SONARQUBE_HOST_URL }}
        with:
          scanMetadataReportFile: xx/.scannerwork/report-task.txt
          pollingTimeoutSec: '900'
14

Hi Dennis,

For Java analysis, please note that the skipUnchanged parameter cannot be enabled for branch analysis; otherwise, the issue you’re experiencing is expected. The cache’s goal is to make PR analysis faster; however, for branch analysis, we always perform a full scan.

To summarize, correct me if I am wrong: You moved your CI from Bamboo to GitHub, and in branch analysis, the skipUnchanged parameter was set to true without you explicitly configuring it, leading to the reported issue. Hardcoding skipUnchanged to false is a good workaround for now. You should not need to adjust the cache parameter, and it would be better not to do so. I will assign this issue to the team responsible for the integration with GitHub.

One last question: It seems the cache is working for PR analysis. Are you experiencing the performance you were expecting in that context?

1 Like
16

Question
You moved your CI from Bamboo to GitHub, and in branch analysis, the skipUnchanged parameter was set to true without you explicitly configuring it, leading to the reported issue.
Answer
Correct

Question
Hardcoding skipUnchanged to false is a good workaround for now.
Answer
Not really hard coded. Dynamic based on GitHub variable check. Default GitHub workflow functionality:
-Dsonar.analysisCache.enabled=${{ github.event_name == ‘pull_request’ }}

Statement
I will assign this issue to the team responsible for the integration with GitHub.
=> Great! Either implement the logic inside the action or update the documentation to configure the property in the workflow like I have.

Question
It seems the cache is working for PR analysis. Are you experiencing the performance you were expecting in that context?
Answer
Everything is relative. About 10 minutes for a PR scan, which is fast enough for us.
In the past with java8 the cache didn’t seem to work properly and we had scan times of 30-60 minutes for a PR. After moving to java17, this issue was magically fixed. See my previous topics about this issue.
The full branch scan takes about 1 hour for 3 million lines of code. As we are not actively waiting on the branch scan, this is again fast enough for us.
PR scan times are what matter to us.
=> Performance is meeting expectations.

21

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.