Sonar guidance for coverage in GitHub Actions builds

Until recently, I’ve been using automatic analysis on SonarCloud and this gives me visual feedback of the tests:

but it doesn’t include coverage. Test Coverage and Execution | SonarCloud Docs says that I need to switch to CI-based analysis, which I’ve done:

  sonarcloud:
    name: SonarCloud
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: "3.9"
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"
      - name: Install
        run: make dev-install
      - name: Unit Tests
        run: make test
      - name: SonarCloud Scan
        uses: SonarSource/sonarcloud-github-action@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # Needed to get PR information, if any
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

However, there are two problems with this approach:

  1. The visual feedback has been lost
  2. If the code changes fail to pass the Sonar gate, that isn’t reflected in the CI status. The GitHub Action believes it has succeeded. This particularly means I cannot block PRs that don’t meet the quality requirements.

How can I solve those issues, please?

Hi Philip, sorry to get back to you late!

If I understand correctly, by “visual feedback” you are referring to the comment by the sonarcloud bot as in your screenshot, with a brief summary of the quality gate passed or failed, and some stats about issues and coverage. Whether you use automatic analysis or a CI should not make a difference here at all.

If you are not getting this kind of comment after moving from automatic analysis to GitHub Actions, that indicates a problem during the step that executes SonarSource/sonarcloud-github-action@master. Can you please check the output of that step and see if it terminates normally or with an error? The last couple of lines should make it clear, and include a link to SonarCloud where you can see the result of the analysis, or more details about the failure.

In the end, you should be able to have the same end result with GitHub Actions as you did with automatic analysis, plus the results from the test coverage reports.

@janos Thanks for the reply.

Yes, I am referring to the comment made by the sonarcloud bot.

If I look at the output from the sonarcloud-github-action step, the last time it was run, the final few lines were:

INFO: Analysis report generated in 154ms, dir size=233 KB
INFO: Analysis report compressed in 13ms, zip size=55 KB
INFO: Analysis report uploaded in 321ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: https://sonarcloud.io/dashboard?id=linaro-its_aws2-wrap&pullRequest=49
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=AXte2sIgGNbRtOZQIstY
INFO: Analysis total time: 14.037 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 43.659s
INFO: Final Memory: 9M/34M
INFO: ------------------------------------------------------------------------

but I have to drill into the running of the Action to see that, rather than have a concise summary of what Sonar has determined within the GitHub Pull Request.

And you should not have to drill down. When we get this working, you will have a comment, as you had before.

The summary comment you are used to is part of what we call pull request decoration. The comment is one piece of it, another piece is setting the status of the pull request. It seems this is not working anymore for your repo.

Following the link from your output, I notice that the GitHub icon and a link is not there next to the title of the pull request, and it is correctly there on the previous pull request. It looks like something changed in between these two pull requests, and the pull request decoration is not happening anymore. I assume you went into Administration / Analysis Method, switched off the automatic analysis option, and followed the tutorial to use GitHub Actions. Is that correct? Have you made other changes in the configuration of your project on SonarCloud?

That is correct.

I don’t believe so.

I’m happy to revert the analysis method changes to check that it is the only change being made if that helps with this issue.

Is there a solution to this?

Hello,

It’s weird that the decoration didn’t happen, it seems like everything is correctly configured on your side.
Could you try to re-run the github action that does the analysis ?

I’ve made a trivial comment change to the code in order to force a re-run of the action.

image

INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=0ms
INFO: SCM Publisher SCM provider for this project is: git
INFO: SCM Publisher 2 source files to be analyzed
INFO: SCM Publisher 2/2 source files have been analyzed (done) | time=212ms
INFO: CPD Executor Calculating CPD for 1 file
INFO: CPD Executor CPD calculation finished (done) | time=6ms
INFO: SCM writing changed lines
INFO: SCM writing changed lines (done) | time=42ms
INFO: Analysis report generated in 190ms, dir size=239 KB
INFO: Analysis report compressed in 15ms, zip size=56 KB
INFO: Analysis report uploaded in 334ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: https://sonarcloud.io/dashboard?id=linaro-its_aws2-wrap&pullRequest=49
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=AXy2_tl7YP4I2HRhDl15
INFO: Analysis total time: 14.400 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 45.549s
INFO: Final Memory: 10M/37M
INFO: ------------------------------------------------------------------------

However, going to the SonarCloud link shows that Sonar has failed on the quality gate:

Looking at our logs it’s seems like there is an issue with the SonarCloud Github app token, and it’s failing in the middle of the PR decoration process because of this.

Could you try to re-install the SonarCloud Github application and re-run the analysis to see if it fixes the issue ?
To re-install the app you need to:

  • Got to your organization on Github.com → Installed Apps → and uninstall the SonarCloud App
  • This will remove the binding to your organization in SonarCloud too
  • Then go to your Sonarcloud.io organization → Administration → Organization Settings and bind you organization again with same Github organization as before, doing this should refresh the token

image

Thanks, @Gregoire_Aubert.

So what I’m seeing now is this:

followed fairly quickly afterwards by this:

and then, at the top of the PR, this:

So it looks like it is working now. Thank you!

Ah that’s great to see! Happy to have been of some help!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.