Status not being correctly reported to Github

SonarQube Cloud
1

Hi,

We use a mono-repo approach for our python code. This is working well mostly. We are experiencing intermittent issues the Github Action not reporting the status back to Github from SonarCloud. This means we cannot use the quality gates as a blocking step which is not ideal.

Strangely the PR is annotated with the correct metrics and it is reported to be successful under the waiting for Analysis step - it reads SonarCloud Code Analysis Successful in 28s. The PR can be viewed correctly in SonarCloud.

Here are our sonar logs:

/usr/bin/docker run --name fdd3d067422565d45453b88842d59e641b51d_f0d472 --label 4fdd3d --workdir /github/workspace --rm -e "pythonLocation" -e "PKG_CONFIG_PATH" -e "Python_ROOT_DIR" -e "Python2_ROOT_DIR" -e "Python3_ROOT_DIR" -e "LD_LIBRARY_PATH" -e "CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE" -e "GOOGLE_APPLICATION_CREDENTIALS" -e "GOOGLE_GHA_CREDS_PATH" -e "CLOUDSDK_PROJECT" -e "CLOUDSDK_CORE_PROJECT" -e "GCP_PROJECT" -e "GCLOUD_PROJECT" -e "GOOGLE_CLOUD_PROJECT" -e "VENV" -e "GITHUB_TOKEN" -e "SONAR_TOKEN" -e "args" -e "INPUT_PROJECTBASEDIR" -e "INPUT_ARGS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "GITHUB_ACTION_PATH" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_ID_TOKEN_REQUEST_URL" -e "ACTIONS_ID_TOKEN_REQUEST_TOKEN" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true --entrypoint "/entrypoint.sh" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/acme-python/acme-python":"/github/workspace" 4fdd3d:067422565d45453b88842d59e641b51d
INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
INFO: Project root configuration file: /github/workspace/src/apis/finance-api/sonar-project.properties
INFO: SonarScanner 5.0.1.3006
INFO: Java 17.0.8 Alpine (64-bit)
INFO: Linux 5.15.0-1053-azure amd64
INFO: User cache: /opt/sonar-scanner/.sonar/cache
INFO: Analyzing on SonarCloud
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=377ms
INFO: Server id: 1BD809FA-AWHW8ct9-T_TB3XqouNu
INFO: User cache: /opt/sonar-scanner/.sonar/cache
INFO: Loading required plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=377ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=1164ms
INFO: Found an active CI vendor: 'Github Actions'
INFO: Load project settings for component key: 'acme_python_finance-api'
INFO: Store cache: Starting
INFO: Store cache: Time spent was 00:00:00.000
INFO: python security sensor: Time spent was 00:00:01.179
INFO: python security sensor: Begin: 2024-01-22T22:52:25.836679735Z, End: 2024-01-22T22:52:27.016374420Z, Duration: 00:00:01.179
  Load type hierarchy and UCFGs: Begin: 2024-01-22T22:52:25.836952442Z, End: 2024-01-22T22:52:26.433363808Z, Duration: 00:00:00.596
    Load type hierarchy: Begin: 2024-01-22T22:52:25.837114747Z, End: 2024-01-22T22:52:25.919714530Z, Duration: 00:00:00.082
    Load UCFGs: Begin: 2024-01-22T22:52:25.920101840Z, End: 2024-01-22T22:52:26.431289653Z, Duration: 00:00:00.511
  Check cache: Begin: 2024-01-22T22:52:26.433930523Z, End: 2024-01-22T22:52:26.435081353Z, Duration: 00:00:00.001
    Load cache: Begin: 2024-01-22T22:52:26.434092827Z, End: 2024-01-22T22:52:26.434376435Z, Duration: 00:00:00.000
  Create runtime call graph: Begin: 2024-01-22T22:52:26.435359161Z, End: 2024-01-22T22:52:26.575703271Z, Duration: 00:00:00.140
    Variable Type Analysis #1: Begin: 2024-01-22T22:52:26.436168782Z, End: 2024-01-22T22:52:26.517430530Z, Duration: 00:00:00.081
      Create runtime type propagation graph: Begin: 2024-01-22T22:52:26.437440216Z, End: 2024-01-22T22:52:26.487087828Z, Duration: 00:00:00.049
      Run SCC (Tarjan) on 6880 nodes: Begin: 2024-01-22T22:52:26.487767846Z, End: 2024-01-22T22:52:26.496804485Z, Duration: 00:00:00.009
      Propagate runtime types to strongly connected components: Begin: 2024-01-22T22:52:26.497390801Z, End: 2024-01-22T22:52:26.517026820Z, Duration: 00:00:00.019
    Variable Type Analysis #2: Begin: 2024-01-22T22:52:26.519308680Z, End: 2024-01-22T22:52:26.573466012Z, Duration: 00:00:00.054
      Create runtime type propagation graph: Begin: 2024-01-22T22:52:26.519570787Z, End: 2024-01-22T22:52:26.552402055Z, Duration: 00:00:00.032
      Run SCC (Tarjan) on 6807 nodes: Begin: 2024-01-22T22:52:26.552874267Z, End: 2024-01-22T22:52:26.558728122Z, Duration: 00:00:00.005
      Propagate runtime types to strongly connected components: Begin: 2024-01-22T22:52:26.559198034Z, End: 2024-01-22T22:52:26.573172304Z, Duration: 00:00:00.013
  Load config: Begin: 2024-01-22T22:52:26.575991778Z, End: 2024-01-22T22:52:26.687661630Z, Duration: 00:00:00.111
  Compute entry points: Begin: 2024-01-22T22:52:26.688604555Z, End: 2024-01-22T22:52:26.859012760Z, Duration: 00:00:00.170
  Slice call graph: Begin: 2024-01-22T22:52:26.859690778Z, End: 2024-01-22T22:52:26.862082841Z, Duration: 00:00:00.002
  Live variable analysis: Begin: 2024-01-22T22:52:26.862378049Z, End: 2024-01-22T22:52:26.877266242Z, Duration: 00:00:00.014
  Taint analysis for python: Begin: 2024-01-22T22:52:26.877695954Z, End: 2024-01-22T22:52:27.011307086Z, Duration: 00:00:00.133
  Report issues: Begin: 2024-01-22T22:52:27.011594593Z, End: 2024-01-22T22:52:27.014921381Z, Duration: 00:00:00.003
  Store cache: Begin: 2024-01-22T22:52:27.015231989Z, End: 2024-01-22T22:52:27.015426994Z, Duration: 00:00:00.000
INFO: python security sensor peak memory: 179 MB
INFO: Sensor PythonSecuritySensor [security] (done) | time=1185ms
INFO: Sensor JsSecuritySensor [security]
INFO: Enabled taint analysis rules: S5146, S2083, S6287, S5131, S2076, S5144, S5696, S6350, S6105, S5334, S2631, S3649, S5883, S6096, S5147
INFO: Load type hierarchy and UCFGs: Starting
INFO: Load type hierarchy: Starting
INFO: Reading type hierarchy from: /github/workspace/src/apis/finance-api/.scannerwork/ucfg2/js
INFO: Read 0 type definitions
INFO: Load type hierarchy: Time spent was 00:00:00.000
INFO: Load UCFGs: Starting
INFO: Load UCFGs: Time spent was 00:00:00.000
INFO: Load type hierarchy and UCFGs: Time spent was 00:00:00.001
INFO: No UCFGs have been included for analysis.
INFO: js security sensor: Time spent was 00:00:00.002
INFO: Sensor JsSecuritySensor [security] (done) | time=3ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=7ms
INFO: SCM Publisher SCM provider for this project is: git
INFO: SCM Publisher 1 source file to be analyzed
WARN: Shallow clone detected, no blame information will be provided. You can convert to non-shallow with 'git fetch --unshallow'.
INFO: SCM Publisher 0/1 source files have been analyzed (done) | time=6ms
WARN: Missing blame information for the following files:
WARN:   * service/src/app.py
WARN: This may lead to missing/broken features in SonarCloud
INFO: CPD Executor 7 files had no CPD blocks
INFO: CPD Executor Calculating CPD for 24 files
INFO: CPD Executor CPD calculation finished (done) | time=35ms
INFO: SCM writing changed lines
WARN: Could not find ref: main in refs/heads, refs/remotes/upstream or refs/remotes/origin
INFO: SCM writing changed lines (done) | time=4ms
INFO: Analysis report generated in 158ms, dir size=275 KB
INFO: Analysis report compressed in 49ms, zip size=78 KB
INFO: Analysis report uploaded in 517ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: https://sonarcloud.io/dashboard?id=acme_python_finance-api&pullRequest=300
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=AY0zYAaSgp1pf0BXysGu
INFO: Analysis total time: 23.922 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 28.883s
INFO: Final Memory: 37M/130M
INFO: ------------------------------------------------------------------------

This is proving to be a big issue for us as our CI pipelines fail incorrectly and intermittently.

2

Does this mean you have mutiple SonarCloud projects – and have you configured your projects as a monorepo?

3

Hi,

Thanks for your reply. We have configured it as a mono repo as per the instructions.

The issue appears to be intermittent, across many projects.

4

Thanks.

I find this weird to see in the context of a Github actions pipeline – can you share your GitHub actions YML file?

5

Hi Colin,

Thanks for your response. Sure thing - we use a composite action, legacy from the pre-mono repo days; but here it is:

name: Run pull request steps with poetry
description: This action installs poetry, runs tests and sonarqube.

inputs:
  sonar_token:
    description: "token to access sonar-cloud/sonarqube"
    required: true
  sonar_cloud:
    description: "true: use sonar-cloud, false: use sonarqube"
    required: false
    default: "false"
  github_pat:
    description: "github access token"
    required: false
    default: "false"
  python_version:
    description: "version of python to run the tests at"
    required: false
    default: 3.10.3
  test_directory:
    description: "test directory"
    required: false
    default: service/src
  test_exclusion_directory:
    description: "test exclusion directory"
    required: false
    default: tests/e2e
  poetry_version:
    description: "version of poetry being used"
    required: false
    default: 1.6.0
  target_directory:
    description: "test directory to run, defaults to active working directory"
    required: false
    default: .
  working_directory:
    description: "Working Directory"
    required: false
    default: .

runs:
  using: "composite"
  steps:
    - name: Set Up Python ${{ inputs.python_version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ inputs.python_version }}

    - name: Authenticate to Google Cloud To retrieve from Artifact Registry
      uses: google-github-actions/auth@v0.4.3
      with:
        workload_identity_provider: "projects/xxxx/locations/global/workloadIdentityPools/github-actions-identity-pool/providers/github-provider"
        service_account: "svc-python-publisher@acme-dev-tooling.iam.gserviceaccount.com"

    - name: Install and configure Poetry
      uses: snok/install-poetry@v1
      with:
        version: ${{ inputs.poetry_version }}
        virtualenvs-in-project: true

    - name: Load cached venv
      id: cached-poetry-dependencies-mono
      uses: actions/cache@v2
      with:
        path: .venv
        key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}

    - name: Install keyring
      shell: bash
      run: poetry self add "keyrings.google-artifactregistry-auth"

    - name: Install packages
      shell: bash
      run: poetry install --all-extras
      working-directory: ${{ inputs.working_directory}}

    - name: Run Tests & Generate Coverage Report
      working-directory: ${{ inputs.working_directory }}
      shell: bash
      run: |

        echo "RUNNING POETRY TEST STUFF wkdir=${{ inputs.working_directory }} dir=${{ inputs.target_directory }} ignore=${{ inputs.test_exclusion_directory }}"
        poetry run coverage run -m pytest ${{ inputs.target_directory }} --ignore=${{ inputs.test_exclusion_directory }}
        poetry run coverage xml -o '${{ inputs.working_directory }}/service/src/coverage.xml'

    - name: fix code coverage paths
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: |
        echo "RUNNING POETRY SED STUFF"
        ls -R
        sed -i 's@'$GITHUB_WORKSPACE'@/github/workspace/@g' '${{ inputs.working_directory }}/service/src/coverage.xml'
        cp ${{ inputs.working_directory }}/service/src/coverage.xml ${{ inputs.working_directory }}/coverage.xml

    - name: SonarCloud Scan
      uses: SonarSource/sonarcloud-github-action@master
      with:
        projectBaseDir: ${{ inputs.working_directory }}
      env:
        GITHUB_TOKEN: ${{ inputs.github_pat }} # Needed to get PR information, if any
        SONAR_TOKEN: ${{ inputs.sonar_token }}
        args: >
          -Dsonar.organization=acme
          -Dsonar.python.coverage.reportPaths=coverage.xml
          -Dsonar.scm.revision=${{github.event.pull_request.head.sha}}
          -Dsonar.verbose=true
6

Thanks. Where’s the checkout step? is the fetch-depth set to 0?

  - uses: actions/checkout@v3
    with:
      fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis
7

The checkout step is handled via the workflow that calls the composite action. Fetch depth is set to 0

 the_matrix:
    needs: [create-matrix]
    strategy:
      fail-fast: false
      matrix:
        include: ${{ fromJson(needs.create-matrix.outputs.matrix-json) }}
    name: "${{ matrix.service }}"
    runs-on: ubuntu-20.04
    permissions:
      id-token: write
      contents: write
    steps:
      - uses: actions/checkout@v3
        with: 
          fetch-depth: 0

      - name: Get composite action
        uses: actions/checkout@v3
        with:
          repository: acme/poetry-pull-request-mono
          token: ${{secrets.ACTIONS_PAT}}
          path: ./poetry-pull-request-mono

      - name: Run composite action (tests, and sonarqube)
        uses: ./poetry-pull-request-mono
        with:
          github_token: ${{ secrets.CUSTOM_PAT }}
          sonar_token: "${{ secrets.SONAR_TOKEN }}"
          target_directory: "service/src/tests/unit"
          working_directory: "${{ github.workspace }}/src/${{ matrix.service_type }}/${{ matrix.service }}"
8

Thanks! Could you try running git show-ref right before the SonarCloud step?

We really need to get to the bottom of this, so it will be useful to see what refs are available at this point in the job.