Pull Request Decorations Missing in Submodules

sns-seb · June 17, 2022, 8:58am

You are reporting missing Pull Request decorations.
I realize your SonarCloud setup is not described on this topic.

For the sake of removing assumptions, can you confirm you setup your organization and projects following the process describe on that page?

Could you then describe how many project(s) exist on SonarCloud which relate to the master-repo and the 3 submodules and which BBC repo they are bound to?
How many pipelines are configured to use these projects?
Does the bitbucket-pipelines.yml differ from the one recommended in the C/C++ pipeline setup tutorial and how (for what relates to sonarcloud configuration)?

Thanks

dorothyk · June 22, 2022, 4:36pm

Hi Sébastien,

Yes, the projects were all set up following SonarCloud’s instructions.

We have one parent/master repo and 12 separate child/submodule repositories. Each of these 13 repositories has their own pipeline. The master repo PRs show the pull request decorations but the 12 submodules do not.

The submodule pipelines have identical bitbucket-pipelines.yml files. Here are the relevant excerpts from a submodule’s .yml. It does not include any changes that I have attempted above.

image:
  name: $DOCKER_IMAGE_NAME
  username: $DOCKER_HUB_USERNAME
  password: $DOCKER_HUB_PASSWORD
  email: $DOCKER_HUB_EMAIL

clone:
  depth: full              # SonarCloud scanner needs the full history to assign issues properly

definitions:
  caches:
    sonar: ~/.sonar  # Caching SonarCloud artifacts will speed up your build
    sonar-master: ~/.sonar_results/master   # Cache SonarCloud results for incremental analysis
    sonar-pr: ~/.sonar_results/pull_request # Cache SonarCloud results for incremental analysis
  steps:
    - step: &build-test-sonarcloud
        name: Build and analyze necessary code for this submodule's build
        size: 2x
        caches:
          - sonar
          - sonar-master
          - sonar-pr
        script:
          # save name of submodule name for generic script
          - submoduleName=$(echo ${BITBUCKET_REPO_FULL_NAME#*/})

          # check out parent repo at depth 1 https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt
          - cd ..
          - git clone git@bitbucket.org:armor1sia/master-repo.git --depth 1 --no-single-branch
          - cd master-repo
          - git checkout $BITBUCKET_BRANCH || git checkout master

          # Convert HTTPS URLs to SSH so Pipelines can properly clone submodules
          - sed -i 's/https:\/\/bitbucket.org\//git@bitbucket.org:/g' .gitmodules

          # check out submoduleName and required dependencies only - keeps it lightweight
          # this checks out submodules at  /opt/atlassian/pipelines/agent/master-repo/<submodule-name>
          - ./scripts/pipeline/checkoutDependencies.sh ${submoduleName} $BITBUCKET_BRANCH

          # Install SonarCloud via script - skips if already installed.  Copied below
          - source ./scripts/pipeline/installSonar.sh

          # Run build wrapper with our build script - 4 threads, use build directory, enable testing
          - build-wrapper-linux-x86-64 --out-dir $BW_OUTPUT scripts/build.py -j4 -bt Release -t ${submoduleName}_pipeline -e
          # Specify to use C++17 in Sonar Scanner (default is 20 or 2a, which Sonar does not support)
          - sed -i 's/c++2a/c++17/g' $BW_OUTPUT/build-wrapper-dump.json

          # Run SonarCloud analysis
          - ./scripts/pipeline/runSonarScanner.sh

# More info here: https://confluence.atlassian.com/bitbucket/configure-bitbucket-pipelines-yml-792298910.html
pipelines:
  branches:
    master:
      - step: *build-test-sonarcloud
  pull-requests:
    '**':
      - step: *build-test-sonarcloud

Here is the intallSonar.sh script:

# Source variables to be used in pipeline build
export BW_OUTPUT=$HOME/.sonar/bw-output
export SONAR_SCANNER_VERSION=4.4.0.2170

DIR="$HOME/.sonar/sonar-scanner.zip"
if [[ -e $DIR ]];
then
   echo "Sonar directory $DIR exists, skipping install"
else
   echo "Sonar directory $DIR does not exist, installing" >&2
   mkdir -p $BW_OUTPUT
   curl --create-dirs -sSLo $HOME/.sonar/sonar-scanner.zip https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-$SONAR_SCANNER_VERSION-linux.zip
   unzip -o -qq $HOME/.sonar/sonar-scanner.zip -d $HOME/.sonar/

   curl --create-dirs -sSLo $HOME/.sonar/build-wrapper-linux-x86.zip https://sonarcloud.io/static/cpp/build-wrapper-linux-x86.zip
   unzip -o -qq $HOME/.sonar/build-wrapper-linux-x86.zip -d $HOME/.sonar/
fi

# Source variables to be used in pipeline build
export SONAR_SCANNER_OPTS="-server"
export SONAR_SCANNER_HOME=$HOME/.sonar/sonar-scanner-$SONAR_SCANNER_VERSION-linux
export PATH=$SONAR_SCANNER_HOME/bin:$HOME/.sonar/build-wrapper-linux-x86:$PATH

Here is the runSonarScanner.sh script:

#!/bin/bash
# Using "/root/" because SonarCloud has trouble with "~/" (they are the same to bitbucket pipelines)
cache_master="/root/.sonar_results/master"
cache_pr="/root/.sonar_results/pull_request"

# Get name of current repository
repo_name=$(echo ${BITBUCKET_REPO_FULL_NAME#*/})

##################################
## Determine which cache to use ##
##################################
echo "Attempt: Determining which cached SonarCloud results to use"
if [ $BITBUCKET_BRANCH = "master" ];
then
    echo "On master, checking if master cache exists"
    if ls -1qA $cache_master | grep -q .
    then  
        echo "Master-cache is not empty, using master-cache"
    else 
        echo "Master-cache is empty, expect longer processing time"
    fi
    cache_dir=$cache_master
else
    echo "Not on master, checking if pr-cache exists"
    if ls -1qA $cache_pr | grep -q .
    then  
        echo "Pr-cache is not empty, using pr-cache"
    else 
        echo "Pr-cache is empty, checking if master has existing cache"
        if ls -1qA $cache_master | grep -q .
        then  
            echo "Master-cache is not empty, copying contents of master-cache to pr-cache"
            cp -r $cache_master $cache_pr
        else 
            echo "Master-cache is empty, expect longer processing time"
        fi
    fi
    cache_dir=$cache_pr
fi
echo "Success: Using cache $cache_dir"

###########################################
## Make and format the coverage document ##
###########################################

if [ $repo_name = "master-repo" ];
then
    # master-repo builds debug for regression testing 
    build_dir="build_debug"
else
    # submodules build release on purpose
    build_dir="build_release"
fi

cd $build_dir
make coverage
cd ..
xmllint --format coverage.xml --output coverage.xml
sed -i '1d' coverage.xml

#######################
## Run Sonar-Scanner ##
#######################

# Notes: (running on ms_full)
# - Threaded Analysis (8=fail, 6=29min, 4=24min, 1=70min)
# - Caching Results (no files change = 1min, all files change = 24min )

# Sonar-generated command to run scanner based off of build-wrapper output
# Arguments:
	# exclusions.disabled             - include submodules in analysis
	# build-wrapper-output=$BW_OUTPUT - point to build wrapper output (from build step)
	# coverageReportPaths             - include coverage report of unit testing
    # cfamily.threads                 - the number of threads the sonar analysis uses
    # cfamily.cache.enabled           - enable/disable the use of caching
    # cfamily.cache.path              - path to the cache directory
    # coverage.exclusions             - prevent files from counting towards code coverage
    # python.version                  - specify python version
	# -X                              - enable debug logging
sonar-scanner -Dsonar.scm.exclusions.disabled \
              -Dsonar.cfamily.build-wrapper-output=$BW_OUTPUT \
              -Dsonar.coverageReportPaths=./coverage.xml \
              -Dsonar.cfamily.threads=4 \
              -Dsonar.cfamily.cache.enabled=true \
              -Dsonar.cfamily.cache.path=$cache_dir \
              -Dsonar.coverage.exclusions=**/src/**/test/**/*.*,**/src/**/req_tests/**/*.*,**/src/**/unittests/**/*.*,**/src/**/unit_tests/**/*.* \
              -Dsonar.python.version=3 \
              -X

Best,
Dorothy

sns-seb · June 24, 2022, 8:03am

Hello @dorothyk ,

Does each of these repo have their own SonarCloud project?
Do you see the relevant analysis results in each SonarCloud project (are the files in each project, the files of the associated submodule)?

I’m investigating assuming you’ll answer positively to the above questions and will keep you posted.

Seb’