Pull Request Decorations Missing in Submodules

[sns-seb (Sébastien Lesaint)]

Hello @dorothyk ,

You are reporting missing Pull Request decorations.
I realize your SonarCloud setup is not described on this topic.

For the sake of removing assumptions, can you confirm you setup your organization and projects following the process describe on that page?

Could you then describe how many project(s) exist on SonarCloud which relate to the master-repo and the 3 submodules and which BBC repo they are bound to?
How many pipelines are configured to use these projects?
Does the bitbucket-pipelines.yml differ from the one recommended in the C/C++ pipeline setup tutorial and how (for what relates to sonarcloud configuration)?

Thanks

[dorothyk (Dorothy)]

Hi Sébastien,

Yes, the projects were all set up following SonarCloud’s instructions.

We have one parent/master repo and 12 separate child/submodule repositories. Each of these 13 repositories has their own pipeline. The master repo PRs show the pull request decorations but the 12 submodules do not.

The submodule pipelines have identical bitbucket-pipelines.yml files. Here are the relevant excerpts from a submodule’s .yml. It does not include any changes that I have attempted above.

image:
  name: $DOCKER_IMAGE_NAME
  username: $DOCKER_HUB_USERNAME
  password: $DOCKER_HUB_PASSWORD
  email: $DOCKER_HUB_EMAIL

clone:
  depth: full              # SonarCloud scanner needs the full history to assign issues properly

definitions:
  caches:
    sonar: ~/.sonar  # Caching SonarCloud artifacts will speed up your build
    sonar-master: ~/.sonar_results/master   # Cache SonarCloud results for incremental analysis
    sonar-pr: ~/.sonar_results/pull_request # Cache SonarCloud results for incremental analysis
  steps:
    - step: &build-test-sonarcloud
        name: Build and analyze necessary code for this submodule's build
        size: 2x
        caches:
          - sonar
          - sonar-master
          - sonar-pr
        script:
          # save name of submodule name for generic script
          - submoduleName=$(echo ${BITBUCKET_REPO_FULL_NAME#*/})

          # check out parent repo at depth 1 https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt
          - cd ..
          - git clone git@bitbucket.org:armor1sia/master-repo.git --depth 1 --no-single-branch
          - cd master-repo
          - git checkout $BITBUCKET_BRANCH || git checkout master

          # Convert HTTPS URLs to SSH so Pipelines can properly clone submodules
          - sed -i 's/https:\/\/bitbucket.org\//git@bitbucket.org:/g' .gitmodules

          # check out submoduleName and required dependencies only - keeps it lightweight
          # this checks out submodules at  /opt/atlassian/pipelines/agent/master-repo/<submodule-name>
          - ./scripts/pipeline/checkoutDependencies.sh ${submoduleName} $BITBUCKET_BRANCH

          # Install SonarCloud via script - skips if already installed.  Copied below
          - source ./scripts/pipeline/installSonar.sh

          # Run build wrapper with our build script - 4 threads, use build directory, enable testing
          - build-wrapper-linux-x86-64 --out-dir $BW_OUTPUT scripts/build.py -j4 -bt Release -t ${submoduleName}_pipeline -e
          # Specify to use C++17 in Sonar Scanner (default is 20 or 2a, which Sonar does not support)
          - sed -i 's/c++2a/c++17/g' $BW_OUTPUT/build-wrapper-dump.json

          # Run SonarCloud analysis
          - ./scripts/pipeline/runSonarScanner.sh

# More info here: https://confluence.atlassian.com/bitbucket/configure-bitbucket-pipelines-yml-792298910.html
pipelines:
  branches:
    master:
      - step: *build-test-sonarcloud
  pull-requests:
    '**':
      - step: *build-test-sonarcloud

Here is the intallSonar.sh script:

# Source variables to be used in pipeline build
export BW_OUTPUT=$HOME/.sonar/bw-output
export SONAR_SCANNER_VERSION=4.4.0.2170

DIR="$HOME/.sonar/sonar-scanner.zip"
if [[ -e $DIR ]];
then
   echo "Sonar directory $DIR exists, skipping install"
else
   echo "Sonar directory $DIR does not exist, installing" >&2
   mkdir -p $BW_OUTPUT
   curl --create-dirs -sSLo $HOME/.sonar/sonar-scanner.zip https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-$SONAR_SCANNER_VERSION-linux.zip
   unzip -o -qq $HOME/.sonar/sonar-scanner.zip -d $HOME/.sonar/

   curl --create-dirs -sSLo $HOME/.sonar/build-wrapper-linux-x86.zip https://sonarcloud.io/static/cpp/build-wrapper-linux-x86.zip
   unzip -o -qq $HOME/.sonar/build-wrapper-linux-x86.zip -d $HOME/.sonar/
fi

# Source variables to be used in pipeline build
export SONAR_SCANNER_OPTS="-server"
export SONAR_SCANNER_HOME=$HOME/.sonar/sonar-scanner-$SONAR_SCANNER_VERSION-linux
export PATH=$SONAR_SCANNER_HOME/bin:$HOME/.sonar/build-wrapper-linux-x86:$PATH

Here is the runSonarScanner.sh script:

#!/bin/bash
# Using "/root/" because SonarCloud has trouble with "~/" (they are the same to bitbucket pipelines)
cache_master="/root/.sonar_results/master"
cache_pr="/root/.sonar_results/pull_request"

# Get name of current repository
repo_name=$(echo ${BITBUCKET_REPO_FULL_NAME#*/})

##################################
## Determine which cache to use ##
##################################
echo "Attempt: Determining which cached SonarCloud results to use"
if [ $BITBUCKET_BRANCH = "master" ];
then
    echo "On master, checking if master cache exists"
    if ls -1qA $cache_master | grep -q .
    then  
        echo "Master-cache is not empty, using master-cache"
    else 
        echo "Master-cache is empty, expect longer processing time"
    fi
    cache_dir=$cache_master
else
    echo "Not on master, checking if pr-cache exists"
    if ls -1qA $cache_pr | grep -q .
    then  
        echo "Pr-cache is not empty, using pr-cache"
    else 
        echo "Pr-cache is empty, checking if master has existing cache"
        if ls -1qA $cache_master | grep -q .
        then  
            echo "Master-cache is not empty, copying contents of master-cache to pr-cache"
            cp -r $cache_master $cache_pr
        else 
            echo "Master-cache is empty, expect longer processing time"
        fi
    fi
    cache_dir=$cache_pr
fi
echo "Success: Using cache $cache_dir"

###########################################
## Make and format the coverage document ##
###########################################

if [ $repo_name = "master-repo" ];
then
    # master-repo builds debug for regression testing 
    build_dir="build_debug"
else
    # submodules build release on purpose
    build_dir="build_release"
fi

cd $build_dir
make coverage
cd ..
xmllint --format coverage.xml --output coverage.xml
sed -i '1d' coverage.xml

#######################
## Run Sonar-Scanner ##
#######################

# Notes: (running on ms_full)
# - Threaded Analysis (8=fail, 6=29min, 4=24min, 1=70min)
# - Caching Results (no files change = 1min, all files change = 24min )

# Sonar-generated command to run scanner based off of build-wrapper output
# Arguments:
	# exclusions.disabled             - include submodules in analysis
	# build-wrapper-output=$BW_OUTPUT - point to build wrapper output (from build step)
	# coverageReportPaths             - include coverage report of unit testing
    # cfamily.threads                 - the number of threads the sonar analysis uses
    # cfamily.cache.enabled           - enable/disable the use of caching
    # cfamily.cache.path              - path to the cache directory
    # coverage.exclusions             - prevent files from counting towards code coverage
    # python.version                  - specify python version
	# -X                              - enable debug logging
sonar-scanner -Dsonar.scm.exclusions.disabled \
              -Dsonar.cfamily.build-wrapper-output=$BW_OUTPUT \
              -Dsonar.coverageReportPaths=./coverage.xml \
              -Dsonar.cfamily.threads=4 \
              -Dsonar.cfamily.cache.enabled=true \
              -Dsonar.cfamily.cache.path=$cache_dir \
              -Dsonar.coverage.exclusions=**/src/**/test/**/*.*,**/src/**/req_tests/**/*.*,**/src/**/unittests/**/*.*,**/src/**/unit_tests/**/*.* \
              -Dsonar.python.version=3 \
              -X

Best,
Dorothy

[sns-seb (Sébastien Lesaint)]

Hello @dorothyk ,

@dorothyk

We have one parent/master repo and 12 separate child/submodule repositories. Each of these 13 repositories has their own pipeline. The master repo PRs show the pull request decorations but the 12 submodules do not.

Does each of these repo have their own SonarCloud project?
Do you see the relevant analysis results in each SonarCloud project (are the files in each project, the files of the associated submodule)?

I’m investigating assuming you’ll answer positively to the above questions and will keep you posted.

Seb’

[dorothyk (Dorothy)]

Hi Seb,

Yes, each submodule has its own SonarCloud project.
Yes, on SonarCloud, each submodule has a report of Bugs, Code Smells, etc. The branches and pull requests of that submodule are listed. Within the “Code” page of a submodule, it lists the folders build_release, scripts, and src. “src” contains the submodule and the submodule it depends on.

Thank you,
Dorothy

[sns-seb (Sébastien Lesaint)]

Hello @dorothyk ,

I looked a bit and there may be some hints of what’s wrong to get from the UI.

On the SonarCloud of a submodule, where PR decoration is not working, could have you a look at the background tasks of Pull Requests (in Administration > Background Tasks) and checks if there are any warnings?

[dorothyk (Dorothy)]

Hi @sns-seb ,

I spot-checked a few and they have the same warnings:

Thanks,
Dorothy

[sns-seb (Sébastien Lesaint)]

Hello @dorothyk ,

I expected a warning to provide a clue on the reason the PR decoration was not happening.
Unfortunately, none of these warnings relate to Pull Request decoration.

I would have to dig deeper in the code and the configuration which allow PR decoration on Bitbucket Cloud. Conditions I had identified would have lead to a warning.

I will be off next week and, due to the Holiday period, it’s not easy to find someone else to step in to help you.

I hope it’s ok with you.

[sns-seb (Sébastien Lesaint)]

Hello @dorothyk ,

I’ve tried to reproduce a situation where a SonarCloud project was correctly configured for a Bitbucket repository, being analyzed with a BBC pipeline (ie. issue and source code correctly showing up on sonarcloud.io) and would not have quality information showing on Bitbucket side, neither on the project nor on pull requests.

I did hit this situation when I setup my Bitbucket repository. I had not enabled the SonarCloud integration on my Bitbucket project yet.

Before I go any further: could it be you are in this situation as well? Have you also configured the modules’ projects according to this page?

[dorothyk (Dorothy)]

Hi @sns-seb,

Welcome back!

All modules (both parent and child) are set up and configured with SonarCloud. The code quality widget shows at the top and there’s a link to a report on SonarCloud on the right. However, the decorations are not showing up and the code smells linked in the widget are for the dependencies (eg. submoduleA and submoduleB) not for the submodule the scan was run on (submoduleC).

Best,
Dorothy

[sns-seb (Sébastien Lesaint)]

Hi @dorothyk ,

Let me try and summarize my understanding of your situation and what you try and want to achieve.

You have a C++ program made of multiple pieces of C++ code, let’s call them modules.
Each module’s code live in a separated git repository on Bitucket Cloud (BBC), therefore each module has its own BBC project.
Some modules depend on others to be built.

To ease the build of the dependent modules, you’ve created a master Git repository. This master repository puts all modules together, thanks to Git submodule feature I assume, as subdirectories (one per module).

(It’s not clear to me whether the master repository contains any C++ code of its own or not)

You would like to have the static analysis results from SonarCloud of each module reported on their respective’s BBC project: as a repository overview, a pull request view and a Code Insights report.
All three should report only the issues of the current module.

To this purpose, you have:

• a dedicated SonarCloud project for the BBC project of each module
• a SonarCloud project for the master repository
• each SonarCloud project was created on SonarCloud following the SonarCloud instructions
• SonarCloud integration was enabled on each BBC project
• each BBC project has its own BBC pipeline to build the module and run the SonarCloud analysis
• the pipelines of the modules are identical and proceeds as follow
  • create a shallow clone of the master repository in directory master-repo and pull the gitsubmodule of the modules the current module depends on
  • download the build-wrapper SonarCloud scanner
  • build and analyse the project by running the build-wrapper in the directory master-repo

You report that:

1. analysis results on BBC projects of modules are showing issues for other modules
2. analysis results on BBC projects of modules do not have annotations
3. analysis results on BBC project of master repo is showing issues for all modules and has annotations for all of them

Please let me know if anything of the above is incorrect or lack precision relevant to the following.

I think the root of the situation you’re in is that you are not building nor analyzing in the directory of the module but in a subdirectory: master-repo.

As a consequence all paths known to SonarCloud are relative to master-repo.
A file in module C has a path submodulec/foo/bar.cpp while the path to this file in the BBC project and git repository of module C is foo/bar.cpp.
This prevents the annotations to be rendered: they are likely correctly pushed by SonarCloud to BBC but since they don’t match the path of the files in the BBC project, nothing renders.

To prevent the analysis results of other modules to appear, you could configure source file exclusions on each module’s SonarCloud project (see doc).
Mind it, they are relative to the directory where the build-wrapper runs. so, Eg. for moduleC, exclusions could be submoduleA/**/*,submoduleB/**/*.
(disclaimer: I don’t have extensive practice with exclusions, my example may be wrong but I hope you get the idea)

In general, the build-wrapper should run in the root directory of the repository to be analyzed.

Assuming I understood your situation correctly, I can make two suggestions:

1. change your build script so that it runs in the root directory.
   It would have to support source code of the current module being at the root of the repository and source code of dependencies being in some subdirectories (the depth should not matter, so both master-repo/submoduleA and submoduleA should work). These subdirectories will have to be excluded from analysis
2. switch to a mono repo setup.
   All SonarCloud projects are linked to the master-repo repository and BBC project (see doc on how to set them up). You will have to configure multiple pipelines on the BBC project, one for each module.
   This might have other consequences and its own complexity on top of not necessarily providing the integration rendering on BBC side you are expecting. But that would keep you build structure close to what you have today (one subdirectory per module).

I hope this helps

Cheers