Pull Request Decorations Missing in Submodules

Hello,

I am using Bitbucket Cloud to analyze multiple C++ repositories, one of which is the parent/master repo. The rest are submodules within the parent repo. The structure is roughly:

  • master-repo
    • CMakeLists.txt
    • submoduleA
    • submoduleB (depends on A)
    • submoduleC (depends on A and B)

When I run SonarScanner in the master-repo Bitbucket pipeline, pull request decorations show on the master-repo files that have code smells. However, when I run SonarScanner in the submodules, the pull request decorations do not show and the Sonar widget on the PR always shows no new smells, no bugs, etc. I would like to get SonarScanner running properly in the submodules.

The pipeline steps in the submodule are roughly:

  1. Start in submoduleC (let’s call this root/submoduleC)
  2. cd …
  3. git clone master-repo
  4. cd master-repo
  5. init submodules, check out submoduleA, submoduleB, and submoduleC (let’s call this master-repo/submoduleC)
  6. build repo
  7. run SonarScanner

I assume the problem is that the checked out master-repo/submoduleC (step 5) does not link back properly to the same root/submoduleC code (step 1) above. I have tried 1. replacing master-repo/submoduleC with a symlink to root/submoduleC and 2. updating master-repo/CMakeLists.txt to point to root/submoduleC instead of master-repo/submoduleC. Neither of these approaches have worked.

Is there a way to run SonarScanner such that the code smells and pull request decorations show up in a submodule build?

Thank you!

Hi,

You may be running into a bug very similar to SONAR-16080. I can’t give you an ETA on a fix, but as a workaround, can you try setting sonar.scm.exclusions.disabled=true?

 
Ann

Hi Ann,

I updated the setting and pull request decorations still do not show. Here are some excerpts from the pipeline:

11:41:14.359 DEBUG: Init module 'submoduleC
11:41:14.359 DEBUG: Base dir: /opt/atlassian/pipelines/agent/master-repo
11:41:14.359 DEBUG: Working dir: /opt/atlassian/pipelines/agent/master-repo/.scannerwork
11:41:14.359 DEBUG: Module global encoding: UTF-8, default locale: en

11:41:19.839 INFO: Included sources: **/submoduleC/**
11:41:19.839 INFO: Excluded sources: **/build-wrapper-dump.json
11:41:19.839 INFO: Excluded sources for coverage: **/src/**/test/**/*.*, **/src/**/req_tests/**/*.*, **/src/**/unittests/**/*.*, **/src/**/unit_tests/**/*.*

11:41:19.882 DEBUG: ‘src/submoduleC/src/main.cpp’ indexed with language ‘cpp’

11:41:19.991 INFO: ------------- Run sensors on module submoduleC

11:41:23.584 DEBUG: ‘src/submoduleC/bitbucket-pipelines.yml’ generated metadata with charset ‘UTF-8’

11:41:23.767 DEBUG: Project ‘submoduleC’: Property missing: ‘sonar.cs.analyzer.projectOutPaths’. No protobuf files will be loaded for this project.
11:41:23.767 DEBUG: Project 'submoduleC': Property missing: 'sonar.cs.analyzer.projectOutPaths'. No protobuf files will be loaded for this project.
11:41:23.768 DEBUG: Project 'submoduleC': No Roslyn issues reports have been found.

11:44:18.081 DEBUG: Detection of duplications for /opt/atlassian/pipelines/agent/master-repo/src/submoduleC/src/main.cpp

11:44:18.832 DEBUG: POST 200 https://sonarcloud.io/api/ce/submit?organization=projectName&projectKey=submoduleC&characteristic=pullRequest%3D14 | time=438ms
11:44:18.836 INFO: Analysis report uploaded in 442ms
11:44:18.841 DEBUG: Report metadata written to /opt/atlassian/pipelines/agent/master-repo/.scannerwork/report-task.txt
11:44:18.841 INFO: ANALYSIS SUCCESSFUL, you can find the results at: https://sonarcloud.io/dashboard?id=submoduleC&pullRequest=14

Hi,

Have you ‘Enable(d) support for monorepo’ on this project in SonarCloud?

 
Ann

Hi Ann,

I enabled support for monorepo in the settings on SonarCloud.io and didn’t see any changes. The website also says that this setting only works for Azure DevOps.

Thanks,
Dorothy

Hi Dorothy,

Sorry for the confusion. I’ve just learned that you can’t turn a repo into a monorepo with after-the-fact configuration. It has to be configured as a monorepo on initial import. (When you go to create a project, notice the “Setup a monorepo.” option in the lower right).

It looks like you may need to drop this and re-import.

 
Ann

Hi Ann,

Interested! I deleted submoduleC and recreated it as a mon-repo with support for mono-repo enabled. I also updated the SonarScanner version from 4.4.0.2170 to 4.7.0.2747 and cleared the pipeline caches for submoduleC. I’m seeing the same results - no new bugs or smells.

Hi,

I want to back up

You’ve started in an existing submodC directory, cdd up a level(?) into master-repo, then cloned master-repo… into master-repo?

You then cd into the newly-cloned master-repo and run the full build of all modules(?), before running analysis.

What parameters are passed into that analysis?

Also, if I’m correct that there’s one build for all the code, why try to handle the submodules separately, rather than as part of the project?

 
Ann

Hi Ann,

This is in a Bitbucket Cloud pipeline for the submoduleC repository, not the master-repo repository. submoduleC needs submoduleA and submoduleB to build, which are defined in master-repo. In the repository for master-repo, there is a .gitmodules file that points to submodules A-Z, let’s say. We have repositories and therefore pipelines for all submodules, as well as the parent/master repository. We build in the submodule pipelines because the code in submoduleC doesn’t effect submoduleD (and vice-versa), although both submoduleC and submoduleD depend on submoduleA and submoduleB. I hope that makes sense.

To rephrase my initial comment, the SonarCloud analysis and decorations in master-repo are working fine. However, the analysis in any submodule shows that there are no code smells, no bugs, no errors at all. Even when they are deliberately introduced, the submodule SonarCloud analysis shows no errors.

In the pipeline for submoduleC, we clone master-repo for two reasons. The first is that it contains the .gitmodules file that points to the correct submodules. This makes it easier to git init the submodule dependencies (step 5 in my initial topic). The second reason is that master-repo contains helpful scripts for our build, including our build script and our script to run SonarScanner, rather than duplicating the scripts in each submodule.

Here is the build command we use (step 6):

export BW_OUTPUT=$HOME/.sonar/bw-output
- build-wrapper-linux-x86-64 --out-dir $BW_OUTPUT master-repo/scripts/build.py -arg1 -arg2

Here are the sonar scanner commands we use (step 7):

export SONAR_SCANNER_VERSION=4.4.0.2170
export SONAR_SCANNER_OPTS="-server"
export SONAR_SCANNER_HOME=$HOME/.sonar/sonar-scanner-$SONAR_SCANNER_VERSION-linux
export PATH=$SONAR_SCANNER_HOME/bin:$HOME/.sonar/build-wrapper-linux-x86:$PATH
sonar-scanner -Dsonar.scm.exclusions.disabled=true \
              -Dsonar.cfamily.build-wrapper-output=$BW_OUTPUT \
              -Dsonar.coverageReportPaths=./coverage.xml \
              -Dsonar.cfamily.threads=4 \
              -Dsonar.cfamily.cache.enabled=true \
              -Dsonar.cfamily.cache.path=$cache_dir \
              -Dsonar.coverage.exclusions=**/src/**/test/**/*.*,**/src/**/req_tests/**/*.*,**/src/**/unittests/**/*.*,**/src/**/unit_tests/**/*.* \
              -Dsonar.python.version=3 \
              -X

As an aside, I’ve tried several combinations of cloning strategies within the submoduleC pipeline, including cloning master-repo inside submoduleC and symlinking root/submoduleC/master-repo/submoduleC to root/submoduleC. I haven’t had any luck with any combination of these strategies, but I’m open to suggestions!

Hi,

Could you post your full analysis log? Also, could you check our build-wrapper output to make sure it contains the files in your submodule?

 
Ann

Hi Ann,

I attached the pipeline output from the analysis portion of the pipeline. I’m not sure what you mean by the build-wrapper output - where would I find that?

Thanks,
Dorothy

pipelineLog-{52af0f07-865f-4536-8792-064fbfcb1b6d}.txt (6.1 MB)

Hi,

You’ll find it wherever $BW_OUTPUT resolves to.

 
:smiley:
Anb

Hi Ann,

There were two fines in $BW_OUTPUT, build-wrapper.log and build-wrapper-dump.json . Their contents are attached.

Best,
Dorothy
pipelineLog-{b0b4bda5-73ab-4142-a6c6-920c11ec5970}-modified.txt (3.3 MB)

Thanks Dorothy!

I’ve flagged this for more expert attention.

 
Ann

1 Like

Hi Ann,

Hope you’re well. Have you heard anything more on this?

Thanks,
Dorothy

Hi Dorothy,

Sorry for the delay in responding. I was off for a few days. This has been dragging for over a month & you’ve been very patient, which I really appreciate.

Behind the scenes, I got one of our C/C++ experts to take a look & he didn’t see anything on his side.

So I stepped back and started over from your first post. What wasn’t immediately obvious to me when I first came to this thread is that you’re doing multiple checkouts. And I’m thinking now that’s the heart of the problem.

Each SonarCloud project is associated with a source repo. And at PR decoration time, SonarCloud knows where to do the decoration from that association. But your submodules don’t exist there. I think that’s why decoration is failing for your submodules. It’s a mis-match between the basic expectations.

Does that make sense?

 
Ann

Hi Ann,

I believe I do understand! The directory structure I’m creating in the pipeline is basically:

  • root
    • submoduleC ← where I want SonarCloud to look
    • master-repo
      • submoduleA
      • submoduleB
      • submoduleC ← where SonarCloud is looking

I’ve tried to work around this by replacing root/master-repo/submoduleC with a symlink to root/submoduleC, but that doesn’t seem to work. Is there a way to redirect SonarCloud to root/submoduleC?

Hi,

You still face the problem that you’re analyzing master-repo, so that’s where SonarCloud is going to go to decorate the PR.

The best way I can see to do this - no idea if it’s feasible in your situation - is to have an entirely different SonarCloud project for submodule C and to start by checking out submodule C and building / analyzing it (with any additional checkouts as necessary/feasible). And the same for the other submodules.

 
:woman_shrugging:
Ann

Hi Ann,

Because submoduleC has dependencies in other modules, it can’t build on its own. I changed the pipeline to check out and build the following structure:

  • submoduleC (pwd is /opt/atlassian/pipelines/agent/build)
    • src
    • include
    • CMakeLists.txt
    • master-repo
      • CMakeLists.txt ← master-repo’s
      • src
        • submoduleA
        • submoduleB

master-repo’s CMake includes the following lines:

add_subdirectory(submoduleA)
add_subdirectory(submoduleB)
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}../../../build ${CMAKE_CURRENT_BINARY_DIR}/submoduleC)
^ This last line adds a relative path to submoduleC’s src directory (/opt/atlassian/pipelines/agent/build) and sets the directory for the build output to be the same as the other submodules.

Each module builds correctly, but there are still no pull request decorations, nor new bugs or smells in the Widget. Funnily enough, the widgets in other submodules have started populating with updated coverage, smells, bugs, etc., but are still lacking the PR decorations.

Hi Ann,

Do you have any insight on this latest update?

Thanks,
Dorothy