Main branch analysis cache not used for PR

[DNavrotskyi (Dmitry Navrotskyi)]

Hello Geoffray,

We are using SonarQube 9.6 for C and C++ projects with server side cache enabled and we use the following New Code for project strategy: New Code is referencing to master branch and master branch is referencing to Previous version.
You mentioned that SonarQube can now cope with changing code location, but in our case SonarQube is not using cache from master when the new Pull Request branch is analysed ( -Dsonar.branch.name parameter is used) What would be our approach to solve this cache issue?
In case we need to move to fs cache (nfs mount), would we able to use it for multiple scanning jobs simultaneously from same location?

[Abbas (Abbas Sabra)]

Hi @DNavrotskyi,

Welcome to the community!

@DNavrotskyi

You mentioned that SonarQube can now cope with changing code location

yes, both filesystem and server cache are independent of the project location.

@DNavrotskyi

but in our case SonarQube is not using cache from master when the new Pull Request branch is analysed

Can you share the full verbose scanner log of that PR?

When you say it is not using the cache do you mean that you aren’t getting any cache hits? or not even loading the cache?
do you see in the log Loading cache from or Found empty cache on?
Was your master branch analyzed with 9.6?

@DNavrotskyi

In case we need to move to fs cache (nfs mount), would we able to use it for multiple scanning jobs simultaneously from same location?

One fs cache shouldn’t be used by multiple analyses at the same time. It is not safe as one analysis might be writing the results while the other is reading the cache.
If you want multiple branches to use the same cache, you should first copy the cache to a unique location per branch. Also, you should decide which branch is supposed to update the fs cache.

Thanks,

[DNavrotskyi (Dmitry Navrotskyi)]

Hi,

I am getting Loading cache from server and cache: 0/452 hits
The master branch was analyzed with 9.6 and on every rebuild I can see that sonar-scanner has cache hits 0, same as every PR. The full log of SonarQube execution step with DEBUG from Jenkins is in attach.
output.zip (7.0 MB)

[Abbas (Abbas Sabra)]

hi @DNavrotskyi,

In the log I can see:

response file not found, request changed

This means that the analysis request changed between the two runs. Analysis request contains the things like the compilation config. Like predefined macros, include directories, and compiler flags. When these things change it is expected that you get a cache miss. Is there an obvious change in the build config between the master branch and the PR? if not I can ask for more info to investigate.

Thanks,

[berezhinskiy (Yaroslav Berezhinskiy)]

Hi Abbas,

There are no changes between the two runs. We have emulated the following scenario:

1. Scan the master branch to fill the cache
2. Create a new branch on top of the master without any new commits
3. Scan new branch

What additional info can we provide to help with the investigation?
Thanks!

[Abbas (Abbas Sabra)]

@berezhinskiy,

@berezhinskiy

What additional info can we provide to help with the investigation?

Generate a reproducer for the same source file in the 2 environments. One from your PR and the other from your master branch.

To generate the reproducer file:

• Search in the analysis log for the full path of the source file for which you want to create a reproducer (for instance, a file that contains a false-positive). You will have to use exactly this name (same case, / or \…)
• Add the reproducer option to the scanner configuration:
  sonar.cfamily.reproducer= “Full path to the .cpp”
• Re-run the scanner to generate a file named sonar-cfamily.reproducer in the project folder.
• Please share this file. If you think this file contains private information, let us know, and we’ll send you a private message that will allow you to send it privately.

in case of an issue in a header file, you want to generate the reproducer of the source file that includes that header.

Thanks,

[DNavrotskyi (Dmitry Navrotskyi)]

Scanner is failing with following error Unable to guess SonarQube task id and/or SQ server details. Please use the ‘withSonarQubeEnv’ wrapper to run your analysis.
So we could analyze only master branch. The reproducer file
sonar-cfamily-reproducer.zip (1.1 MB)
is in attach.
Tried to execute sonar-scanner with such atributes:
sonar-scanner -X
-Dsonar.scm.exclusions.disabled=true
-Dsonar.log.level=DEBUG
-Dsonar.sourceEncoding=UTF-8
-Dsonar.cfamily.reproducer=absolute/path/to/.cpp
-Dsonar.cfamily.analysisCache.mode=server
-Dsonar.cfamily.compile-commands=compile_commands.json
-Dsonar.sources=path/to/src
-Dsonar.projectKey=Project key
-Dsonar.projectName=“Project name”
-Dsonar.branch.name=…
-Dsonar.inclusions=“**/.h, **/.c, **/.cpp, **/.py, **/.hpp, **/.a, **/.so, **/.xml, **/*.json”

[DNavrotskyi (Dmitry Navrotskyi)]

Hi Abbas,

The reproducer is generated from master branch according to every new PR scenario - WORKSPACE directory was changed from DevOps_Manual to DevOpsManual1 . That includes changing of path to compile commands, compiler, etc.
What are possible solutions to use benefits from caching in such scenarios?
sonar-cfamily.zip (996.9 KB)

[Abbas (Abbas Sabra)]

Hi @DNavrotskyi,

I’m sorry I missed your reply.

You provided a reproducer for different files one NssDbUtils.cpp and the other log.cpp.

I cannot say 100% if it is gonna be the case if you provide the reproducers of the same file, but I notice the change in one preprocessor macro between the two: FOLLY_XLOG_STRIP_PREFIXES.

one is defined to "/home/jenkins/agent/workspace/DevOpsManual1/src/siklu-main" and the other to "/home/jenkins/agent/workspace/DevOps_Manual/src/siklu-main"

Can you try to stabilize that and see if you get a cache hit?

Thanks,

[Abbas (Abbas Sabra)]

@DNavrotskyi, any update on this?