So I am scanning a Java pull request on GitHub with AWS CodeBuild for CI using sonar-scanner-4.4.0.2170-linux and it is not publishing analysis on new lines to SonarCloud PR. Scanner command used:
If you go to the Code tab of the pull request on SonarCloud, do you see there all the files that were changed in the pull request that are not in the base branch? In other words, is that consistent with what you would see with the command:
Thank you for replying to my post. Iâm expecting 4 files with additions, I also ran the command in your comment on CodeBuild and saw the 4 files as expected but there are no files in the Code tab of the pull request on SonarCloud. Do you think this is an issue with the state of the local repo in CodeBuild?
This means that the files are excluded from the analysis. In your scanner command I see sonar.sources=src. Are the changed files under src of the project? (More details in whatâs included and excluded on the Narrowing the Focus page of our docs.)
@janos - the source files are not excluded from the analysis. The scan output logs suggest 137 files were analysed. Another user contacted me about having exactly the same issue.
INFO: 137 source files to be analyzed
INFO: 82/137 files analyzed, current file: src/main/java/com/theaa/dip/motor/pmid/dto/vo/MainVO.java
INFO: 115/137 files analyzed, current file: src/main/java/com/theaa/dip/motor/pmid/dto/DriverDTO.java
INFO: 137/137 source files have been analyzed
The most common reasons for not seeing changed files in the pull request:
Incorrect values for sonar.pullrequest.base, sonar.pullrequest.key
The branch specified by sonar.pullrequest.base cannot be found: in this case you would see a warning about this in the scanner output and also on the UI of the PR (a small box with yellow background in the top-right corner)
Not enough commit history is fetched (shallow clone), so the scanner cannot find the common ancestor commit of the base branch and the PRâs branch
Please verify the above, as the solution is most likely there.
As I explained in a previous comment, it would be good to verify the list of changed files that you can expect to be included in the analysis output, by executing the appropriate Git command:
If all your parameters look correct, there are no related warnings in the UI, the git command includes the files you expect but they donât show up in the Code tab, then you might have configuration of exclusions in the SonarCloud UI. A good way to see the effective scanner parameters is on the Background Tasks page, in the Scanner Context option of the task item.
Iâm the other user Taiwo mentioned with the same issue. Thanks for your response detailing the common reasons. I have a similar set up but with two slightly different branch names and change set. Base branch is develop and PR branch is called sonar-scan.
I have attached 2 images 1 of a scan performed from my local machine, second from the CI machine (which is an AWS Codebuild agent)
I have double checked sonar.pullrequest.base, it is set to develop which is the desired branch name. I have this set on local scans and on the CI scans taking place on AWS Codebuild. What would be an incorrect value for the sonar.pullrequest.key currently we set it to the PR number coming from Github e.g 6? I have also experimented hardcoding it to a unique string - âtestKey2â and âtestKey4â in the images supplied (both from a local and a CI build) but doesnât seem to impact the result. Again when executing locally it works but on CI no impact.
I have added a git status command on the CI build script to ensure we can see the develop branch on CI. Itâs there.
We didnât have a deep clone set up but now we have enabled a full clone but doesnât resolve the issue.
In addition to the above I have also executed the git diff command you messaged from Aug 20 on the CI machine. I can see the desired files we want in the diff but still no luck with the files showing in the
Code tab.
PersonServiceTest added to generate some coverage files which you can see successfully getting picked up and one modification to a source file GroupArtifactApplication to force a code smell.
I have looked into the SonarScanner Context in Background Tasks and we donât have any exclusions set up at the Global or the Project level scopes. One thing I do find odd is how we need to set -Dsonar.scm.exclusions.disabled=true, from local it doesnât make a difference but on the CI machine it seems to exclude all the files if we donât have this set which isnât right as the files are 100% indexed in Git.
I can look to create a sample project I can share source code with you if the above doesnât help resolve the issue.
On further investigation on the AWS Codebuild side of things. I narrowed it down to the build agent cache settings. We currently have the following defined in Cloudformation config that creates our Codebuild project:
... more config
Cache:
Type: LOCAL
Modes:
- LOCAL_SOURCE_CACHE
- LOCAL_CUSTOM_CACHE
... more config
The docs state:
LOCAL_SOURCE_CACHE
Caches Git metadata for primary and secondary sources. After the cache is created, subsequent builds pull only the change between commits. This mode is a good choice for projects with a clean working directory and a source that is a large Git repository. If you choose this option and your project does not use a Git repository (GitHub, GitHub Enterprise, or Bitbucket), the option is ignored.
LOCAL_CUSTOM_CACHE
Caches directories you specify in the buildspec file. This mode is a good choice if your build scenario is not suited to one of the other three local cache modes. If you use a custom cache:
Only directories can be specified for caching. You cannot specify individual files.
Symlinks are used to reference cached directories.
Cached directories are linked to your build before it downloads its project sources. Cached items are overridden if a source item has the same name. Directories are specified using cache paths in the buildspec file
Seems like one of these settings screws up how the file diff is calculated even though based on normal Git commands executed on the build machine things look OK.
On a successful scan in the last part of the log under âRun sensors on projectâ can see lines 969-971 showing up where before in Taiwos log from above post these were absent.
INFO: ------------- Run sensors on project
965 INFO: Sensor Zero Coverage Sensor
966 INFO: Sensor Zero Coverage Sensor (done) time=2ms
967 INFO: Sensor Java CPD Block Indexer
968 INFO: Sensor Java CPD Block Indexer (done) time=64ms
969 INFO: SCM Publisher SCM provider for this project is: git
970 INFO: SCM Publisher 2 source files to be analyzed
971 INFO: SCM Publisher 2/2 source files have been analyzed (done) time=106ms
972 INFO: CPD Executor 17 files had no CPD blocks
973 INFO: CPD Executor Calculating CPD for 15 files
974 INFO: CPD Executor CPD calculation finished (done) time=21ms
975 INFO: SCM writing changed lines
976 INFO: SCM writing changed lines (done) time=42ms
977 INFO: Analysis report generated in 2327ms, dir size=208 KB
978 INFO: Analysis report compressed in 49ms, zip size=63 KB
979 INFO: Analysis report uploaded in 234ms
I confirm that in this example the correct setting is sonar.pullrequest.key=6.
When you donât get the same result in local run and CI run, it means that something is different in CI. You need to find out what it is, and then fix it. When you are able to run analysis locally in a way that produces the result that you want, it must be possible to make the CI run work the same way. We can try to give tips for that, but ultimately only you can fully debug and figure out.
Iâm not sure what you added, I would add the diff command (git diff --name-only origin/develop sonar-scan) in CI, which will let you verify the list of files matches what you expect.
Thatâs indeed strange. The only reason I can think of to do this is if you want to analyze some generated code. If you donât have a good reason to do this, then I think you shouldnât, and we should try to understand why this setting seems necessary for the desired effect (and then eliminate it).
It could be interesting to check the files that Git excludes in CI. You can find that by running (in CI):
git check-ignore -v *
git status --ignored
The bottomline is, if you can make the CI run see the changed files correctly (output of git diff ...), then the analysis results should be correct too.
Hey Janos, didnât state it clearly in the previous message but this has been resolved with the changes to the build agent cache settings detailed above. We get successful PR scans removing the cache settings.
