Whole files considered new code in short-lived branch

sonarqube

(Petermbauer) #1
  • versions used: SonarQube 7.6, CFamiliy Plugin 6.1, Git
  • error observed

When changing some lines only in a short-lived branch, the whole files considered to be “new code” and the test code coverage is calculated taking the whole file into account so it is lower than expected.
On the other hand the other rules are not applied to the whole file so no code smells are shown for unchanged code in the rest of the file.
I found a similar issue regarding code duplication:

This is the actual diff to the target branch as shown in Bitbucket:

This is what SonarQube shows (rest of the file looks the same):
with_develop_whole_file_diff

Here is what i found relevant in the logs:

10:03:33.022 DEBUG: Sensors : 
10:03:33.024 INFO: SCM provider for this project is: git
10:03:33.024 INFO: 5 files to be analyzed
10:03:33.030 DEBUG: Blame file XXXXX
...
10:03:33.182 INFO: 5/5 files analyzed
...
10:03:34.228 INFO: SCM writing changed lines
10:03:34.250 DEBUG: Merge base sha1: cf66888384634f2411429c2829aeb40c1bbf9543
10:03:34.276 DEBUG: Merge base sha1: cf66888384634f2411429c2829aeb40c1bbf9543
10:03:34.287 DEBUG: Merge base sha1: cf66888384634f2411429c2829aeb40c1bbf9543
10:03:34.310 DEBUG: Merge base sha1: cf66888384634f2411429c2829aeb40c1bbf9543
10:03:34.318 DEBUG: Merge base sha1: cf66888384634f2411429c2829aeb40c1bbf9543
10:03:34.338 DEBUG: SCM reported changed lines for 5 files in the branch
10:03:34.338 INFO: SCM writing changed lines (done) | time=110ms
10:03:34.358 INFO: Analysis report generated in 1019ms, dir size=525 KB
  • steps to reproduce: change a file in a short-lived branch

Is this the intended behavior? For us this renders the “Code coverage in the branch” metric useless.
Before running the Sonarscanner, a git fetch origin develop:develop is done to make sure it is available locally.

thx,
Peter


(Janos Gyerik) #4

Code in yellow background on SonarQube indicates new code. So it doesn’t match with the diff on Bitbucket.

When running the analysis, did you specify the target branch with sonar.branch.target?


(Petermbauer) #5

yes, here are the options

SONAR_SCANNER_OPTS=-Dsonar.projectKey=XXXXX -Dsonar.projectVersion=develop -Dsonar.sources=sources,test -Dsonar.cfamily.build-wrapper-output=sonar_bw_output -Dsonar.exclusions=build_*/**,**/*.xml -Dsonar.cfamily.bullseye.reportPath=build_bullseye/bullseye_output.xml -Dsonar.coverage.exclusions=conanfile.py,test/* -Dsonar.branch.name=features/YYYYYY -Dsonar.branch.target=develop

(Andreas Petersen) #6

I’m having the exact same issue SonarQube detects whole file as new code in shortlived branches and pull requests, when only part of the file has changed.

It means that we cannot make use of shortlived branches and pull requests analysis.


(Petermbauer) #8

Funny thing is that i added the git fetch origin develop:develop to get rid of a new warning that appeared when updating from SonarQube 7.4 to 7.6 about the missing branch:
image
image
So the warning is gone but the detection of the actual diff got worse.


(Julien Henry) #9

Hi,

Can you check you are not using a shallow clone of the repository? This is often a cause of trouble.

You can run a git fetch --unshallow || true before the SQ analysis.


(Andreas Petersen) #10

In my case:

$ git fetch --unshallow || true
fatal: --unshallow on a complete repository does not make sense

Running analysis after the above command, a single line change still results in the whole file being marked as new code.


(Petermbauer) #11

Same here, there is just the actual branch being checked out, hence i added the additional fetch of the target branch, but it is not just a shallow clone.

git fetch --unshallow 
fatal: --unshallow on a complete repository does not make sense
script returned exit code 128

(G Ann Campbell) split this topic #12

3 posts were split to a new topic: Vulnerability in Confluence version


(Andreas Petersen) #14

I have also tried using the sonar-scanner, rather than the gradle scanner plugin. Using the sonar-scanner the problem still persists: Changing 1 line in a file, marks the whole file as new code.


(Julien Henry) #16

OK, so we can exclude the shallow clone from the problem, thanks.

@Andreas_Petersen I’m not surprised the behavior is the same. All Git related interaction are made by the Git plugin that works in a similar way for all scanners.

The code responsible to find changed lines is:

This is hard for us to investigate, so if you have the opportunity, please get the sources of the sonar-scm-git-plugin in a Java IDE, and attach a debugger to the scanner (using sonar-scanner-debug will automatically start the scanner in debug mode).

Another option would be to share (privately) the copy of your repository so that we could better investigate, but be sure to be allowed to do so.


(Andreas Petersen) #17

I can try to make a dummy repo that reproduces the issue that I could share


(Julien Henry) #18

That would be awesome!


(Andreas Petersen) #19

Hi @Julien_HENRY,

I have reproduced the issue in a new repo, you can find it here: https://github.com/AndreasPetersen/sonarqube-change. The repo has the steps to reproduce.

Additionally, here are some further screenshots of my results (now that I have some code I can actually show):

In the screenshot below are the results after the initial scan on master:

Below are the results of the scan on a branch, where a single line was changed:

And the file itself:

In the git diff I have:

git show
commit 81e87bc2d7827c588b8aed4954ae055a2ed4f489 (HEAD -> sonarqube-test)
Author: Andreas Petersen 
Date:   Mon Apr 15 13:32:24 2019 +0200

    made a change

diff --git a/src/main/java/dk/andreas/Main.java b/src/main/java/dk/andreas/Main.java
index f97860f..afaae6b 100644
--- a/src/main/java/dk/andreas/Main.java
+++ b/src/main/java/dk/andreas/Main.java
@@ -15,6 +15,7 @@ public class Main {
    }

    public static boolean tested() {
+        System.out.println("Adding a line");
        return true;
    }
}

As you can see I only changed one line in the file, yet SonarQube detects the whole file as changed.


(Julien Henry) #20
  • git checkout -b sonarqube-test
  • Add a line in Main.tested() , e.g. a print statement just before the return statement
  • gradle build

Don’t you commit/push the change in the branch? We are supporting analysis of code in branch/PR that have been pushed to the ALM. Analyzing local changes may lead to unexpected results, since we can’t use integration with the ALM to request missing info.


(Andreas Petersen) #21

Ah yes, sorry, I forgot writing the commit step


(Andreas Petersen) #22

I updated the README to include the commit step.

Just to be clear regarding my previous comment: When I wrote “I forgot writing the commit step”, I mean I forgot writing it in the README on the repo. I did commit, as you can see from my git show in my other previous comment.


SonarQube detects whole file as new code in shortlived branches and pull requests, when only part of the file has changed
Old issues from other branches leak into new code short live branch analysis
(Julien Henry) #24

Works fine for me:

Is the Git plugin up to date in your SonarQube instance? Mine is 1.8 (build 1574)

What is your operating system? Do you have some global Git settings that could edit all lines (like auto crlf)?


(Andreas Petersen) #25

The operating system that SonarQube runs on is: Windows 7 Enterprise, Version 6.1 (Build 7601: Service Pack 1)
The operating system that I run the scanner on (my computer): Windows 7 Enterprise, Version 6.1 (Build 7601: Service Pack 1)
Version of the Git plugin: 1.8 (build 1574)

These are my git settings:

file:"C:\\ProgramData/Git/config"       core.symlinks=false
file:"C:\\ProgramData/Git/config"       core.autocrlf=true
file:"C:\\ProgramData/Git/config"       core.fscache=true
file:"C:\\ProgramData/Git/config"       color.diff=auto
file:"C:\\ProgramData/Git/config"       color.status=auto
file:"C:\\ProgramData/Git/config"       color.branch=auto
file:"C:\\ProgramData/Git/config"       color.interactive=true
file:"C:\\ProgramData/Git/config"       help.format=html
file:"C:\\ProgramData/Git/config"       rebase.autosquash=true

and the git settings in the repo itself:

file:.git/config        core.repositoryformatversion=0
file:.git/config        core.filemode=false
file:.git/config        core.bare=false
file:.git/config        core.logallrefupdates=true
file:.git/config        core.symlinks=false
file:.git/config        core.ignorecase=true

(Andreas Petersen) #26

Setting git config --global core.autocrlf false fixed the issue. It now works. Thanks @Julien_HENRY. Perhaps the documentation on branches needs an update to include this information, or perhaps I just missed it?