Block Pull request based on Quality gate code coverage on new code

SonarQube Server / Community Build
, ,
1

Hi,
We are seeing issue with the way Sonarqube is working to recognize and pass/fail the PR based on the quality gate check which is enabled. The way we want to read the new code is really any new lines that get added as part of that PR should really be the new code. It looks like sonarqube is reading new code quality check of a PR and comparing it with target branch overall code and passing or failing the PR based on that. Hence, we came up with too low numbers in quality check.

Here are sonarqube details:
Version: Sonarqube Enterprise Edition Version 8.9.6
Scanner: Standalone scanner
Plugin: Jacoco 0.8.8

This shows the quality gate that is setup for this project in sonarqube:

image
image1407×570 32.9 KB

New code definition is like this:

image
image976×331 17.4 KB

Sharing the yaml lines that prepare sonarqube analysis:

**task: SonarQubePrepare@5**
**    displayName: Prepare analysis on SonarQube**
**    inputs:**
**      SonarQube: SonarQube Production**
**      scannerMode: CLI**
**      configMode: manual**
**      cliProjectKey: $(SONAR_PROJECT_KEY)**
**      cliProjectName: $(SONAR_PROJECT_NAME)**
**      cliSources: $(LOB_DIRECTORY)**
**      extraProperties: |**
**        # Scan everything, but explicitly exclude the following**
**        sonar.exclusions=**/testsrc/*,**/testsrc/**/*,**/jalo/*,**/gensrc/*,**/gensrc/**/*,**/*Test.java**
**        sonar.inclusions=**/*.java,**/*.xml**
**        sonar.dynamicAnalysis=reuseReports**
**        $(sonarBranchName)**
**        **
**        sonar.language=java**
**        sonar.java.source=11**
**        sonar.java.libraries=**/lib/*.jar**
**        **
**        # Jacoco settings for code coverage.**
**        sonar.core.codeCoveragePlugin=jacoco**
**        jacoco.version=0.8.8**
**        sonar.coverage.jacoco.xmlReportPaths=$(BUILD_DIR)/hybris/hybris/log/junit/jacoco.xml**
**        sonar.coverage.exclusions=**/testsrc/*,**/testsrc/**/*,**/*.jar,**/jalo/*,**/*.js,**/*.css,**/*.xml,**/*.scss,**/*Test.java,**/*DaoImpl.java,**/*Dao.java,**/*IntegrationTest.java,**/*DaoImplTest.java**

**        # Path to the directory containing the Junit report xml file.**
**        sonar.junit.reportPaths=$(BUILD_DIR)/hybris/hybris/log/junit/**
**        **
**        # The path to any generated binaries that should be part of the scan.**
**        sonar.tests=$(TEST_PATHS)**
**        sonar.java.binaries=$(LOB_DIRECTORY)**

Please help us to understand how can we really get the PR passed or failed based on new code lines that are written on source branch when trying to merge to target branch.

2

Hi,

Welcome to the community!

Could you give more details on this? What lines are being missed?

 
Ann

3

Hi Ann, thanks for responding. All I want to do is block Pull Request merges if the code coverage on new code of the PR doesn’t meet a threshold as mentioned in quality check. As per our understanding, it looks like (hope we are wrong) PR new code coverage is being compared with overall code coverage of target branch. Because of this understanding, we have enabled the quality gate check for 45% instead of wanting to meet 60%.

As per how we have configured here, please let us know if it is correct and if at all this is correct, what should be the expected behaviour.

4

Hi,

For PRs only conditions on New Code are enforced. So you shouldn’t see a PR fail the Quality Gate because of Overall coverage.

 
Ann

5

In that case, we don’t see the right results. Can you help us understand what is the new code definition as per the settings we have done? Does it mean any change done on top of previous version of source branch? Meaning… if source branch is at 70% new code coverage, any change done on top of that code is meant as new code?

6

Hi,

What setting have you used? You can check by looking in Project Settings → New Code.

 
Ann

7

Yes… here is what is set as new code.

image
image1767×593 36.4 KB

image
image1748×590 39.1 KB

Currently we haven’t configured long lived branch specific setting… All the branch names you see above are required to long live. They are infact mini projects for this master project in the same repo. We haven’t configured individual project for each of these mini project and there is one sonar-project.properties file which is being referred for every PR which runs for each of these long lived branches.

I am supposing if I would need separate projects for each of these branch scans so i could apply new code settings in a way that new code refers the previous version of this long lived branch.

Kindly advise. Hope I don’t cause too much confusion.

8

Hi,

I just went back to the OP to refresh myself & see that you provided your New Code setting there. Sorry about making you give it again.

For a branch, this is going to be code changed since the previous release. Note that there have been improvements in New Code detection related to rebasing since the LTS.

In a PR context, it should be all/only the changes in the PR’s branch versus the base branch. Unless you’ve done some rebasing in the PR branch, I believe it should be fairly straightforward.

This shouldn’t be the case. It should be comparing to the previous version of that branch.

And at this point I’m a bit confused. This thread started with PR coverage. Is that still the focus?

 
Ann

9

Hi Ann,

Sorry for back and forth questions. But I have a case, which probably might give an overview of how it looks.

PR raised has one line of change:

image
image1427×428 42 KB

But the code coverage of this new code in PR seems like scanning all lines of branch or target branch. Coverage on new code shows 46.5% by scanning like 21K lines, which should not be the case.

image
image1638×682 40.2 KB

Here is the overall coverage on target branch.

image
image1538×779 50 KB

How is the code coverage being calculated in this case?
I have added this line in Prepare analysis step- sonar.newCode.referenceBranch=$(sonarBranchName)

I don’t seem to understand what is causing 21k lines of code when the line changed (which should be new code) is just 1 in source branch. I am starting to think my definition of new code might need some change. Please guide.

10

Hi,

Your middle screenshot cuts off the tabs at the top. I guess they also say the New Code Period started yesterday? I think it would be helpful to go to the Measures page and look at what SonarQube is seeing as New Code. See what you can identify as non-new that SonarQube thinks is new and then trace back to the history of the files / lines in question. When was it they actually changed last, and by what route did those changes get into your branch.

 
Ann

11

Thanks Ann for being patient in addressing my queries.

I am starting to think if i need to declare something as long lived branch or short lived branch. As per this page (SonarQube detects whole file as new code in shortlived branches and pull requests, when only part of the file has changed - #7 by Alexandr_Marchenko), it looks like a similar problem for us. The analysis contains entire code as new code.

I made some tests by adding following parameters in my sonar-project.properties file, i started seeing this error in PR. What does this mean?

Error on PR page: for merge into # Do nothing from $(BRANCH)

image
image1476×740 49.7 KB

12

Hi,

By SonarQube 8.9, there are only long-lived branches.

Are you saying the entire file is marked as new code, when it should be only a few lines in the file?

Can you provide details of your checkout? Also, is it possible you have some automation in your process that alters whitespace? We’ve seen this sort of thing before when E.G. the IDE is inserting tab characters and they’re automatically being changed to spaces somewhere in the process.

Again, can you see what your SCM has as the history for the “not-new” lines in the file?

Sorry, I’ve never seen that error before. I don’t think it’s coming from SonarQube.

 
HTH,
Ann

13

Here is the PR checkout part:

  • script: |
    repoName=echo "$(system.pullRequest.sourceRepositoryUri)" | sed "s/^.*\///"
    # Override the project directory based on the pull request variables.
    echo “##vso[task.setvariable variable=PROJECT_DIR]$(ROOT_DIR)/${repoName}”
    git init “$(Pipeline.Workspace)/s/${repoName}”
    cd “$(Pipeline.Workspace)/s/${repoName}”
    git remote add origin “$(system.pullRequest.sourceRepositoryUri)”
    git config gc.auto 0
    git config --get-all http.$(system.pullRequest.sourceRepositoryUri).extraheader
    git config --get-all http.extraheader
    git config --get-regexp .extraheader
    git config --get-all http.proxy
    git config http.version HTTP/1.1
    BRANCH=“pull/$(system.pullRequest.pullRequestId)/merge”
    git -c http.extraheader=“AUTHORIZATION: bearer $(System.AccessToken)” fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin "+refs/heads/    :refs/remotes/origin/*" “+refs/${BRANCH}:refs/remotes/${BRANCH}”
    git checkout “refs/remotes/${BRANCH}”
    git checkout --progress --force “${BRANCH}”
    condition: ${{parameters.isPrBuild}}
    displayName: Checkout as PR build
14

Hi,

You’re doing a fetch and checkout rather than a clone. The docs are explicit that a “full clone is required for this integration to be able to collect the required blame information”. I’ve been going in circles in the Git docs trying to figure out whether or not your sequence of operations puts the required history in place to properly detect new code. It’s possible this will work properly if you --unshallow your fetch. Otherwise, you may need to switch to the expected clone operation.

 
HTH,
Ann

15

Hi Ann…

I tried doing ‘git fetch --unshallow’ and it gave me this error:
Initialized empty Git repository in /agent/_work/2/s/test1-bpnextgen-sapcommerce/.git/
fatal: --unshallow on a complete repository does not make sense

Here is the exact updated command(let me know what is the exact update required):

git -c http.extraheader=“AUTHORIZATION: bearer $(System.AccessToken)” fetch –unshallow --force --tags --prune --prune-tags --progress --no-recurse-submodules origin "+refs/heads/ :refs/remotes/origin/*" “+refs/${BRANCH}:refs/remotes/${BRANCH}”
git checkout “refs/remotes/${BRANCH}”
git checkout --progress --force “${BRANCH}”

Here is the sequence of jobs in my build and scan pipeline, please let me know if this needs some kind of reordering:

Checkout as PR build
Resolve target branch
find environment variables
Prepare analysis on sonarqube
Build
Run unit test
Publish test results
Run Code Analysis
Publish Quality gate

16

Sending logs of run code analysis step for reference.

LOG.txt (19.2 KB)

17

I noticed something that I might be useful to check to understand why PR is comparing old lines and old code is probably because it reads the previous version as too old.

Here is a screenshot from Activity tab and it shows ‘new code period starts here’ on an analysis done on the day project was onboarded on sonarqube. So, although the pull request scan should really consider new code compared to previous version of target branch, as the previous version of target branch is really old, it is considering too many lines as new code. Please advise. Is this the reason for my incorrect new code calculation in PR scans?

image

image

image
image1370×560 47.9 KB

18

Hi,

I see this in the log you sent:

2022-09-05T12:28:11.2202724Z INFO: SCM Publisher No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.

Are you not analyzing from the checkout directory? The SCM data needs to be in the analysis root directory.

Regarding your screenshot & PR analysis, they’re two different things. PR analysis doesn’t have a New Code Period, since PR analysis is only of new code.

 
Ann