Quality Gate for Short Lived vs Long Lived vs Master Branch

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension):
  • SonarQube Enterprise Edition Version 8.0 (build 29455)
  • what are you trying to achieve
  • Consistent Quality Gate behaviour
  • what have you tried so far to achieve this

Is more detailed information available on the behaviour of Quality Gates for Short Lived vs Long Lived vs Master Branch? Or is someone able to explain the diffent behaviour we are seeing.

I’ve read the information available here:
https://docs.sonarqube.org/latest/branches/short-lived-branches/
https://docs.sonarqube.org/latest/branches/overview/

However the behaviour is not as I would expect. We currently have the following Conditions setup for our quality gates

However the metrics which are returned for a Short Lived Branch, Long Lived branch and a Master branch is not what we expect. Below are the various metrics we are getting back for the various types of branches.

Short lived branch
https://sonarqube/api/qualitygates/project_status?analysisId=AW7FMdHrrVkB8MhlBehW

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“new_reliability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_security_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_maintainability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_duplicated_lines_density”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“3”,“actualValue”:“0.0”}],“periods”:,“ignoredConditions”:false}}

Long lived branch
https://sonarqube/api/qualitygates/project_status?analysisId=AW7JvzmbrVkB8MhlBknC

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“new_reliability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_security_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_maintainability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_duplicated_lines_density”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“3”,“actualValue”:“0.0”},{“status”:“OK”,“metricKey”:“test_errors”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“test_failures”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”}],“periods”:[{“index”:1,“mode”:“PREVIOUS_VERSION”,“date”:“2019-12-03T09:45:24+1100”}],“ignoredConditions”:false}}

Master
https://sonarqube/api/qualitygates/project_status?analysisId=AW7EHuxtrVkB8MhlBGNN

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“new_reliability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_security_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_maintainability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“test_errors”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“test_failures”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”}],“periods”:[{“index”:1,“mode”:“PREVIOUS_VERSION”,“date”:“2019-11-30T16:33:39+1100”}],“ignoredConditions”:false}}

Master (The first time)
https://sonarqube/api/qualitygates/project_status?analysisId=AW66zQsZrVkB8Mhl_7_8

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“test_errors”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“test_failures”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”}],“periods”:,“ignoredConditions”:false}}

This table summarises the metrics we are receiving for our Quality gate setting, vs what we would have expected to get.

Conditions
Conditions on New Code Short Lived Long Lived Branch Master Branch Master Branch First time Expect for short lived Expect for Long lived
Coverage new_coverage y y
Duplicated Lines % new_duplicated_lines_density y y y y
Maintainability Rating new_maintainability_rating y y y y y
Reliability Rating new_reliability_rating y y y y y
Security Rating new_security_rating y y y y y

Condtions on Overall Code
Unit Test Errors test_errors y y y y
Unit Test Failures test_failures y y y y

We never get a value for new_coverage, in the Master branch we also never got a new_duplicated_lines_density. Oddly on Master the first time an analysis was performed we only got back values for test_errors and test_failures

Hi,

There are actually a number of questions packed in here. First, the lack of New Duplicated Lines% is puzzling to me. Could you provide a screenshot of your project homepage? Are duplication values reported there?

Regarding the first analysis. When you analyze a project for the first time, there’s nothing to compare the current state to, so nothing can be “new”. Thus there are no on New Code metrics. So that deficit isn’t surprising.

For the missing coverage metrics, I believe if you don’t pass any coverage reports at all, we just skip those metrics. At least, I know we don’t report them on the homepage. Can you confirm whether or not you generate coverage reports and pass them into analysis?

 
Ann

Hi,

This is the screenshot of our bordertech_ci Project. The Duplications is available and shows 1.3%

This is a screenshot of the measures tab

Happy with the explanation for the first analysis on master and why some metrics are unavailable.

The coverage is being reported on the homepage, as can be seen with the provided screenshot 89.9%. We use Jenkins to perform our builds. The logs Jenkins indicate that the JaCoCo coverage data was built

[JaCoCo plugin] Collecting JaCoCo coverage data…
[JaCoCo plugin] /target/jacoco.exec;/classes;**/src/main/java; locations are configured
[JaCoCo plugin] Number of found exec files for pattern **/target/jacoco.exec: 1
[JaCoCo plugin] Saving matched execfiles: /home/xxxxxx/.jenkins/workspace/bordertech_ci_sonar_badges-CUGSYM4YRMIYGIT7XOBDYUMVLDKVT7CIL5MI7YMVFIFW3AKMXOFQ/target/jacoco.exec
[JaCoCo plugin] Saving matched class directories for class-pattern: **/classes: /home/xxxxxx/.jenkins/workspace/bordertech_ci_sonar_badges-CUGSYM4YRMIYGIT7XOBDYUMVLDKVT7CIL5MI7YMVFIFW3AKMXOFQ/target/classes
[JaCoCo plugin] Saving matched source directories for source-pattern: **/src/main/java:
[JaCoCo plugin] Loading inclusions files…
[JaCoCo plugin] inclusions:
[JaCoCo plugin] exclusions:
[JaCoCo plugin] Thresholds: JacocoHealthReportThresholds [minClass=0, maxClass=0, minMethod=0, maxMethod=0, minLine=0, maxLine=0, minBranch=0, maxBranch=0, minInstruction=0, maxInstruction=0, minComplexity=0, maxComplexity=0]
[JaCoCo plugin] Publishing the results…
[JaCoCo plugin] Loading packages…
[JaCoCo plugin] Done.

Our settings for JaCoCo report paths are the defaults

Regards
David

Hi David,

Thanks for the screenshots. The homepage and measures page make me think there are no new (added or changed) Lines of Code. Did you actually change anything, code-wise?

 
Ann

1 Like

Hi Ann,

Looks good for us now, we are getting Coverage in our Short Lived and Long Lived branches. After we had our Master analyzed we only made minor documentation changes.

After we made an actual code change, the coverage now appears.

Thank you

David

@dlow - I’m struggling with some similar issues myself and in particular in my case I’ve noted that, though my Unit Test report is being processed by Sonar, I don’t see any information on Unit Test failures, counts, or run time on my short lived branches. (I do see information from the processing of my coverage report.) Is that consistent with what you are seeing and do you know if that is expected behavior?

Yes. That is consistent with my experience. No information on Unit test for short lived branches. It shows up for Long lived and the Master

:frowning_face: that is disappointing. that means i can’t use unit test failure as quality gate to prevent merging a PR. why in the world would it have that as a restriction? did you ever see anything in the docs mentioning that limitation?

That is also what we wanted - “unit test failure as quality gate to prevet merging a PR”. I have made a suggestion for it to be added. Unit Test Conditions on New Code Quality Gate

1 Like

Hi,

with Sonarqube 8.1, short-lived / long-lived branches have vanished.
Now a branch is a branch and every branch has full dashboard and metrics.
So updating to Sonarqube 8.1 may solve your problems.

But you will hit another problem if using Quality gates with ‘new’ conditions because of
the definition of new, see this threads for details

Gilbert

1 Like

Thanks for the link, I’ll follow that. I still haven’t found anywhere that states this is an expected limitation; however, so I posted this one as well to hopefully find out: Why don't Unit Test failures show up in short term branches?