Quality Gate for Short Lived vs Long Lived vs Master Branch

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension):
  • SonarQube Enterprise Edition Version 8.0 (build 29455)
  • what are you trying to achieve
  • Consistent Quality Gate behaviour
  • what have you tried so far to achieve this

Is more detailed information available on the behaviour of Quality Gates for Short Lived vs Long Lived vs Master Branch? Or is someone able to explain the diffent behaviour we are seeing.

I’ve read the information available here:
https://docs.sonarqube.org/latest/branches/short-lived-branches/
https://docs.sonarqube.org/latest/branches/overview/

However the behaviour is not as I would expect. We currently have the following Conditions setup for our quality gates

However the metrics which are returned for a Short Lived Branch, Long Lived branch and a Master branch is not what we expect. Below are the various metrics we are getting back for the various types of branches.

Short lived branch
https://sonarqube/api/qualitygates/project_status?analysisId=AW7FMdHrrVkB8MhlBehW

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“new_reliability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_security_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_maintainability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_duplicated_lines_density”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“3”,“actualValue”:“0.0”}],“periods”:,“ignoredConditions”:false}}

Long lived branch
https://sonarqube/api/qualitygates/project_status?analysisId=AW7JvzmbrVkB8MhlBknC

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“new_reliability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_security_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_maintainability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_duplicated_lines_density”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“3”,“actualValue”:“0.0”},{“status”:“OK”,“metricKey”:“test_errors”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“test_failures”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”}],“periods”:[{“index”:1,“mode”:“PREVIOUS_VERSION”,“date”:“2019-12-03T09:45:24+1100”}],“ignoredConditions”:false}}

Master
https://sonarqube/api/qualitygates/project_status?analysisId=AW7EHuxtrVkB8MhlBGNN

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“new_reliability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_security_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“new_maintainability_rating”,“comparator”:“GT”,“periodIndex”:1,“errorThreshold”:“1”,“actualValue”:“1”},{“status”:“OK”,“metricKey”:“test_errors”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“test_failures”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”}],“periods”:[{“index”:1,“mode”:“PREVIOUS_VERSION”,“date”:“2019-11-30T16:33:39+1100”}],“ignoredConditions”:false}}

Master (The first time)
https://sonarqube/api/qualitygates/project_status?analysisId=AW66zQsZrVkB8Mhl_7_8

{“projectStatus”:{“status”:“OK”,“conditions”:[{“status”:“OK”,“metricKey”:“test_errors”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”},{“status”:“OK”,“metricKey”:“test_failures”,“comparator”:“GT”,“errorThreshold”:“0”,“actualValue”:“0”}],“periods”:,“ignoredConditions”:false}}

This table summarises the metrics we are receiving for our Quality gate setting, vs what we would have expected to get.

Conditions
Conditions on New Code Short Lived Long Lived Branch Master Branch Master Branch First time Expect for short lived Expect for Long lived
Coverage new_coverage y y
Duplicated Lines % new_duplicated_lines_density y y y y
Maintainability Rating new_maintainability_rating y y y y y
Reliability Rating new_reliability_rating y y y y y
Security Rating new_security_rating y y y y y

Condtions on Overall Code
Unit Test Errors test_errors y y y y
Unit Test Failures test_failures y y y y

We never get a value for new_coverage, in the Master branch we also never got a new_duplicated_lines_density. Oddly on Master the first time an analysis was performed we only got back values for test_errors and test_failures

Hi,

There are actually a number of questions packed in here. First, the lack of New Duplicated Lines% is puzzling to me. Could you provide a screenshot of your project homepage? Are duplication values reported there?

Regarding the first analysis. When you analyze a project for the first time, there’s nothing to compare the current state to, so nothing can be “new”. Thus there are no on New Code metrics. So that deficit isn’t surprising.

For the missing coverage metrics, I believe if you don’t pass any coverage reports at all, we just skip those metrics. At least, I know we don’t report them on the homepage. Can you confirm whether or not you generate coverage reports and pass them into analysis?

 
Ann

Hi,

This is the screenshot of our bordertech_ci Project. The Duplications is available and shows 1.3%

This is a screenshot of the measures tab

Happy with the explanation for the first analysis on master and why some metrics are unavailable.

The coverage is being reported on the homepage, as can be seen with the provided screenshot 89.9%. We use Jenkins to perform our builds. The logs Jenkins indicate that the JaCoCo coverage data was built

[JaCoCo plugin] Collecting JaCoCo coverage data…
[JaCoCo plugin] /target/jacoco.exec;/classes;**/src/main/java; locations are configured
[JaCoCo plugin] Number of found exec files for pattern **/target/jacoco.exec: 1
[JaCoCo plugin] Saving matched execfiles: /home/xxxxxx/.jenkins/workspace/bordertech_ci_sonar_badges-CUGSYM4YRMIYGIT7XOBDYUMVLDKVT7CIL5MI7YMVFIFW3AKMXOFQ/target/jacoco.exec
[JaCoCo plugin] Saving matched class directories for class-pattern: **/classes: /home/xxxxxx/.jenkins/workspace/bordertech_ci_sonar_badges-CUGSYM4YRMIYGIT7XOBDYUMVLDKVT7CIL5MI7YMVFIFW3AKMXOFQ/target/classes
[JaCoCo plugin] Saving matched source directories for source-pattern: **/src/main/java:
[JaCoCo plugin] Loading inclusions files…
[JaCoCo plugin] inclusions:
[JaCoCo plugin] exclusions:
[JaCoCo plugin] Thresholds: JacocoHealthReportThresholds [minClass=0, maxClass=0, minMethod=0, maxMethod=0, minLine=0, maxLine=0, minBranch=0, maxBranch=0, minInstruction=0, maxInstruction=0, minComplexity=0, maxComplexity=0]
[JaCoCo plugin] Publishing the results…
[JaCoCo plugin] Loading packages…
[JaCoCo plugin] Done.

Our settings for JaCoCo report paths are the defaults

Regards
David

Hi David,

Thanks for the screenshots. The homepage and measures page make me think there are no new (added or changed) Lines of Code. Did you actually change anything, code-wise?

 
Ann

1 Like

Hi Ann,

Looks good for us now, we are getting Coverage in our Short Lived and Long Lived branches. After we had our Master analyzed we only made minor documentation changes.

After we made an actual code change, the coverage now appears.

Thank you

David