The recent application of Quality Gate metrics to new code on short-lived branches is excellent and is working perfectly for my team when violations are found in the bug, vulnerability, or code smell categories (i.e. ones that create explicit issues). However, despite coverage on new code being calculated correctly, it is not failing the Quality Gate I’ve setup for coverage percent on new code.
We’re using SonarQube Enterprise Edition 7.7 and I’m hitting this issue on Git-based Python projects with coverage reporting generated from pytest and security reporting generated from bandit (both reports ingest fine). I’d expect this issue to present itself with other languages.
Some branch information from a sandbox project where I’ve reproduced this:
- master: long-lived, 90% coverage, no security vulnerabilities
- high-code-coverage-with-sec-vuln: short-lived, 90% coverage (100% on new code), one security vulnerability
- lower-code-coverage-no-sec-vuln: short-lived, 78.6% coverage (50% on new code), no security vulnerability
I’ve setup my Quality Gate with the following metrics:
- Coverage < 60%
- Coverage on New Code < 80%
- Duplicated Lines (%) > 20%
- Duplicated Lines (%) on New Code > 3%
- Maintainability Rating worse than B
- Maintainability Rating on New Code worse than A
- Reliability Rating worse than B
- Reliability Rating on new Code worse than A
- Security Rating worse than B
- Security Rating on new Code worse than A
My understanding is that these metrics should allow us to have more lenient standards on our legacy code (so we can improve it over time), but to hold developers to higher standards for any new code they check in since the “… on New Code” metrics will only be exercised on the short-lived branches.
Unfortunately, only the ‘high-code-coverage-with-sec-vuln’ branch fails the Quality Gate (due to the security vulnerability that drops its rating to a B for new code). The ‘lower-code-coverage-no-sec-vuln’ branch should also fail the Quality Gate because the coverage for new code is only 50%, which is lower than the 80% metric.
This seems to be related to https://jira.sonarsource.com/browse/MMF-1369 and Code coverage on branches does not break quality gate but in this case the coverage is being reported accurately; the Quality Gate just isn’t picking up on the failure.
Am I doing something wrong or is SonarQube not correctly reviewing coverage results for short-lived branches? I imagine the same may be true for duplications, but I haven’t tested that yet. If what I’ve outlined here is kosher, my guess is that the Quality Gate for short-lived branches is still only reviewing open issues, and the lack of explicit issues for coverage (and potentially duplications) masks the issue.