Do you fail your build / pipeline when the Quality Gate fails?

Is a failing Quality Gate a stop-the-world event for you or just another day at the office? Do you fail your build / pipeline when the Quality Gate fails, or let it go all the way through? And why?

3 Likes

Not yet as we’re testing our Quality Gate. We allow to fail on branches but require pipelines in our CICD to pass before an MR can be merged.

We will be blocking merges between our env based on a scan of the main branch. Once we resolve any existing bugs.

2 Likes

We will always deploy to DEV regardless of result. However, if it fails we prevent the code from moving up the environment stack into the higher environments. There are exceptions to this rule for some applications tho.

2 Likes

No, most of our projects do not fail the build for Quality Gate failures. Instead, we use the GitHub integration to decorate the Pull Request for Quality Gate failures. This allows us to separate build failures from quality failures and the admin for the project can see and override the failures, if necessary. For example, we had a PR that was 99% documentation and failed our quality gate for test coverage. The human admin reviewed and deemed the quality failure acceptable and merged the code. Keeping the failures separate speeds up the time it takes to investigate a failure as it narrows the scope of what the investigation needs to look at.

2 Likes

Yes, and with immediate email feedback to the developer who committed the code change that triggered the build (through Azure DevOps). This ensures that they can fix any quality issues right away while the relevant code is still fresh in their mind, and prevents “ah, we’ll fix that later” issues from creeping through.

If a build doesn’t pass the Quality Gate I’ll barely even review the PR - it won’t be merged anyway and I don’t consider the code as delivered until the QG shows green. Once that’s the case I’ll review the PR and any other issues that SonarQube may show, but that are too trivial to trigger a QG failure. Most of the time I’ll ask the developer to fix those issues as well, but I may choose to move a PR ahead instead.

While this was painful at first, it’s lead to some really clean code and measurably fewer bugs over time. Vulnerability scans also come back much cleaner.

3 Likes

Our pipeline is „build - sonar - ui tests - release“. If quality gate fails we mark the build as unstable, same with testfailure. If the build is unstable we fail at the end and do never release, no exception. Otherwise the „we fix this later“ mindset will kick in, which is bad for all.

2 Likes