Hi all,
Since this is the first time we’ve posted a retro followup publicly, we don’t have a lot of machinery established around it. So I’m just posting here:
Report summary and improvements following the SonarScanner CLI Docker Release on April 16, 2024
Following a SonarScanner CLI Docker release on April 16, 2024, GitHub Actions and GitLab Workflow users could not use SonarQube or SonarCloud to perform code analysis in their build pipelines for a few hours.
What caused the issue?
The problem was triggered by the release of a new SonarScanner CLI Docker image to address vulnerabilities discovered in this Docker image. Because of the versioning scheme in place at the time, Sonar published the new Scanner CLI Docker image using the previously released version number, which caused the new release to replace the prior version.
While the new version addressed the SonarScanner CLI Docker image vulnerabilities, we unintentionally introduced changes that broke the images built on top of it. This included removing ROOT access on the image and removing packages that were not necessary for Sonar use cases. This impacted Sonar’s GitHub Actions and GitLab Workflows integrations, as the SonarScanner CLI Docker image is the base image for both of these due to a lack of pinned versions.
Restoring GitHub Actions
To fix GitHub Actions and GitLab Workflows, Sonar republished the prior working SonarScanner CLI Docker image version (basically a rollback). This action restored SonarQube and SonarCloud functionality so that GitHub Actions and GitLab Workflows performed as expected.
However, as many users pointed out, this was a short-term solution that did not prevent the issue from happening again, and we recognize that replacing existing versions is a bad practice.
Long-term fix implementation
On May 21st, Sonar released a new version of the SonarScanner CLI Docker, patched against the following vulnerabilities (CVE-2018-20225, CVE-2024-0853, CVE-2024-2511, CVE-2023-3635). Also, Sonar can now deliver changes for the Scanner Docker CLI and flip the version only for the dependent stacks after testing the impact.
Additionally, Sonar has implemented several changes to prevent this issue from occurring again. This includes the following:
- Decoupling. Moving forward, the sonar-scanner-cli-docker GitHub tag will be decoupled from the CLI version. This will allow us to make versioned changes to the docker image while still keeping track of the underlying SonarScanner CLI version.
- New versioning strategy. For each release, Sonar will publish a new full version tag, update the latest tag, and update or publish major and minor versions.
- Image pinning. Sonar will pin the parent image to a specific version so that it is not automatically updated when there is a change. This will give Sonar and Sonar users an opportunity to manually update and have the CI running, validating that everything is working.
- New naming conventions. Sonar will use GitHub action versions (like sonarsource/sonarcloud-github-action@v1.2) and recommends using them instead of sonarsource/sonarcloud-github-action@master
We acknowledge that this caused significant disruption and presented a security risk for Sonar users and would like to apologize again for any inconvenience the outage may have caused.
Thank you.