Outage: SonarScanner CLI Docker + SonarQube & SonarCloud GitHub Actions

Clean Code Sonar Updates
22

it seems to be the latest release v5.

guerlielton-nunes said that the workaround is set the version to sonarsource/sonarqube-scan-action@v1.2.0 which uses cli-4.

1 Like
23

This is not an outage , this was a bad release that overwrote an existing docker tag that impacted downstream dependencies (that should not need to be touched after 8 months).

2 Likes
24

Using v2.0 does not fix the issue. v1.2.0 does bypass this specific issue, however that version has other compatibility issues and should not be considered a valid work around. Please fix this asap.

3 Likes
25

Can confirm, 2.0 doesnā€™t fix it.

1 Like
26

As a temporary solution, canā€™t you just rerun the release from 8 months ago to re-override the tag that was overridden in the first place and then proceed to fix the actual issue with the latest? Some people, including myself, may have stringent rules in place to only allow certain versions of actions within workflow files and switching to a new SHA or previous version (like 1.2.0 which actually works, versus 2.0 as mentioned which still suffers the same problem as it is pointing to 5.0.1 of the cli) isnā€™t an easy or quick option.

27

Downgrading causes dependency issues in my workflow :face_with_peeking_eye:

Downgraded to SonarSource/sonarcloud-github-action@v1.9.1 from masterā€¦

ERROR: Error during SonarScanner execution

java.lang.IllegalStateException: The plugin [python] does not support Java 11.0.20

Caused by: java.lang.UnsupportedClassVersionError: org/sonar/plugins/python/PythonPlugin has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

28

Just tried v2.0.2 and it works

1 Like
29

Hi all,

Weā€™ve restored a working version of the underlying Scanner CLI Docker image.

This should now be fixed for everyone.

Sorry, again, for the inconvenience.

 
Ann

6 Likes
30

Could you all provide a postmortem once the dust settles?

1 Like
31

Hi @mattbangert,

We do internal post-mortems after every incident, but we donā€™t usually publish them.

What I can tell you today is that

  1. we released the 5.0.1 version of the scanner image last August as a bug fix
  2. since then, weā€™ve been accumulating unreleased changes in the repo
  3. today we released a new iteration of the image
  4. in doing so, we overrode the working, 5.0.1 image with the new iteration
  5. the fix was re-replacing the original 5.0.1 image with its original contents

I expect the retro to focus on items 2 and 4 above.

 
Ann

2 Likes
32

The fix has been applied! :raised_hands:

2 Likes
33

@ganncamp in my org, we often pin GitHub actions to avoid similar issues

uses: sonarsource/sonarqube-scan-action@69c1a75940dec6249b86dace6b630d3a2ae9d2a7 # v2.0.1

Unfortunately, that didnā€™t help in this case, as the action itself did not pin its reference to the sonarsource/sonar-scanner-cli:5.0.1 image. Would you consider doing that for future action versions?

34

Hi @GreyTeardrop,

I think everything will be up for grabs in the retro.

Weā€™re definitely aware of the pain this caused, and it was all-hands-on-deck yesterday (the Europeans got called back to their desks from dinner) to get this addressed. We are going to figure out how to prevent this in the future.

 
Ann

1 Like
35

I did not see any follow up on this incident. The Dockerfile base image is still hard coded making it the action vulnerable to this kind of failure in the future. Is there any plan to change this or introduce any other preventative measure?

36

Hi all,

Since this is the first time weā€™ve posted a retro followup publicly, we donā€™t have a lot of machinery established around it. So Iā€™m just posting here:

Report summary and improvements following the SonarScanner CLI Docker Release on April 16, 2024

Following a SonarScanner CLI Docker release on April 16, 2024, GitHub Actions and GitLab Workflow users could not use SonarQube or SonarCloud to perform code analysis in their build pipelines for a few hours.

What caused the issue?

The problem was triggered by the release of a new SonarScanner CLI Docker image to address vulnerabilities discovered in this Docker image. Because of the versioning scheme in place at the time, Sonar published the new Scanner CLI Docker image using the previously released version number, which caused the new release to replace the prior version.

While the new version addressed the SonarScanner CLI Docker image vulnerabilities, we unintentionally introduced changes that broke the images built on top of it. This included removing ROOT access on the image and removing packages that were not necessary for Sonar use cases. This impacted Sonarā€™s GitHub Actions and GitLab Workflows integrations, as the SonarScanner CLI Docker image is the base image for both of these due to a lack of pinned versions.

Restoring GitHub Actions

To fix GitHub Actions and GitLab Workflows, Sonar republished the prior working SonarScanner CLI Docker image version (basically a rollback). This action restored SonarQube and SonarCloud functionality so that GitHub Actions and GitLab Workflows performed as expected.

However, as many users pointed out, this was a short-term solution that did not prevent the issue from happening again, and we recognize that replacing existing versions is a bad practice.

Long-term fix implementation

On May 21st, Sonar released a new version of the SonarScanner CLI Docker, patched against the following vulnerabilities (CVE-2018-20225, CVE-2024-0853, CVE-2024-2511, CVE-2023-3635). Also, Sonar can now deliver changes for the Scanner Docker CLI and flip the version only for the dependent stacks after testing the impact.

Additionally, Sonar has implemented several changes to prevent this issue from occurring again. This includes the following:

  • Decoupling. Moving forward, the sonar-scanner-cli-docker GitHub tag will be decoupled from the CLI version. This will allow us to make versioned changes to the docker image while still keeping track of the underlying SonarScanner CLI version.
  • New versioning strategy. For each release, Sonar will publish a new full version tag, update the latest tag, and update or publish major and minor versions.
  • Image pinning. Sonar will pin the parent image to a specific version so that it is not automatically updated when there is a change. This will give Sonar and Sonar users an opportunity to manually update and have the CI running, validating that everything is working.
  • New naming conventions. Sonar will use GitHub action versions (like sonarsource/sonarcloud-github-action@v1.2) and recommends using them instead of sonarsource/sonarcloud-github-action@master

We acknowledge that this caused significant disruption and presented a security risk for Sonar users and would like to apologize again for any inconvenience the outage may have caused.

Thank you.

3 Likes