"S6497: Pulling an image based on its digest is security-sensitive" is harmful to security

S6497: Using a container image based on its digest is security-sensitive is harmful to security. The rule tells users to do the exact opposite of what they should be doing. Image digests are a positive security measure and I believe Sonar should encourage their use - not discourage it.

As the rule documentation says,

This tag can be updated to point to another version of the container at any point in time.

The documentation further touts that ability to unexpectedly, unplanned “change at any point in change” as a benefit:

The problem is that pulling such an image prevents the resulting container from being updated or patched in order to remove vulnerabilities or significant bugs.

Having software change without any schedule, any tracking, or any control at all is generally not good development practice or good security. Change control is a key feature of security, and this Sonar rule is going contrary to that well established best practice.

From a development perspective, even the most well intentioned updates that fix “significant bugs” can have negative side effects. Consider Microsoft’s Patch Tuesday - even with Microsoft’s massive scale, budgets, and testing capabilities, it’s updates still cause problems sometimes and therefore many organizations and users do not apply them randomly, which is what this Sonar rule does.

For another, security focused example of why change control is critical, consider a supply chain compromise. Unless digests are used, supply chain compromises could not be detected until it’s too late. Consider a Dockerfile that uses FROM python:3.9, then a bad actor somehow manages to publish an image tagged python:3.9 that steals credentials, runs a crypto miner, or something else. Eventually, docker hub will notice and remedy that situation, but in the mean time, the people who use that Dockerfile will have unknowingly been affected, and they’ll have no idea when or how because there was no change in source control. For less popular, less watched images than docker library ones, this problem becomes far more difficult to notice and remedy. Supply chain attacks are on the rise, doing real world damage more and more often.

The rule documentation touches on the right solution:

In general, the use of image digests instead of tags is intended to keep determinism stable within a system or infrastructure for reliability reasons.

I believe the rule should be changed to be the exact opposite of what it is now. The rule should warn whenever an image is referenced without a digest. And the rule text should explain that it’s important to monitor for image updates, just like it’s important to monitor for updates to any dependency, and ideally use a automated tool (such as Dependency Bot or Renovate) or other security scanning software (such as Snyk) to do so.

6 Likes

Hi @candrews,

Thanks for the feedback. I agree that promoting the use of tags instead of digest is debatable.

I understand that digest brings immutability and therefore, they guarantee that the base image will always be the same. I also know that it can be hard for an organization to manage digests and I want to be careful about this.

The rule tells users to do the exact opposite of what they should be doing. Image digests are a positive security measure and I believe Sonar should encourage their use - not discourage it.

Apart from the explanation you gave in your initial post, can you point to other resources that promote the use of digest over tags?

GitHub posted Demonstrating end-to-end traceability with pull requests the day after I made my post. The article doesn’t mention implementation details (such as docker images) but does discuss the importance of change traceability. It makes the same point as I do in my post, but GitHub’s article also cites industry standards, including PCI/DSS, ISO 27001, and COBIT. Having software change without any records or change control process, as happens when Sonar’s current advice to not use docker images digests is followed, does not comply with those standards.

Don’t use dynamic versions for your dependencies doesn’t discuss docker images specific, but does cover the general topic of dependency versions. Here’s a particularly salient excerpt:

Bugfix updates should be safe and absorbed as quickly as possible.
In other words, if you’re using 23.0.+, it should never break because the patch version should only update for bug fixes.
There are two problems with this idea. First, you have to trust that the library is using versions to denote patches correctly. And second, you have to trust that the developer never writes a bug into a patch version. Two big ifs; not worth the risk.

Remind Engineering: Docker images digests describes the supply chain attack scenario I raised and indicates that using images digests is a mitigation for it.

IBM’s Choosing image tags or digests discusses this topic, including this statement:

Image tag mutability is useful and convenient in many scenarios, but it can also be dangerous if you are not aware and prepared to manage it.

The common thread among all of these articles, and many more is that having software change without any record or process, regardless of the mechanism (including through using mutable docker tags as Sonar currently advises), is bad practice.

Hi @candrews

And thanks for the resources you shared. We will reevaluate the rule soon.
It’s unlikely we will revert it completely and promot using digest instead of tags as there are so many cases where it can be harmful either.

Another idea we considered during the research phase was to warn users when they use a tag that is mutable by design like “latest”. This would be a different rule than S6497.

Regards,

as there are so many cases where it can be harmful either.

How can immutability be harmful? If reproducibility and immutability is harmful, then the guidance of significant industry standards (such as COBIT, ISO 27001, and PCI/DSS, among others) which all stress the importance of change control is wrong and should be updated.

I want to stress that in no way is the expectation that updates won’t be performed, and I believe the updated rule text should make it clear that keeping docker image references up to date is critical just as it is for other dependencies. I’d love to see the rule tell users how to keep dependencies up to date, such as by suggesting the use of tools like Renovate and Dependabot.

Another idea we considered during the research phase was to warn users when they use a tag that is mutable by design like “latest”.

All docker tags are mutable by design - that’s part of docker’s design.

Another great article which covers this problem is Mend’s Overcoming Docker’s Mutable Image Tags. It discusses a case where a mistake in a yarn release broke many node applications:

yarn support was broken in all latest Node.js Docker images due to a mistake in a Dockerfile refactor. The Node.js version hadn’t changed, so the image tags also remained unchanged, however the new image pushed for each tag was broken. This meant that Docker images that had previously worked (e.g. node:9 or node:8.10.0-alpine) suddenly stopped working the next time somebody or some machine such as CI pulled them.

Note that none of those images use the “latest” tag - they’re “version” tags (ex node:8.10.0-alpine) so they comply with Sonar’s current rules. And yet, in these cases, every users suffered what’s effectively a DoS.

And DoS is the best case. What if the image was compromised with a less obvious security flaw?

If Sonar is looking for even more sources of industry guidance to pin digests, GitHub Actions advises users of its services to Pin actions to a full length commit SHA:

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action’s repository

It’s unlikely we will revert it completely and promote using digest instead of tags

I’m hoping that with the preponderance of evidence indicating a course correction is necessary, Sonar will do so. :crossed_fingers:

1 Like

Hello Craig,

I want to thank you a huge lot for your detailed feedback on this rule.

Designing the rule’s text was a complex task for us. We were not sure how the Sonar community would welcome it, and at the same time, our intention with it was to warn not to use digests mindlessly – not only because a Stackoverflow post said so, for example.

After a long peer-review back-and-forth process, we settled for a security hotspot instead of a more blocking issue, but we still have doubts.

Getting the point of view of a knowledgeable user of our community about this matter is the best thing we could have asked for. You took the time to write detailed feedback and even gave pointers. We are very grateful for that, and it is genuinely appreciated. I want to emphasize our gratitude for your posts and your time.

As we need (and crave :smiley: ) feedback for improvement, please feel free to communicate with us as frequently as you need. Most importantly, for the infrastructure as code languages, which are relatively new in the Sonar ecosystem.

We we will make sure to change the situation to improve it, a ticket has been created internally.

Have a great day,

Regards,

Loris

2 Likes

Hello again!

Given that some time has passed, do you have any updates you can share?

I’m still hoping that Sonar can update this rule in a timely manner, as I’m concerned that the longer the delay, the more potential there is for confusion to spread or even harm to occur.

Thank you again!

1 Like

Hello Craig,

Thank you for your message. We removed this rule from the default quality profile, SonarWay as a first step.

It should be effective in the next release of the sonar-iac plugin (containing our Docker, Kubernetes, Terraform, Cloudformation and ARM rules), which should appear in the next product releases.

Cheers,

Loris