Why did Tekton change its base image?

[skaegi]

Why did Tekton change its base image from gcr.io/distroless/static to ghcr.io/distroless/static? Has Kubernetes and other projects made a similar change? The base image is an absolutely critical part of things and although I'm sure there are good intentions here I have no idea how and if this image will be maintained.

[vdemeester]

For what is worth:

• gcr.io/distroless images comes from https://github.com/GoogleContainerTools/distroless
• ghcr.io/distroless images comes from https://github.com/distroless

gcr.io/distroless is maintain mainly by google I think, ghcr.io/distroless is maintain by the distroless org, which today, is mainly composed by ChainGuard folks I think 🙃. Both are maintained, and kept up-to-date.

/cc @imjasonh @mattmoor

[imjasonh]

We believe the new distroless images are a better alternative to gcr.io/distroless, for a few reasons:

1. They are built using apko, which lets them be described in YAML instead of Bazel's Python-ish bzl. This makes it easier to tell at a glance what's going into the image, easier to update the image, and easier to build your own variant if you so choose (e.g., to include another package, or pin to an older version)
2. Because describing the image is easier, maintaining it is easier, and updates are applied to the image much more quickly. A new distroless.dev/static is built every night with updated packages, compared with gcr.io/distroless/static, which receives updates more slowly. Though there isn't a lot in the image, there's still something, and we'll get updates faster with these images.
3. In addition to being signed as the previous distroless images are, the apko-based distroless images include thorough SBOMs of the contents, which ko is capable of discovering and referencing in its own SBOMs. We plan to improve this even further, so ko's generated SBOMs (i.e., Tekton's SBOMs) are as complete and useful as possible.

(A brief aside on naming; we recommend reaching these images through distroless.dev/static instead of ghcr.io/distroless/static, as we plan to move the images to a new GitHub org soon, and ghcr.io/distroless/* will not receive updates afterward -- distroless.dev currently points to ghcr.io/distroless/*, and we will update it to point to the new repo automatically.)

In any case, if you build your own Tekton images, you are absolutely free to keep building them on gcr.io/distroless/static, or any base image that fits your needs. You'd simply need to update .ko.yaml to point to those images (including the git base image, which is currently based on distroless.dev/git), and update the shell-image used.

Some relevant PRs that may have relevant discussions:

• Use a new base image for the git-init image. #4758
• Switch to ghcr.io/distroless/busybox for shell-image #4762
• Use ghcr.io/distroless/static as base image #5009
• chore: change ghcr.io/distroless/* to distroless.dev/* #5341

[skaegi]

Thanks all. It sounds like we're on a good path here and it would definitely be great if in the future by using this new image ko could generate a perfect SBOM for the dependencies. I'm not sure if this makes sense or would just be overkill but it might be good in this this case if ko generated an apk package for the tekton binary and added it with apko.

For the base image changes in general though please be careful as at least for us it is sort of like API. The ship has sailed here but the way we scan and permit images in our infrastructure requires us to pre-bless a set of known base images so this came as a surprise and because of the one letter change in the image name was made harder to figure out. distroless.dev would have been definitely more obvious so agree that makes sense when the time comes.

[imjasonh]

Thanks all. It sounds like we're on a good path here and it would definitely be great if in the future by using this new image ko could generate a perfect SBOM for the dependencies. I'm not sure if this makes sense or would just be overkill but it might be good in this this case if ko generated an apk package for the tekton binary and added it with apko.

That's certainly possible, especially using apko's sibling project melange, which is designed to make building new apks simpler, like apko makes constructing images from apks simpler.

If we did that, we could build Tekton images without ko at all, just using melange+apko, though we'd lose the YAML templating features, and other Go-specific optimizations that ko is aware of.

As for ko building apks and assembling with apko, that would break users who use ko to build on their own arbitrary base images, but perhaps another tool wrapping melange+apko to build optimal application images in such a way is worth exploring.

For the base image changes in general though please be careful as at least for us it is sort of like API. The ship has sailed here but the way we scan and permit images in our infrastructure requires us to pre-bless a set of known base images so this came as a surprise and because of the one letter change in the image name was made harder to figure out. distroless.dev would have been definitely more obvious so agree that makes sense when the time comes.

Thanks for this feedback. I don't think we plan to make major structural changes to the base image again any time soon, but if we do we'll make sure to announce it better.

[imjasonh]

(Closing this as I think the question has been answered, but feel free to keep this discussion going)

[imjasonh]

Also related: #5542 proposes changing the image references from distroless.dev/static (etc) to the equivalent cgr.dev/chainguard/static (etc) refs.

This is not a functional change at all; both URLs point to exactly the same backend and blob storage, and there will never be a diff between distroless.dev/static:$tag and cgr.dev/chainguard/static:$tag or even $digest. But just in case there's a runtime policy about what image refs are allowed, this may require an update to that policy.