Push signatures first

[mattmoor]

Generally whenever distroless publishes new images, we see a rash of CI failures (e.g. cosigned e2e tests) because the tagged images aren't signed.

Looking through, distroless is also unfortunately signing tags (bad!):

distroless/cloudbuild_cosign.sh

Lines 12 to 48 in 3fe389d

	# Sign 'latest' images with cosign
	for distro_suffix in "" -debian10 -debian11; do
	cosign sign "$@" gcr.io/$PROJECT_ID/static${distro_suffix}:nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/static${distro_suffix}:latest
	cosign sign "$@" gcr.io/$PROJECT_ID/static${distro_suffix}:debug-nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/static${distro_suffix}:debug
	cosign sign "$@" gcr.io/$PROJECT_ID/base${distro_suffix}:nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/base${distro_suffix}:latest
	cosign sign "$@" gcr.io/$PROJECT_ID/base${distro_suffix}:debug-nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/base${distro_suffix}:debug
	cosign sign "$@" gcr.io/$PROJECT_ID/cc${distro_suffix}:nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/cc${distro_suffix}:latest
	cosign sign "$@" gcr.io/$PROJECT_ID/cc${distro_suffix}:debug-nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/cc${distro_suffix}:debug
	done
	# python, java and nodejs are debian11 only
	cosign sign "$@" gcr.io/$PROJECT_ID/python3-debian11:nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/python3-debian11:latest
	cosign sign "$@" gcr.io/$PROJECT_ID/python3-debian11:debug-nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/python3-debian11:debug
	for distro_suffix in "" -debian11; do
	cosign sign "$@" gcr.io/$PROJECT_ID/nodejs${distro_suffix}:latest
	cosign sign "$@" gcr.io/$PROJECT_ID/nodejs${distro_suffix}:debug
	cosign sign "$@" gcr.io/$PROJECT_ID/nodejs${distro_suffix}:16
	cosign sign "$@" gcr.io/$PROJECT_ID/nodejs${distro_suffix}:16-debug
	done
	for java_version in -base 11 17; do
	cosign sign "$@" gcr.io/$PROJECT_ID/java${java_version}-debian11:latest
	cosign sign "$@" gcr.io/$PROJECT_ID/java${java_version}-debian11:nonroot
	cosign sign "$@" gcr.io/$PROJECT_ID/java${java_version}-debian11:debug
	cosign sign "$@" gcr.io/$PROJECT_ID/java${java_version}-debian11:debug-nonroot
	done

This is part of why ko publishes SBOMs before publishing the images, although here it's tricky because a lack of integration in the build tooling.

cc @loosebazooka @imjasonh @jonjohnsonjr @dlorenc

[imjasonh]

The cosign version is also old:

distroless/cloudbuild.yaml

Line 53 in 1a3e90f

- name: gcr.io/projectsigstore/cosign:v1.3.1@sha256:3cd9b3a866579dc2e0cf2fdea547f4c9a27139276cc373165c26842bc594b8bd

I'm not sure it's possible to sign the platform-specific images before they're pushed (I'm also not sure it's useful), but I think it might be possible to sign the multi-platform manifest list we all know and love before pushing it.

That's done here:

distroless/cloudbuild_docker.sh

Line 15 in 1a3e90f

docker_manifest() {

Between docker manifest create and docker manifest push you could take the digest of that manifest with docker manifest inspect $_image | sha256sum -, sign that, then push the manifest.

[imjasonh]

docker manifest inspect helpfully inserts a trailing newline, which isn't there when you push it. So:

  _digest=$(printf "%s" "$(docker manifest inspect ${_image})" | sha256sum -)                                                                                                                             
  cosign sign ${_image}@${_digest}
  docker manifest push ${_image}

The next trick is going to be having a GCB step that has both docker and cosign available. Today that's two steps, one that has docker, and one that has cosign -- there's also one that installs and invokes jq.

[loosebazooka]

Sgtm. What I really want to do is just stage every build and then crane copy (or cosign copy) over the images to prod gcr. But that can be for another day

[mattmoor]

Staging also sounds good, but we should sign the digests wherever we sign them 😅

[loosebazooka]

this is fixed by rules_oci