Add local-image-proxy command to proxy container image fetches

[cgwalters]

This command runs a webserver that can fetch manifests and blobs;
it's a bit like an OCI/Docker Distribution API except we can fetch
from multiple registries. It may be more useful to think of it
as a streaming HTTP version of podman pull.

The primary intended use case is for ostree+containers integration.
What we want is to fetch and stream the underlying tarball from a container
image that will be unpacked and used for operating system updates.
The ostree code here will do things like validate the embedded
OSTree signatures inside the tarball.

Design constraints/goals:

• This should not read or write to the containers/image storage; we
  want to stream the result and have other code unpack it
  into the target ostree repo. I don't want to have to think about
  any potential interactions between the operating system update mechanism/storage
  and container storage.
• I want to stream instead of pulling the whole image to a temporary
  directory and then re-parse it.
• But, crucially we do want to reuse any configuration in /etc/containers/registries.conf
  particularly around mirroring. In particular for e.g. OpenShift
  it must work to set up Image Content Source Policy to pull from
  a local registry mirror of the CoreOS update stream.
• We want to reuse the containers/image default handling for
  authentication (~/.docker/config.json etc.)
• I'd really rather not vendor in all of containers/image into
  another program and have to keep it up to date
• ostree is in C and Rust, adding Go in-process would be a huge change
• By having this out of process, it can easily be runnable fully unprivileged under e.g. a transient
  uid to fully ensure it has no effect on the system and can be sandboxed.

It's of course a bonus here if we also support containers signing, etc.

The expected primary use case is that ostree tooling allocates a socketpair()
and passes it down to skopeo, which will run a webserver on that anonymous
socketpair.

For ease of debugging/testing, we also support listening on a localhost TCP port.

More information here: https://github.com/ostreedev/ostree-rs-ext/tree/b84955aa170bf4b5518603547f33d1aec7719f44#pulling-an-ostree-container-directly

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cgwalters
To complete the pull request process, please assign mheon after the PR has been reviewed.
You can assign the PR to them by writing /assign @mheon in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cgwalters]

Other notes

• Marking this as draft because it's just something I hacked up quickly and it seems to work, but I'm looking for feedback before I do a lot more work
• AFAICS, the containers/image code only validates the blob sha256 and diffID uncompressed sha256 when copying between backends; so doing that stuff seems like it'd involve an API that is basically "pull to a Writer but do validation"? Or hmm, maybe we can expose an API that saves to docker-archive?
• I didn't actually test that this honors registry mirroring etc.
• I also don't know the podman test suite at all and this obviously needs one

[cgwalters]

Hmm and a deeper problem here is I really want this command to be runnable inside a container; all it does is HTTP requests. But it looks like all of the containers/common machinery entangles things like "parse /etc/containers/ stuff" with "find conmon and load storage" and so it errors out around /etc/subuid when running nonroot inside a toolbox container. OK, looks like I can kind of work around this by passing --root $(mktemp -d).

[rhatdan]

Should this be a podman system or podman image command? Not a big fan of the local-image-proxy name.

podman system proxy?

podman image proxy?

[mheon]

This feels a bit more Skopeo than Podman... A special transport so skopeo copy can directly produce the output format you want?

[cgwalters]

I actually originally wrote this patch for skopeo, then changed my mind because it feels like the long term direction is to avoid having lots of binaries vendoring and re-exporting the containers/ library stack. We need to keep podman and cri-o up to date for example but skopeo isn't currently critical path for anything.

That said, I agree it does more clearly fit into skopeo so...happy to close this and reopen there if that's the consensus.

[mheon]

There is precedent for including other commands in Podman (Buildah and podman build)... Having skopeo copy exposed as, say, podman image copy would not be unprecedented. I wouldn't want to reimplement, to be clear, just call the skopeo code.

[cgwalters]

OK will move to skopeo.

[cgwalters]

This initiative moved to https://github.com/cgwalters/container-image-proxy for now, but I still plan to propose for skopeo at some point probably soon.