Generate and embed build sources #2311

crazy-max · 2021-08-15T16:35:55Z

BuildKit image: crazymax/buildkit:buildsources
Dockerfile frontend image: crazymax/dockerfile:master
Repo example: crazy-max/buildkit-buildsources-test

Create and boot builder:

$ docker buildx create \
  --name buildsources \
  --driver docker-container \
  --driver-opt image=crazymax/buildkit:buildsources \
  --use
$ docker buildx inspect --bootstrap

Build image:

$ docker buildx bake --set *.context=https://github.com/crazy-max/buildkit-buildsources-test.git#master \
  git://github.com/crazy-max/buildkit-buildsources-test.git

Image config generated with the above command:

{
  "architecture": "amd64",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "DOCKER_TLS_CERTDIR=/certs",
      "DOCKER_CLI_EXPERIMENTAL=enabled"
    ],
    "Entrypoint": [
      "docker-entrypoint.sh"
    ],
    "Cmd": [
      "sh"
    ],
    "ArgsEscaped": true,
    "OnBuild": null
  },
  "created": "2021-08-16T13:03:14.6614506Z",
  "history": [
    {
      "created": "2021-04-14T19:19:39.267885491Z",
      "created_by": "/bin/sh -c #(nop) ADD file:8ec69d882e7f29f0652d537557160e638168550f738d0d49f90a7ef96bf31787 in / "
    },
    {
      "created": "2021-04-14T19:19:39.643236135Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
      "empty_layer": true
    },
    {
      "created": "2021-08-16T13:03:03.4051337Z",
      "created_by": "RUN /bin/sh -c apk --update --no-cache add     bash     ca-certificates     openssh-client   && rm -rf /tmp/* /var/cache/apk/* # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:13.2739443Z",
      "created_by": "COPY /opt/docker/ /usr/local/bin/ # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:13.3978023Z",
      "created_by": "COPY /usr/bin/buildctl /usr/local/bin/buildctl # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:13.6345091Z",
      "created_by": "COPY /usr/bin/buildkit* /usr/local/bin/ # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:13.8411541Z",
      "created_by": "COPY /buildx /usr/libexec/docker/cli-plugins/docker-buildx # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:13.9693716Z",
      "created_by": "COPY /opt/docker-compose /usr/libexec/docker/cli-plugins/docker-compose # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:14.0593842Z",
      "created_by": "ADD https://raw.githubusercontent.com/moby/moby/master/README.md / # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:14.4899149Z",
      "created_by": "ENV DOCKER_TLS_CERTDIR=/certs",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2021-08-16T13:03:14.4899149Z",
      "created_by": "ENV DOCKER_CLI_EXPERIMENTAL=enabled",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2021-08-16T13:03:14.4899149Z",
      "created_by": "RUN /bin/sh -c docker --version   && buildkitd --version   && buildctl --version   && docker buildx version   && docker compose version   && mkdir /certs /certs/client   && chmod 1777 /certs /certs/client # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:14.5729468Z",
      "created_by": "COPY rootfs/modprobe.sh /usr/local/bin/modprobe # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:14.6614506Z",
      "created_by": "COPY rootfs/docker-entrypoint.sh /usr/local/bin/ # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2021-08-16T13:03:14.6614506Z",
      "created_by": "ENTRYPOINT [\"docker-entrypoint.sh\"]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2021-08-16T13:03:14.6614506Z",
      "created_by": "CMD [\"sh\"]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    }
  ],
  "moby.buildkit.buildinfo.v0": "eyJzb3VyY2VzIjpbeyJ0eXBlIjoiaW1hZ2UiLCJyZWYiOiJkb2NrZXIuaW8vZG9ja2VyL2J1aWxkeC1iaW46MC42LjFAc2hhMjU2OmE2NTJjZWQ0YTQxNDE5NzdjN2RhYWVkMGEwNzRkY2Q5ODQ0YTc4ZDdkMjYxNTQ2NWIxMmY0MzNhZTZkZDI5ZjAiLCJwaW4iOiJzaGEyNTY6YTY1MmNlZDRhNDE0MTk3N2M3ZGFhZWQwYTA3NGRjZDk4NDRhNzhkN2QyNjE1NDY1YjEyZjQzM2FlNmRkMjlmMCJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL2xpYnJhcnkvYWxwaW5lOjMuMTMiLCJwaW4iOiJzaGEyNTY6MWQzMGQxYmEzY2I5MDk2MjA2N2U5YjI5NDkxZmJkNTY5OTc5NzlkNTQzNzZmMjNmMDE0NDhiNWM1Y2Q4YjQ2MiJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL21vYnkvYnVpbGRraXQ6djAuOS4wIiwicGluIjoic2hhMjU2OjhkYzY2OGU3ZjY2ZGIxYzA0NGFhZGJlZDMwNjAyMDc0MzUxNmE5NDg0ODc5M2UwZjgxZjk0YTA4N2VlNzhjYWIifSx7InR5cGUiOiJpbWFnZSIsInJlZiI6ImRvY2tlci5pby90b25pc3RpaWdpL3h4QHNoYTI1NjoyMWE2MWJlNDc0NGY2NTMxY2I1ZjMzYjBlNmY0MGVkZTQxZmEzYTFiOGM4MmQ1OTQ2MTc4ZjgwY2M4NGJmYzA0IiwicGluIjoic2hhMjU2OjIxYTYxYmU0NzQ0ZjY1MzFjYjVmMzNiMGU2ZjQwZWRlNDFmYTNhMWI4YzgyZDU5NDYxNzhmODBjYzg0YmZjMDQifSx7InR5cGUiOiJnaXQiLCJyZWYiOiJodHRwczovL2dpdGh1Yi5jb20vY3JhenktbWF4L2J1aWxka2l0LWJ1aWxkc291cmNlcy10ZXN0LmdpdCNtYXN0ZXIiLCJwaW4iOiIyNTlhNWFhNWFhNWJiMzU2MmQxMmNjNjMxZmUzOTlmNDc4ODY0MmMxIn0seyJ0eXBlIjoiaHR0cCIsInJlZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9tb2J5L21vYnkvbWFzdGVyL1JFQURNRS5tZCIsInBpbiI6InNoYTI1Njo0MTk0NTUyMDJiMGVmOTdlNDgwZDdmODE5OWIyNmE3MjFhNDE3ODE4YmMwZTJkMTA2OTc1Zjc0MzIzZjI1ZTZjIn1dfQ==",
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116",
      "sha256:0be2b598701c9e5e20f6dc3a023c3d262507d409da178b35eb70e5e992fac7df",
      "sha256:abb5f8b0746d775b23932c49b7ebde441168e461c7472ae63f8529e7cf223d3b",
      "sha256:b29d379fee3b90454773f9f62b8f7acff7a20e2e2cd00b783a2ec1b09c22a9f2",
      "sha256:96093014fc25d50ec7c9afd4c1dde0b16f59c71b23726286340fc12e7fe4d79e",
      "sha256:aed05b2daef84123d803020b09c597a4e951ce4a469f369d1b520bda09d6e9d6",
      "sha256:52850e447962b0942620d4ca65199067aa8f3802dca6da0175508774adcdddb6",
      "sha256:9801c319e1c66c5d295e78b2d3e80547e73c7e3c63a4b71e97c8ca357224af24",
      "sha256:1f64aa2191610843642f99d90bd0880b6dfa24b313a8173918c32f596cbc0338",
      "sha256:72feec5ddd0961db4bcd8bbb697b87a3d3acf6106e48cb96ffd98e3e68ab8bf6",
      "sha256:a1ea3f52b375527e2ac5f4c282f83d72155efd094518a4684d8d3f7f55e82a15"
    ]
  }
}

moby.buildkit.buildinfo.v0 is a single base64 encoded string and will look like this with the above response:

{
  "sources": [
    {
      "type": "image",
      "ref": "docker.io/docker/buildx-bin:0.6.1@sha256:a652ced4a4141977c7daaed0a074dcd9844a78d7d2615465b12f433ae6dd29f0",
      "pin": "sha256:a652ced4a4141977c7daaed0a074dcd9844a78d7d2615465b12f433ae6dd29f0"
    },
    {
      "type": "image",
      "ref": "docker.io/library/alpine:3.13",
      "pin": "sha256:1d30d1ba3cb90962067e9b29491fbd56997979d54376f23f01448b5c5cd8b462"
    },
    {
      "type": "image",
      "ref": "docker.io/moby/buildkit:v0.9.0",
      "pin": "sha256:8dc668e7f66db1c044aadbed306020743516a94848793e0f81f94a087ee78cab"
    },
    {
      "type": "image",
      "ref": "docker.io/tonistiigi/xx@sha256:21a61be4744f6531cb5f33b0e6f40ede41fa3a1b8c82d5946178f80cc84bfc04",
      "pin": "sha256:21a61be4744f6531cb5f33b0e6f40ede41fa3a1b8c82d5946178f80cc84bfc04"
    },
    {
      "type": "git",
      "ref": "https://github.com/crazy-max/buildkit-buildsources-test.git#master",
      "pin": "259a5aa5aa5bb3562d12cc631fe399f4788642c1"
    },
    {
      "type": "http",
      "ref": "https://raw.githubusercontent.com/moby/moby/master/README.md",
      "pin": "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c"
    }
  ]
}

ExporterResponse also contains a new key containerimage.buildinfo if we want to use it in the client.SolveResponse:

{
  "ExporterResponse": {
    "containerimage.buildinfo": "eyJzb3VyY2VzIjpbeyJ0eXBlIjoiaW1hZ2UiLCJyZWYiOiJkb2NrZXIuaW8vZG9ja2VyL2J1aWxkeC1iaW46MC42LjFAc2hhMjU2OmE2NTJjZWQ0YTQxNDE5NzdjN2RhYWVkMGEwNzRkY2Q5ODQ0YTc4ZDdkMjYxNTQ2NWIxMmY0MzNhZTZkZDI5ZjAiLCJwaW4iOiJzaGEyNTY6YTY1MmNlZDRhNDE0MTk3N2M3ZGFhZWQwYTA3NGRjZDk4NDRhNzhkN2QyNjE1NDY1YjEyZjQzM2FlNmRkMjlmMCJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL2xpYnJhcnkvYWxwaW5lOjMuMTMiLCJwaW4iOiJzaGEyNTY6MWQzMGQxYmEzY2I5MDk2MjA2N2U5YjI5NDkxZmJkNTY5OTc5NzlkNTQzNzZmMjNmMDE0NDhiNWM1Y2Q4YjQ2MiJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL21vYnkvYnVpbGRraXQ6djAuOS4wIiwicGluIjoic2hhMjU2OjhkYzY2OGU3ZjY2ZGIxYzA0NGFhZGJlZDMwNjAyMDc0MzUxNmE5NDg0ODc5M2UwZjgxZjk0YTA4N2VlNzhjYWIifSx7InR5cGUiOiJpbWFnZSIsInJlZiI6ImRvY2tlci5pby90b25pc3RpaWdpL3h4QHNoYTI1NjoyMWE2MWJlNDc0NGY2NTMxY2I1ZjMzYjBlNmY0MGVkZTQxZmEzYTFiOGM4MmQ1OTQ2MTc4ZjgwY2M4NGJmYzA0IiwicGluIjoic2hhMjU2OjIxYTYxYmU0NzQ0ZjY1MzFjYjVmMzNiMGU2ZjQwZWRlNDFmYTNhMWI4YzgyZDU5NDYxNzhmODBjYzg0YmZjMDQifSx7InR5cGUiOiJnaXQiLCJyZWYiOiJodHRwczovL2dpdGh1Yi5jb20vY3JhenktbWF4L2J1aWxka2l0LWJ1aWxkc291cmNlcy10ZXN0LmdpdCNtYXN0ZXIiLCJwaW4iOiIyNTlhNWFhNWFhNWJiMzU2MmQxMmNjNjMxZmUzOTlmNDc4ODY0MmMxIn0seyJ0eXBlIjoiaHR0cCIsInJlZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9tb2J5L21vYnkvbWFzdGVyL1JFQURNRS5tZCIsInBpbiI6InNoYTI1Njo0MTk0NTUyMDJiMGVmOTdlNDgwZDdmODE5OWIyNmE3MjFhNDE3ODE4YmMwZTJkMTA2OTc1Zjc0MzIzZjI1ZTZjIn1dfQ==",
    "containerimage.config.digest": "sha256:f9b46e53f841560ba75645d3a8727e8ce3b051da89ffca3d9937130d8e5af02c",
    "containerimage.digest": "sha256:9d3b157491a5dbb009a22a9c8aaef86b04fb247b1b8b5b0c745e32d256d028a2",
    "image.name": "docker.io/library/docker:local"
  }
}

containerimage.buildinfo is also a single base64 encoded string like moby.buildkit.buildinfo.v0.

Multi-platform is also handled:

$ docker buildx bake --set *.context=https://github.com/crazy-max/buildkit-buildsources-test.git#master \
  --set *.platform=linux/amd64,linux/arm64 \
  --set *.output=type=oci,dest=/tmp/docker.tar \
  git://github.com/crazy-max/buildkit-buildsources-test

Each image config will have the same structure but ExporterResponse will have a key for each platform:

{
  "ExporterResponse": {
    "containerimage.buildinfo/linux/amd64": "eyJzb3VyY2VzIjpbeyJ0eXBlIjoiaW1hZ2UiLCJyZWYiOiJkb2NrZXIuaW8vZG9ja2VyL2J1aWxkeC1iaW46MC42LjFAc2hhMjU2OmE2NTJjZWQ0YTQxNDE5NzdjN2RhYWVkMGEwNzRkY2Q5ODQ0YTc4ZDdkMjYxNTQ2NWIxMmY0MzNhZTZkZDI5ZjAiLCJwaW4iOiJzaGEyNTY6YTY1MmNlZDRhNDE0MTk3N2M3ZGFhZWQwYTA3NGRjZDk4NDRhNzhkN2QyNjE1NDY1YjEyZjQzM2FlNmRkMjlmMCJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL2xpYnJhcnkvYWxwaW5lOjMuMTMiLCJwaW4iOiJzaGEyNTY6MWQzMGQxYmEzY2I5MDk2MjA2N2U5YjI5NDkxZmJkNTY5OTc5NzlkNTQzNzZmMjNmMDE0NDhiNWM1Y2Q4YjQ2MiJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL21vYnkvYnVpbGRraXQ6djAuOS4wIiwicGluIjoic2hhMjU2OjhkYzY2OGU3ZjY2ZGIxYzA0NGFhZGJlZDMwNjAyMDc0MzUxNmE5NDg0ODc5M2UwZjgxZjk0YTA4N2VlNzhjYWIifSx7InR5cGUiOiJpbWFnZSIsInJlZiI6ImRvY2tlci5pby90b25pc3RpaWdpL3h4QHNoYTI1NjoyMWE2MWJlNDc0NGY2NTMxY2I1ZjMzYjBlNmY0MGVkZTQxZmEzYTFiOGM4MmQ1OTQ2MTc4ZjgwY2M4NGJmYzA0IiwicGluIjoic2hhMjU2OjIxYTYxYmU0NzQ0ZjY1MzFjYjVmMzNiMGU2ZjQwZWRlNDFmYTNhMWI4YzgyZDU5NDYxNzhmODBjYzg0YmZjMDQifSx7InR5cGUiOiJnaXQiLCJyZWYiOiJodHRwczovL2dpdGh1Yi5jb20vY3JhenktbWF4L2J1aWxka2l0LWJ1aWxkc291cmNlcy10ZXN0LmdpdCNtYXN0ZXIiLCJwaW4iOiIyNTlhNWFhNWFhNWJiMzU2MmQxMmNjNjMxZmUzOTlmNDc4ODY0MmMxIn0seyJ0eXBlIjoiaHR0cCIsInJlZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9tb2J5L21vYnkvbWFzdGVyL1JFQURNRS5tZCIsInBpbiI6InNoYTI1Njo0MTk0NTUyMDJiMGVmOTdlNDgwZDdmODE5OWIyNmE3MjFhNDE3ODE4YmMwZTJkMTA2OTc1Zjc0MzIzZjI1ZTZjIn1dfQ==",
    "containerimage.buildinfo/linux/arm64": "eyJzb3VyY2VzIjpbeyJ0eXBlIjoiaW1hZ2UiLCJyZWYiOiJkb2NrZXIuaW8vZG9ja2VyL2J1aWxkeC1iaW46MC42LjFAc2hhMjU2OmE2NTJjZWQ0YTQxNDE5NzdjN2RhYWVkMGEwNzRkY2Q5ODQ0YTc4ZDdkMjYxNTQ2NWIxMmY0MzNhZTZkZDI5ZjAiLCJwaW4iOiJzaGEyNTY6YTY1MmNlZDRhNDE0MTk3N2M3ZGFhZWQwYTA3NGRjZDk4NDRhNzhkN2QyNjE1NDY1YjEyZjQzM2FlNmRkMjlmMCJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL2xpYnJhcnkvYWxwaW5lOjMuMTMiLCJwaW4iOiJzaGEyNTY6MWQzMGQxYmEzY2I5MDk2MjA2N2U5YjI5NDkxZmJkNTY5OTc5NzlkNTQzNzZmMjNmMDE0NDhiNWM1Y2Q4YjQ2MiJ9LHsidHlwZSI6ImltYWdlIiwicmVmIjoiZG9ja2VyLmlvL21vYnkvYnVpbGRraXQ6djAuOS4wIiwicGluIjoic2hhMjU2OjhkYzY2OGU3ZjY2ZGIxYzA0NGFhZGJlZDMwNjAyMDc0MzUxNmE5NDg0ODc5M2UwZjgxZjk0YTA4N2VlNzhjYWIifSx7InR5cGUiOiJpbWFnZSIsInJlZiI6ImRvY2tlci5pby90b25pc3RpaWdpL3h4QHNoYTI1NjoyMWE2MWJlNDc0NGY2NTMxY2I1ZjMzYjBlNmY0MGVkZTQxZmEzYTFiOGM4MmQ1OTQ2MTc4ZjgwY2M4NGJmYzA0IiwicGluIjoic2hhMjU2OjIxYTYxYmU0NzQ0ZjY1MzFjYjVmMzNiMGU2ZjQwZWRlNDFmYTNhMWI4YzgyZDU5NDYxNzhmODBjYzg0YmZjMDQifSx7InR5cGUiOiJnaXQiLCJyZWYiOiJodHRwczovL2dpdGh1Yi5jb20vY3JhenktbWF4L2J1aWxka2l0LWJ1aWxkc291cmNlcy10ZXN0LmdpdCNtYXN0ZXIiLCJwaW4iOiIyNTlhNWFhNWFhNWJiMzU2MmQxMmNjNjMxZmUzOTlmNDc4ODY0MmMxIn0seyJ0eXBlIjoiaHR0cCIsInJlZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9tb2J5L21vYnkvbWFzdGVyL1JFQURNRS5tZCIsInBpbiI6InNoYTI1Njo0MTk0NTUyMDJiMGVmOTdlNDgwZDdmODE5OWIyNmE3MjFhNDE3ODE4YmMwZTJkMTA2OTc1Zjc0MzIzZjI1ZTZjIn1dfQ==",
    "containerimage.digest": "sha256:0984b39f779f14077f6f7c324fe0f3cd5cc9af7483105281031a5e2308556c39",
    "image.name": "docker.io/library/docker:local"
  }
}

Wonder if we should merge containerimage.buildinfo/* to a single containerimage.buildinfo key.

Walk from result
Redact git and http refs
Integration tests
Exporter option for opt-out

tonistiigi

@thaJeztah @AkihiroSuda Please verify you're ok with the keys in the config structure.

I see some TODOs still left. In addition, needs integration tests.

Wonder if we should merge containerimage.buildinfo/* to a single containerimage.buildinfo key.

Different platforms can have different dependencies.

frontend/dockerfile/dockerfile2llb/convert.go

solver/jobs.go

solver/llbsolver/solver.go

solver/llbsolver/ops/source.go

tonistiigi

We also need to add an exporter option for opt-out.

thaJeztah

Left some random thoughts.

Also wondering; should this be a field in the image, or should they be stored as annotation / label? (label org.mobyproject.buildkit.buildinfo.v0=<data>, or a com.docker. reserved prefix. When doing so, we need to take into account that containerd limits label data to 4k (if I remember correctly)

exporter/containerimage/exptypes/types.go

source/containerimage/pull.go

solver/jobs.go

tonistiigi · 2021-08-16T14:40:02Z

or should they be stored as annotation / label?

This is not human-readable so it should produce extra noise for the user. I guess the inline cache variant also can easily hit size limits.

thaJeztah · 2021-08-16T15:09:30Z

This is not human-readable so it should produce extra noise for the user. I guess the inline cache variant also can easily hit size limits.

Agreed, it's not human readable; that said, labels were to add metadata, which not necessarily has to be human-readable. Wouldn't the same apply to the current field (showing up in docker inspect) or do we want to ignore the field there?

thaJeztah · 2021-08-16T15:13:31Z

Basically, my concern was that History is in the image-spec, but this field is not; https://github.com/opencontainers/image-spec/blob/fe0a24978a6629f4b7159928e538dda36c7cec8e/specs-go/v1/config.go#L63

Will that cause issues?

crazy-max · 2021-08-16T15:20:15Z

Basically, my concern was that History is in the image-spec, but this field is not; https://github.com/opencontainers/image-spec/blob/fe0a24978a6629f4b7159928e538dda36c7cec8e/specs-go/v1/config.go#L63

Will that cause issues?

Don't think it should as this is BuildKit specific and unknown fields in the oci spec are skipped afaik.

tonistiigi · 2021-08-16T15:31:20Z

which not necessarily has to be human-readable.

I do think labels/annotations are human-readable and searchable by definition. This is just noise as it is an encoded field. Possibly very large as well. Cache/opts will be larger than this buidinfo but they should all use the same pattern.

(showing up in docker inspect) or do we want to ignore the field there?

It should not show up in docker inspect atm (same as cache). Inspect does not show raw config.

History is in the image-spec, but this field is not;

Lots of Dockerfile fields are not. This is buildkit specific so we shouldn't use a field that could collide with something else.

solver/jobs.go

solver/llbsolver/ops/source.go

README.md

exporter/containerimage/export.go

tonistiigi · 2021-09-13T06:00:53Z

solver/llbsolver/solver.go

@@ -173,12 +175,20 @@ func (s *Solver) Solve(ctx context.Context, id string, sessionID string, req fro
 			}
 			inp.Ref = workerRef.ImmutableRef

-			dt, err := inlineCache(ctx, exp.CacheExporter, r, session.NewGroup(sessionID))
+			dtbi, err := buildinfo.Merge(ctx, res.BuildInfo(), inp.Metadata[exptypes.ExporterImageConfigKey])


why can't this be just inside the imagewriter exporter?

we need the merged result for the exporter response too (metadata).

but metadata is also returned by the exporter

I guess I done that in the solver to avoid changing the ExporterInstance.Export signature because ResultProxy is not passed to it. Do you see any downside about it?

Not a strict requirement but thought it could be cleaner. You can test in a follow-up and see if it improves.

source/git/gitsource.go

exporter/containerimage/exptypes/types.go

tonistiigi · 2021-09-13T06:10:45Z

util/buildinfo/buildinfo.go

+		case *source.ImageIdentifier:
+			for _, bi := range icbi {
+				// Use original user input from image config
+				if bi.Type == exptypes.BuildInfoTypeDockerImage && bi.Alias == sid.Reference.String() {


this reference does not need TagNameOnly ?

no we want to match the alias which is the actual ref.

What is "actual ref" in here. TagNameOnly would mean we normalize both sides.

The pseudo-ref from alias sorry which is retrieved from ResolveImageConfig in the frontend and will match the one from LLB.

util/buildinfo/buildinfo.go

util/buildinfo/export.go

Signed-off-by: CrazyMax <[email protected]>

crazy-max added the status/do-not-merge label Aug 15, 2021

tonistiigi reviewed Aug 16, 2021

View reviewed changes

solver/llbsolver/ops/source.go Outdated Show resolved Hide resolved

tonistiigi reviewed Aug 16, 2021

View reviewed changes

crazy-max force-pushed the buildsources branch 2 times, most recently from 4e3d1f4 to e9767fb Compare August 16, 2021 07:37

thaJeztah reviewed Aug 16, 2021

View reviewed changes

crazy-max commented Aug 16, 2021

View reviewed changes

solver/jobs.go Outdated Show resolved Hide resolved

crazy-max force-pushed the buildsources branch from 6c85966 to 61158d5 Compare August 16, 2021 12:59

crazy-max force-pushed the buildsources branch from 61158d5 to 7fdd8cc Compare August 16, 2021 14:57

crazy-max force-pushed the buildsources branch from c9bff8b to 1ef01a3 Compare August 16, 2021 15:30

crazy-max force-pushed the buildsources branch from 1633ae6 to 6f3c815 Compare August 16, 2021 19:28

tonistiigi reviewed Aug 17, 2021

View reviewed changes

solver/jobs.go Show resolved Hide resolved

tonistiigi reviewed Aug 17, 2021

View reviewed changes

solver/llbsolver/ops/source.go Outdated Show resolved Hide resolved

crazy-max force-pushed the buildsources branch from 1000f5e to 1bb9a73 Compare August 19, 2021 11:21

crazy-max force-pushed the buildsources branch from f7b3770 to bdb7ddb Compare September 2, 2021 13:23

crazy-max removed the status/do-not-merge label Sep 8, 2021

crazy-max marked this pull request as ready for review September 8, 2021 14:50

crazy-max force-pushed the buildsources branch 3 times, most recently from 125967e to cdd00d2 Compare September 8, 2021 17:48

tonistiigi mentioned this pull request Sep 8, 2021

Can't schedule build of dependent images #2354

Closed

crazy-max requested a review from tonistiigi September 8, 2021 18:49

crazy-max force-pushed the buildsources branch from cdd00d2 to 06d50f1 Compare September 9, 2021 09:56

tonistiigi reviewed Sep 13, 2021

View reviewed changes

crazy-max force-pushed the buildsources branch 3 times, most recently from 07272d1 to d269a38 Compare September 14, 2021 13:26

crazy-max mentioned this pull request Sep 14, 2021

Refactor url redacting util #2363

Merged

crazy-max force-pushed the buildsources branch 7 times, most recently from 4d50ac6 to be2fc19 Compare September 16, 2021 20:28

crazy-max requested a review from tonistiigi September 16, 2021 21:44

tonistiigi reviewed Sep 17, 2021

View reviewed changes

util/buildinfo/export.go Outdated Show resolved Hide resolved

crazy-max force-pushed the buildsources branch from be2fc19 to 5b990eb Compare September 17, 2021 13:13

crazy-max requested a review from tonistiigi September 17, 2021 13:15

tonistiigi reviewed Sep 17, 2021

View reviewed changes

util/buildinfo/export.go Outdated Show resolved Hide resolved

crazy-max force-pushed the buildsources branch from 5b990eb to d990c9c Compare September 17, 2021 22:43

crazy-max requested a review from tonistiigi September 17, 2021 22:46

tonistiigi reviewed Sep 17, 2021

View reviewed changes

util/buildinfo/export.go Outdated Show resolved Hide resolved

Generate and embed build sources

5fcc944

Signed-off-by: CrazyMax <[email protected]>

crazy-max force-pushed the buildsources branch from d990c9c to 5fcc944 Compare September 17, 2021 23:39

crazy-max requested a review from tonistiigi September 17, 2021 23:41

tonistiigi approved these changes Sep 17, 2021

View reviewed changes

tonistiigi merged commit a0afb69 into moby:master Sep 20, 2021

crazy-max deleted the buildsources branch September 20, 2021 17:50

crazy-max mentioned this pull request Oct 25, 2021

vendor buildkit v0.9.1-0.20211025222436-33fb83eb7166 moby/moby#42968

Closed

crazy-max mentioned this pull request Nov 22, 2021

buildinfo: add build attributes and frontend #2476

Merged

crazy-max added this to the v0.10.0 milestone Feb 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate and embed build sources #2311

Generate and embed build sources #2311

crazy-max commented Aug 15, 2021 •

edited

Loading

tonistiigi left a comment

tonistiigi left a comment

thaJeztah left a comment

tonistiigi commented Aug 16, 2021

thaJeztah commented Aug 16, 2021

thaJeztah commented Aug 16, 2021

crazy-max commented Aug 16, 2021 •

edited

Loading

tonistiigi commented Aug 16, 2021

tonistiigi Sep 13, 2021

crazy-max Sep 14, 2021 •

edited

Loading

tonistiigi Sep 14, 2021

crazy-max Sep 16, 2021

tonistiigi Sep 17, 2021

tonistiigi Sep 13, 2021

crazy-max Sep 14, 2021

tonistiigi Sep 14, 2021

crazy-max Sep 15, 2021

Generate and embed build sources #2311

Generate and embed build sources #2311

Conversation

crazy-max commented Aug 15, 2021 • edited Loading

tonistiigi left a comment

Choose a reason for hiding this comment

tonistiigi left a comment

Choose a reason for hiding this comment

thaJeztah left a comment

Choose a reason for hiding this comment

tonistiigi commented Aug 16, 2021

thaJeztah commented Aug 16, 2021

thaJeztah commented Aug 16, 2021

crazy-max commented Aug 16, 2021 • edited Loading

tonistiigi commented Aug 16, 2021

Choose a reason for hiding this comment

crazy-max Sep 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max commented Aug 15, 2021 •

edited

Loading

crazy-max commented Aug 16, 2021 •

edited

Loading

crazy-max Sep 14, 2021 •

edited

Loading