Buildah reuse gzip layers when "--compression-format zstd" assigned #4613

Placeboy · 2023-02-21T07:36:44Z

Description
I pushed my image with flag "--compression-format zstd", but when I checked its manifest I found that it just reused gzip layers.

Steps to reproduce the issue:

Install buildah from scratch:

sudo mkdir ~/buildah
cd ~/buildah
export GOPATH=`pwd`
sudo git clone https://github.com/containers/buildah ./src/github.com/containers/buildah
cd ./src/github.com/containers/buildah
sudo make BUILDTAGS=containers_image_openpgp
sudo make install
sudo cp /usr/local/bin/buildah /usr/bin/

Build image:

sudo buildah bud -t placeboy/buildah-from-dockerfile .

Here is my Dockerfile:

FROM node:10
WORKDIR /usr/src/app
RUN npm init -y
RUN npm install express --save
COPY HelloWorld.js .
CMD [ "node", "HelloWorld.js" ]

Push with no compression-format assigned:

sudo buildah push placeboy/buildah-from-dockerfile

Check manifest:

$ sudo docker buildx imagetools inspect --raw placeboy/buildah-from-dockerfile | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:0ee844816c31997f60ad0c6eeab16094d044b5c2e06319bfbf6d845b88626563",
    "size": 7735
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:76b8ef87096fa726adbe8f073ef69bb5664bac19474c5cce4dd69e08a234903b",
      "size": 45380037
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:2e2bafe8a0f40509cc10249087268e66a662e437f10e9598a09abb5687038a57",
      "size": 11286411
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b53ce1fd2746e8d2037f1b0b91ddea0cc7411eb3e5949fe10c0320aca8f7392b",
      "size": 4342420
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:84a8c1bd5887cc4a89e1f286fed9ee31ce12dba9f6813cf14082ada3e9ab6f63",
      "size": 49786874
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:7a803dc0b40fcd10faee3fb3ebb2d7aaa88500520e6295295f5163c4bb48548b",
      "size": 214347112
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b800e94e7303e276b8fb4911a40bfe28f46180d997022c69bf1ee02fb7b86721",
      "size": 4190
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:0da9fbf60d485c74d153bfc7562f34533550e3dd8aa78f5e8c2476ed500e0e73",
      "size": 21417193
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:04dccde934cf0d6ab9d303fe73b36bc4ed84329d9d562049a69a5d38afb83f14",
      "size": 2484048
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:73269890f6fdf96184d0fdb0afe3cf697b0c8dbe1e7a077f0fab445d74372c78",
      "size": 294
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d04736ed33aa34ba551e9525a9b3c3527ed683afa212a39c581c3f5aa7231a00",
      "size": 1891961
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.digest": "sha256:686e0e859358f28bfe1641e1627549b9cd0ad74f222b953d74209213488c6858",
    "org.opencontainers.image.base.name": "docker.io/library/node:10"
  }
}

Push the image again, with zstd compression-format assigned:

sudo buildah push --compression-format zstd placeboy/buildah-from-dockerfile

Check manifest again now:

$ sudo docker buildx imagetools inspect --raw placeboy/buildah-from-dockerfile | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:0ee844816c31997f60ad0c6eeab16094d044b5c2e06319bfbf6d845b88626563",
    "size": 7735
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:76b8ef87096fa726adbe8f073ef69bb5664bac19474c5cce4dd69e08a234903b",
      "size": 45380037
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:2e2bafe8a0f40509cc10249087268e66a662e437f10e9598a09abb5687038a57",
      "size": 11286411
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b53ce1fd2746e8d2037f1b0b91ddea0cc7411eb3e5949fe10c0320aca8f7392b",
      "size": 4342420
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:84a8c1bd5887cc4a89e1f286fed9ee31ce12dba9f6813cf14082ada3e9ab6f63",
      "size": 49786874
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:7a803dc0b40fcd10faee3fb3ebb2d7aaa88500520e6295295f5163c4bb48548b",
      "size": 214347112
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b800e94e7303e276b8fb4911a40bfe28f46180d997022c69bf1ee02fb7b86721",
      "size": 4190
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:0da9fbf60d485c74d153bfc7562f34533550e3dd8aa78f5e8c2476ed500e0e73",
      "size": 21417193
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:04dccde934cf0d6ab9d303fe73b36bc4ed84329d9d562049a69a5d38afb83f14",
      "size": 2484048
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:73269890f6fdf96184d0fdb0afe3cf697b0c8dbe1e7a077f0fab445d74372c78",
      "size": 294
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d04736ed33aa34ba551e9525a9b3c3527ed683afa212a39c581c3f5aa7231a00",
      "size": 1891961
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.digest": "sha256:686e0e859358f28bfe1641e1627549b9cd0ad74f222b953d74209213488c6858",
    "org.opencontainers.image.base.name": "docker.io/library/node:10"
  }
}

Describe the results you expected:

$ sudo docker buildx imagetools inspect --raw placeboy/buildah-from-dockerfile | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:29fc3b654b32f36604ec1d4473ed98635e43fae2bd9137392b4097a15c40ce49",
    "size": 7734
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:342dd87184b7f48dba284f855fd1acb181f9822a205840066597760040b3c39d",
      "size": 36063577
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:3043dc57d5854957650610db9881dedde2b563794c8440a4b0313a78acb7a294",
      "size": 9726868
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:17a80d72fc5e7631a931da627fe82cf316246180a38f61ef7e8ac2c709929250",
      "size": 3239525
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:dc063313fc998715c944a9216e0fc96def311a2484d593b174cf0bce3635e7b2",
      "size": 38695908
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:ec271c8a034a45dc4a8d0c0c415b6fc41ed808762314ea51ea21ff0dbe36ab68",
      "size": 183371283
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:c659975e02f6477ae0e36f613b6baecc5687a6271d5e08d834cc8ba47d7188f3",
      "size": 3647
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:6427d59e665293132649d4b660e6b59611339ac7d3162ee7f4a2611601583ddf",
      "size": 19422790
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:3188faec5ac3e6d2c329d58a4b409cba0dadf2cc16d28381292d42faf92f1e03",
      "size": 2450837
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:f8467ea4a05e4636f7163d7f55e009c829ac6913af95b8589d7512030443694c",
      "size": 284
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:dcc2ac1ec89846d6267cb56b94235e5e6539084b1c848e13027b31397d4ae16e",
      "size": 1792789
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.digest": "sha256:686e0e859358f28bfe1641e1627549b9cd0ad74f222b953d74209213488c6858",
    "org.opencontainers.image.base.name": "docker.io/library/node:10"
  }
}

Output of rpm -q buildah or apt list buildah:

package buildah is not installed

Output of buildah version:

WARN[0000] Failed to decode the keys ["storage.options.override_kernel_check"] from "/etc/containers/storage.conf"
Version:         1.30.0-dev
Go Version:      go1.20
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.24.1
Git Commit:      5a63d7f61903b1da22343bc021a7f7d64c1c2895
Built:           Tue Feb 21 15:03:18 2023
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of cat /etc/*release:

CentOS Linux release 7.8.2003 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.8.2003 (Core)
CentOS Linux release 7.8.2003 (Core)

Output of uname -a:

Linux bjrz-c497.lf 4.18.0-2.6.4.2.kwai.x86_64 #1 SMP Tue Oct 25 12:40:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""

# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"

# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

The text was updated successfully, but these errors were encountered:

rhatdan · 2023-02-21T19:02:15Z

@giuseppe @vrothberg @mtrmac PTAL

mtrmac · 2023-02-21T19:28:19Z

Yes; this is currently working as designed. The compression algorithm choice only applies when we compress blobs, and we prefer to reuse blobs rather than to push and compress. (And there are no available options to choose a different behavior, right now.)

That will need to change, or at least give callers more options, to allow correctly creating multi-variant Zstd images and the like; and that’s being worked on in the nearer term.

vrothberg · 2023-02-22T08:23:46Z

I think we should leave the issue open until the multi-variant images are done.

github-actions · 2023-03-25T00:09:45Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2023-03-27T18:44:37Z

@vrothberg @flouthoc @mtrmac @giuseppe any movement on this?

flouthoc · 2023-04-17T12:56:14Z

@Placeboy I believe this will be fixed as a byproduct of this PR containers/image#1875, i was able to reproduce the problem you faced and part of the commit containers/image#1875 does what you want.

mtrmac · 2023-04-17T17:41:43Z

That’s not 100% obvious to me. In the Zstd conversion case, we clearly don’t want to reuse differently-compressed layers.

In this case, there’s still a conceptual ambiguity as to whether --compression-format should apply only to those layers that we must compress, or whether it should result in all layers using that format. We might need a new option to choose between the two semantics.

We can also decide to only implement one of the two, or even to just change the existing option’s semantics to the other alternative, and I don’t feel very strongly about that (although I’m slightly biased in favor of not breaking performance characteristics of existing options). Primarily, let’s recognize that this is a choice we can make, and merging code without consciously deciding to make a choice is still making that choice.

github-actions · 2023-05-18T00:09:30Z

A friendly reminder that this issue had no activity for 30 days.

sanmai-NL · 2023-06-22T09:13:43Z

@mtrmac See this note: https://docs.docker.com/build/exporters/#compression

Use the force-compression=true option to force re-compressing layers imported from a previous image, if the requested compression algorithm is different from the previous compression algorithm.

github-actions · 2023-07-25T00:11:57Z

A friendly reminder that this issue had no activity for 30 days.

flouthoc · 2023-07-25T06:03:46Z

This will be closed once we wire, requireCompressionFormatMatch from containers/image#2047 with CLI, I will take this work immediately after once I am done with entire manifest API change.

ForceCompressionFormat allows end users to force selected compression format (set in DestinationCtx.CompressionFormat) which ensures that while copying blobs of other compression algorithms are not reused. Following flag is a frontend wrapper for: containers#2023 Will help in: * containers/buildah#4613 * containers/podman#18660 Signed-off-by: Aditya R <[email protected]>

Adds support for `--force-compression` which allows end-users to force push blobs with the selected compresison in `--compression` option, in order to make sure that `blobs` of other compression on registry are not reused. Is equivalent to: `force-compression` here: https://docs.docker.com/build/exporters/#compression Closes: containers#4613 Also Implements: `--compression-format` and `--compression-level` for `manifest push` just like `podman`'s `manifest push` Signed-off-by: Aditya R <[email protected]>

flouthoc · 2023-08-15T05:15:45Z

PR #4973 closes this.

Adds support for `--force-compression` which allows end-users to force push blobs with the selected compresison in `--compression` option, in order to make sure that `blobs` of other compression on registry are not reused. Is equivalent to: `force-compression` here: https://docs.docker.com/build/exporters/#compression Closes: containers#4613 Also Implements: `--compression-format` and `--compression-level` for `manifest push` just like `podman`'s `manifest push` Signed-off-by: Aditya R <[email protected]>

Placeboy changed the title ~~Buildah does not push all layers with zstd compression-format~~ Buildah reuse gzip layers when "--compression-format zstd" assigned Feb 21, 2023

rhatdan closed this as completed Feb 21, 2023

vrothberg reopened this Feb 22, 2023

github-actions bot added the stale-issue label Mar 25, 2023

rhatdan removed the stale-issue label Mar 27, 2023

github-actions bot added the stale-issue label May 18, 2023

github-actions bot removed the stale-issue label Jun 23, 2023

mtrmac mentioned this issue Jul 19, 2023

copy/multiple: instanceCopyCopy honor UpdateCompressionAlgorithms containers/image#2047

Merged

github-actions bot added the stale-issue label Jul 25, 2023

github-actions bot removed the stale-issue label Jul 26, 2023

flouthoc mentioned this issue Aug 2, 2023

copy: add support for ForceCompressionFormat containers/image#2068

Merged

mtrmac mentioned this issue Aug 10, 2023

libimage: add support for ForceCompressionFormat for image copier and manifest containers/common#1602

Merged

flouthoc mentioned this issue Aug 11, 2023

buildah/push/manifest-push: add support for --force-compression to prevent reusing other blobs #4973

Merged

openshift-merge-robot closed this as completed in #4973 Aug 15, 2023

github-actions bot added the locked - please file new issue/PR label Nov 14, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buildah reuse gzip layers when "--compression-format zstd" assigned #4613

Buildah reuse gzip layers when "--compression-format zstd" assigned #4613

Placeboy commented Feb 21, 2023 •

edited

Loading

rhatdan commented Feb 21, 2023

mtrmac commented Feb 21, 2023

vrothberg commented Feb 22, 2023

github-actions bot commented Mar 25, 2023

rhatdan commented Mar 27, 2023

flouthoc commented Apr 17, 2023 •

edited

Loading

mtrmac commented Apr 17, 2023

github-actions bot commented May 18, 2023

sanmai-NL commented Jun 22, 2023

github-actions bot commented Jul 25, 2023

flouthoc commented Jul 25, 2023

flouthoc commented Aug 15, 2023

Buildah reuse gzip layers when "--compression-format zstd" assigned #4613

Buildah reuse gzip layers when "--compression-format zstd" assigned #4613

Comments

Placeboy commented Feb 21, 2023 • edited Loading

rhatdan commented Feb 21, 2023

mtrmac commented Feb 21, 2023

vrothberg commented Feb 22, 2023

github-actions bot commented Mar 25, 2023

rhatdan commented Mar 27, 2023

flouthoc commented Apr 17, 2023 • edited Loading

mtrmac commented Apr 17, 2023

github-actions bot commented May 18, 2023

sanmai-NL commented Jun 22, 2023

github-actions bot commented Jul 25, 2023

flouthoc commented Jul 25, 2023

flouthoc commented Aug 15, 2023

Placeboy commented Feb 21, 2023 •

edited

Loading

flouthoc commented Apr 17, 2023 •

edited

Loading