Layer compression/flattening

[dmikusa]

Description

When you build more complicated buildpacks and then pull many of those together into a builder, you can quickly hit the layer limit for a container image.

Proposed solution

There are some seemingly natural points where pack could compress or flatten layers for the images it generates. For example, composite/meta buildpacks. When you are creating a builder or buildpack that consists of other composite/meta buildpacks pack could first flatten all of the layers from the composite buildpack and then create the new image with just a single layer for that composite/meta buildpack (this is opposed to the current behavior which inlines all of the layers from the composite/meta buildpack).

Describe alternatives you've considered

We might be able to hack this together with something like the following but I haven't tried it.

1. pack buildpack package a composite buildpack
2. somehow compress/flatten the layers on the image generated in step 1.
3. repeat for all buildpacks in the builder
4. pack builder create the builder

I worry that this will create problems with metadata though. 🤔

Additional context

This is a fairly urgent need for Paketo. We have hit the layer limit in our Paketo builders and are struggling to add new functionality because we can't add any more layers.

[loewenstein]

One thought that comes up. We are talking about a balancing thing, don't we. On the one hand side, small individual layers support commonly used buildpacks to be packaged without duplication. On the other hand side, they increase the layers and 127 is a hard and pretty low limit.

If pack learns how to compress builder layers, it might be beneficial if it also learns how to optimise the balance - maybe using the system buildpacks feature from RFC #101?

[dmikusa]

On the one hand side, small individual layers support commonly used buildpacks to be packaged without duplication.

In practice, this doesn't always happen though. At least in Paketo, what happens is that we see slightly different versions of component buildpacks used by different composite buildpacks which cannot be deduplicated.

I get what you're saying though. I don't think this should be the default behavior, but I do think pack should support a flag that can enable this behavior.

[jjbustamante]

I think @ForestEckhardt mentioned this in WG at some point last year. Is it something like squash a commit on git?

[dmikusa]

Yes, basically. A builder is made up of a base build image plus layers and layers of buildpacks on top of that. If you add enough buildpacks, you'll eventually hit the layer limit.

Ex: a paketo builder looks roughly like this

| paketo Java buildpack|
| paketo Go buildpack |
| paketo Python buildpack |
| paketo ... buildpacks |
| paketo base builder|

but since all of the buildpacks above are composite buildpacks, they get expanded and you end up with a much larger set of buildpacks and a much larger number of layers.

Because of the way buildpacks are laid out, I think we can be reasonably sure that you won't have overlapping files across layers (i.e. across buildpacks), so what I'd like to see when pack creates a builder image with composite buildpacks, is a set of layers like what I described above. One layer per composite buildpack.

Maybe compressing/flattening layers isn't the best way to describe it. Another way to look at it would be when pack builds the layers, just build fewer layers. Put all of the expanded buildpacks from a composite buildpack into one layer.

Hope that helps to clarify. Thanks

[jjbustamante]

Hi @dmikusa!

cc @matthewmcnew @natalieparellano (I copied you because we were discussing about this)

I've being working on this, and I created a draft PR, could you take a look at the pull request description? I tried to show there the current behavior/output and I will like to double check that is what you had in mind.

Another question, I am also working on adding a similar behavior to the command pack buildpack package I also added a --flatten-layers flag.

Here is an example of what is doing.

Based on our current samples repository I created a meta-buildpack called samples/my-programming-lang which has the following buildpack.toml

# Buildpack API version
api = "0.2"

# Buildpack ID and metadata
[buildpack]
id = "samples/my-programming-lang"
version = "0.0.1"
name = "My programming languages buildpacks"
homepage = "https://github.com/buildpacks/samples/tree/main/buildpacks/my-programming-lang"

# Order used for detection
[[order]]
[[order.group]]
id = "samples/java-maven"
version = "0.0.1"

[[order.group]]
id = "samples/kotlin-gradle"
version = "0.0.1"

[[order.group]]
id = "samples/ruby-bundler"
version = "0.0.1"

I packaged this buildpack and save it into my docker daemon, as expected, it has 4 layers as we can see in this image:

I now update the hello-universe meta-buildpack to include this new meta-buildpack, adding the following to the hello-universe/buildpack.toml

# added

[[order.group]]
id = "samples/my-programming-lang"
version = "0.0.1"

The package.toml file to package the hello-universe meta-buildpack looks like:

[buildpack]
uri = "samples/buildpacks/hello-universe/"

[[dependencies]]
uri = "samples/buildpacks/hello-moon"

[[dependencies]]
uri = "docker://cnbs/sample-package:hello-world"

[[dependencies]]
uri = "docker://my-programming-package"

After that, I package the hello-universe meta-buildpack using the new flag and saving (for now) on disk

pack  buildpack package my-hello-universe-flatten.cnb --config ./package.toml --format file  --flatten-layers

Checking the labels in the Config file

cat blobs/sha256/dc7dc9552d981001a660084d310a9435644fe5a6a3dad6f0ddaae733d5a685db | jq ' .config.Labels | ."io.buildpacks.buildpack.layers" | fromjson'

I get the following:

{
  "samples/hello-moon": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [ ... ],
      "layerDiffID": "sha256:0a29dd5e581a43de61913fc348606faf450c7ba1557f8c1d5a961cbb335bcc17",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/hello-moon",
      "name": "Hello Moon Buildpack"
    }
  },
  "samples/hello-universe": {
    "0.0.1": {
      "api": "0.2",
      "order": [...],
      "layerDiffID": "sha256:a7752f36f04c03e6dda5a21741b1b12d8aa564556b317193d72b76286fe00464",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/hello-universe",
      "name": "Hello Universe Buildpack"
    }
  },
  "samples/hello-world": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [...],
      "layerDiffID": "sha256:18e9caa9f9836f546d7c5c0ed6ce0f87c99b3987b686eac40f244bb1e91db471",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/hello-world",
      "name": "Hello World Buildpack"
    }
  },
  "samples/java-maven": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [...],
      "layerDiffID": "sha256:32f20af929eda77a48ab04a2851c657d06c43a6e63106a2c536295d6c2e35479",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/java-maven",
      "name": "Sample Java Maven Buildpack"
    }
  },
  "samples/kotlin-gradle": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [...],
      "layerDiffID": "sha256:32f20af929eda77a48ab04a2851c657d06c43a6e63106a2c536295d6c2e35479",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/kotlin-gradle",
      "name": "Sample Kotlin Gradle Buildpack"
    }
  },
  "samples/my-programming-lang": {
    "0.0.1": {
      "api": "0.2",
      "order": [
        {
          "group": [
            {
              "id": "samples/java-maven",
              "version": "0.0.1"
            },
            {
              "id": "samples/kotlin-gradle",
              "version": "0.0.1"
            },
            {
              "id": "samples/ruby-bundler",
              "version": "0.0.1"
            }
          ]
        }
      ],
      "layerDiffID": "sha256:32f20af929eda77a48ab04a2851c657d06c43a6e63106a2c536295d6c2e35479",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/my-programming-lang",
      "name": "My programming languages buildpacks"
    }
  },
  "samples/ruby-bundler": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [...],
      "layerDiffID": "sha256:32f20af929eda77a48ab04a2851c657d06c43a6e63106a2c536295d6c2e35479",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/ruby-bundler",
      "name": "Sample Ruby Buildpack"
    }
  }
}

I removed some data for simplicity, but we can see that all the layers for meta-buidpack my-programming-lang were flatten into one layer (sha256:32f20af929eda77a48ab04a2851c657d06c43a6e63106a2c536295d6c2e35479), and the reason for that, is because when we iterate over the dependencies in the package.toml file for hello-universe we found a meta-buildpack there and the flag enables the compression.

The final manifest has 4 layers:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 4089,
    "digest": "sha256:dc7dc9552d981001a660084d310a9435644fe5a6a3dad6f0ddaae733d5a685db"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 3748,
      "digest": "sha256:77e05dd57bf427eb4f972075f514b9ffa26a8804a148d85b4675fe04252cbf52"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 541,
      "digest": "sha256:48440a87a2e4637e5704e993f569a89ad6331289a13eb4cbb61ad125d94b8f8f"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1051,
      "digest": "sha256:d35cb095ab7fc25ca4bb0e61c3f802d7edb8505545464399f23ed98831a7556c"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1143,
      "digest": "sha256:df91948a4b0953fdac230b90cc3d7370eddb0e3f14a3b040a7402da22fa916b7"
    }
  ]
}

• 1 layer for hello-universe meta bp
• 1 layer for hello-world bp
• 1 layer for hello0moon bp
• 1 layer for my-programming-lang meta bd and its 3 buidpacks

I just want to validate this behavior is correct.

Thanks

[natalieparellano]

Just thinking out loud here,

Similar to how we have pack buildpack inspect --depth to control the max depth for order display, we may want a --depth option to control how many nested buildpacks we flatten. It might be nice to have the option to produce a buildpack package that is a single layer, or one that flattens nested meta buildpacks as you describe.

[dmikusa]

This is fantastic, thanks for working on this @jjbustamante! I will give things a try and let you know how it goes.