Add --flatten, --depth and flatten-exclude flags to pack builder create and pack buildpack package command

[jjbustamante]

Summary

This PR adds new flags to:

• pack builder create
• pack buildpack package

The flags will try to flatten the layers in the final image.

for both commands the following flags were added:

      --depth int                   Max depth to flatten compose buildpacks layers.
                                    Omission of this flag or values < 0 will flatten the entire tree. (default -1)
      --flatten                     Flatten the composite buildpacks layers
  -e, --flatten-exclude strings     Buildpack to exclude for being flatten. Use buildpack by id and version in the form of '<buildpack>@<version>'

These flags can be used with the following combinations:

• --flatten only OR with --depth < 0: it will flatten ALL the buildpacks layers into 1 final layer, because the default value for --depth is < 0
• --flatten and --depth = 1: it will flatten only the top level of composites to be flattened (i.e. composites directly referenced from either the builder or directly referenced from a composite if building a buildpack)
• --flatten and --depth = 2: it will flatten the top 2 levels of composites buildpacks

And so on

In case the users want to EXCLUDE some buildpacks for being flatten, they can use the --flatten-exclude flag in combination with the previous one to exclude any buildpack they want. The buildpack MUST be specify using the format flatten-exclude '<buildpack>@<version>'

Example

For testing the feature I used the buildpacks defined in our sample repository and a builder.toml with the following content:

# Buildpacks to include in builder
[[buildpacks]]
uri = "samples/buildpacks/hello-processes"

[[buildpacks]]
# Packaged buildpacks to include in builder;
# the "hello-universe" package contains the "hello-world" and "hello-moon" buildpacks
uri = "docker://cnbs/sample-package:hello-universe"

# Order used for detection
[[order]]
    # This buildpack will display build-time information (as a dependency)
    [[order.group]]
    id = "samples/hello-world"
    version = "0.0.1"

    # This buildpack will display build-time information (as a dependant)
    [[order.group]]
    id = "samples/hello-moon"
    version = "0.0.1"

    # This buildpack will create a process type "sys-info" to display runtime information
    [[order.group]]
    id = "samples/hello-processes"
    version = "0.0.1"

# Stack that will be used by the builder
[stack]
id = "io.buildpacks.samples.stacks.bionic"
# This image is used at runtime
run-image = "cnbs/sample-stack-run:bionic"
# This image is used at build-time
build-image = "cnbs/sample-stack-build:bionic"

Graphically we can imagine the example as follow

Output

$ pack builder create cnbs/sample-builder:alpine --config builder.toml --flatten

Here we can see all the buildpacks were flatten into one layer:

$ pack builder create cnbs/sample-builder:alpine --config builder.toml --flatten --depth 0

It will flatten the hello-universe in one layer and

And the hello-processes in another layer

But if we want to EXCLUDE we can do

$ pack builder create cnbs/sample-builder:alpine --config builder.toml --verbose --flatten --depth 0 --flatten-exclude samples/[email protected]

Metadata

Inspecting the metadata io.buildpacks.buildpack.layers

> docker inspect cnbs/sample-builder:alpine | jq '.[] | .Config.Labels | ."io.buildpacks.buildpack.layers" | fromjson'

shows the layerDiffID value is the same for the squashed buildpacks

{
  "samples/hello-extensions": {
    "0.0.1": {
      "api": "0.9",
      "stacks": [
        {
          "id": "*"
        }
      ],
      "layerDiffID": "sha256:baf4f7c28c15dbaadc953bb6d98ce9e83473e842fe92956ea2a328495237ddfc",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/hello-extensions",
      "name": "Hello Extensions Buildpack"
    }
  },
  "samples/hello-moon": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [
        {
          "id": "io.buildpacks.samples.stacks.jammy"
        },
        {
          "id": "io.buildpacks.samples.stacks.alpine"
        },
        {
          "id": "io.buildpacks.stacks.jammy"
        },
        {
          "id": "*"
        }
      ],
      "layerDiffID": "sha256:4ad60d1ea1836600073b50ae066e946068fdba9babbee257d3584cbe1ceb9cb7",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/hello-moon",
      "name": "Hello Moon Buildpack"
    }
  },
  "samples/hello-universe": {
    "0.0.1": {
      "api": "0.2",
      "order": [
        {
          "group": [
            {
              "id": "samples/hello-world",
              "version": "0.0.1"
            },
            {
              "id": "samples/hello-moon",
              "version": "0.0.1"
            }
          ]
        }
      ],
      "layerDiffID": "sha256:4ad60d1ea1836600073b50ae066e946068fdba9babbee257d3584cbe1ceb9cb7",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/hello-universe",
      "name": "Hello Universe Buildpack"
    }
  },
  "samples/hello-world": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [
        {
          "id": "io.buildpacks.samples.stacks.jammy"
        },
        {
          "id": "io.buildpacks.samples.stacks.alpine"
        },
        {
          "id": "io.buildpacks.stacks.jammy"
        },
        {
          "id": "*"
        }
      ],
      "layerDiffID": "sha256:4ad60d1ea1836600073b50ae066e946068fdba9babbee257d3584cbe1ceb9cb7",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/hello-world",
      "name": "Hello World Buildpack"
    }
  },
  "samples/java-maven": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [
        {
          "id": "io.buildpacks.samples.stacks.jammy"
        },
        {
          "id": "io.buildpacks.samples.stacks.alpine"
        },
        {
          "id": "*"
        },
        {
          "id": "io.buildpacks.stacks.jammy",
          "mixins": [
            "build:set=shell-utils"
          ]
        }
      ],
      "layerDiffID": "sha256:a74e11d973b550235a0f9f1041bdeb0d810d20f5f646a53465d812a925c8b671",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/java-maven",
      "name": "Sample Java Maven Buildpack"
    }
  },
  "samples/kotlin-gradle": {
    "0.0.1": {
      "api": "0.2",
      "stacks": [
        {
          "id": "io.buildpacks.samples.stacks.jammy"
        },
        {
          "id": "io.buildpacks.samples.stacks.alpine"
        },
        {
          "id": "*"
        },
        {
          "id": "io.buildpacks.stacks.jammy",
          "mixins": [
            "build:set=shell-utils"
          ]
        }
      ],
      "layerDiffID": "sha256:5511b6ad86c688fc70ef7056a21f7dcf82eff299c9e03d05cfb8dc47c7e37b90",
      "homepage": "https://github.com/buildpacks/samples/tree/main/buildpacks/kotlin-gradle",
      "name": "Sample Kotlin Gradle Buildpack"
    }
  }
}

Note: pack buildpack package has a similar behavior

Before

When the pack builder create is executed without the flag, each buildpack is added into a different layer

Documentation

• Should this change be documented?
  • Yes, see #___
  • No

Related

Resolves #1595

[jjbustamante]

I still need to work in the unit test coverage, and probably I will like to refactor a little bit more, but I think in general the PR is ready for review.

Another thing I need to take a look is adding the depth flag Natalie mentioned.

[natalieparellano]

I want to bikeshed a little bit about the --flatten-meta-buildpacks flag name. I think technically according to our terminology it should be --flatten-composite-buildpacks. If we do --depth we might only need --flatten.

[jkutner]

I agree with @natalieparellano on --flatten-composite-buildpacks. But I didn't understand the --depth comment.

Also, is "flatten" common terminology for this? It feels more like "merge" or "combine". but if flatten is used in other places that's fine.

[jjbustamante]

I agree with @natalieparellano on --flatten-composite-buildpacks. But I didn't understand the --depth comment.

Also, is "flatten" common terminology for this? It feels more like "merge" or "combine". but if flatten is used in other places that's fine.

Natalie's depth comments is here

"flatten" is used by crane for example:

$ crane flatten --help
Flatten an image's layers into a single layer

Usage:
  crane flatten [flags]

Flags:
  -h, --help         help for flatten
  -t, --tag string   New tag to apply to flattened image. If not provided, push by digest to the original image repository.

Global Flags:
      --allow-nondistributable-artifacts   Allow pushing non-distributable (foreign) layers
      --insecure                           Allow image references to be fetched without TLS
      --platform platform                  Specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). (default all)
  -v, --verbose                            Enable debug logs

[jkutner]

@jjbustamante do you mind if we hold this until after 0.30.0-pre2?

[jjbustamante]

@jjbustamante do you mind if we hold this until after 0.30.0-pre2?

No problem, I am going to draft the PR again, because I am checking Nataliee's feedback

[natalieparellano]

@jjbustamante amazing work - the code seems well tested and appears to do what it sets out to do :) I added a few suggestions to hopefully improve the readability a bit, but the only that are truly blocking are:

• Add --flatten, --depth and flatten-exclude flags to pack builder create and pack buildpack package command #1691 (comment)
• Add --flatten, --depth and flatten-exclude flags to pack builder create and pack buildpack package command #1691 (comment)

[jjbustamante]

@jjbustamante amazing work - the code seems well tested and appears to do what it sets out to do :) I added a few suggestions to hopefully improve the readability a bit, but the only that are truly blocking are:

• Add --flatten, --depth and flatten-exclude flags to pack builder create and pack buildpack package command #1691 (comment)
• Add --flatten, --depth and flatten-exclude flags to pack builder create and pack buildpack package command #1691 (comment)

Thanks for this amazing feedback @natalieparellano ❤️ .. I just added some of them, I will take a look on the Windows support and the remaining suggestions! ..

[jkutner]

I'm reviewing this today

[jjbustamante]

I'm reviewing this today

Thanks @jkutner, I just added some TODO to add more test coverage as @natalieparellano suggested, but the important things for me to fix are the Windows support and the improvement in the flatten-exclude flag to avoid compressing a composite buildpack and its dependencies without having to explicitly add the dependencies with the flag.

As we talk in the leadership meeting, I like the strategy for merging this PR to include it in the next pre-release, and I will create a new PR with the improvements mentioned above.

[jkutner]

are you using buildpack api 0.3 on purpose in these?

[jjbustamante]

Not really. I think I copied and paste from another similar file