From 3f72dd0ba6f7c4e7b935e9c59f2575b07cfe7ee6 Mon Sep 17 00:00:00 2001 From: Emily Casey Date: Mon, 12 Oct 2020 17:51:35 -0400 Subject: [PATCH 1/5] Opt-in layer caching Signed-off-by: Emily Casey --- buildpack.md | 83 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 50 insertions(+), 33 deletions(-) diff --git a/buildpack.md b/buildpack.md index 55cd9e5e..8bdb2684 100644 --- a/buildpack.md +++ b/buildpack.md @@ -162,9 +162,9 @@ Executable: `/bin/build `, Working Dir: ` Using the [Layer Content Metadata](#layer-content-metadata-toml) provided by a buildpack in a `/.toml` file, the lifecycle MUST determine: -- Whether the layer directory in `//` should be available to the app (via the `launch` boolean). -- Whether the layer directory in `//` should be available to subsequent buildpacks (via the `build` boolean). -- Whether and how the layer directory in `//` should be persisted to subsequent builds of the same OCI image (via the `cache` boolean). +- Whether the layer directory `//` should be available to the app (via the `launch` boolean). +- Whether the layer directory `//` should be available to subsequent buildpacks (via the `build` boolean). +- Whether the layer directory `//` should be persisted to subsequent builds of the same OCI image (via the `cache` boolean). All combinations of `launch`, `build`, and `cache` booleans are valid. When a layer declares more than one type (e.g. `launch = true` and `cache = true`), the requirements of each type apply. @@ -178,8 +178,8 @@ The lifecycle MUST include each launch layer in the built OCI image. The lifecycle MUST also store the Layer Content Metadata associated with each layer so that it can be recovered using the layer Diff ID. Before a given re-build: -- If a launch layer is marked `cache = false` and `build = false`, the lifecycle: - - MUST restore the entire `/.toml` file from the previous build to the same path and +- If a launch layer is marked `cache = false` and `build = false` in the previous image metadata, the lifecycle: + - MUST restore Layer Content Metadata to `/.toml`, excluding the `launch`, `build`, and `cache` booleans. - MUST NOT restore the corresponding `//` directory from any previous build. After a given re-build: @@ -209,21 +209,19 @@ If a cache is provided the lifecycle: - SHOULD store the Layer Content Metadata associated with each layer so that it can be recovered using the layer Diff ID Before the next re-build: -- The lifecycle: - - MUST either restore the entire `/.toml` file and corresponding `//` directory from any previous build to the same paths or - - MUST restore neither the `/.toml` file nor corresponding `//` directory. +- The lifecycle MUST do both or neither of the following: + - Restore Layer Content Metadata to `/.toml`, excluding the `launch`, `build`, and `cache` booleans. + - Restore layer contents to the `//` directory. + +#### Ignored Layers -#### Other Layers +Layers marked `launch = false`, `build = false`, and `cache = false` behave like temporary directories, available only to the authoring buildpack, existing for the duration of a single build. -The lifecycle: -- If `build = false` - - MUST NOT make the layer available to subesequent buildpacks. -- If `launch = false` - - MUST NOT include the layer or the Layer Content Metadata in the OCI image. -- If `cache = false` - - MUST NOT store the layer or the Layer Content Metadata in the cache. +At the end of each individual buildpack's build phase: +- The lifecycle: + - MUST rename `//` to `/.ignore/` for all layers where `launch = false`, `build = false`, and `cache = false`. -Layers marked `launch = false`, `build = false`, and `cache = false` behave like temporary directories, available only to the authoring buildpack, that exist for the duration of a single build. +The lifecycle restores all layers from previous builds as ignored layers. Buildpacks MUST set `launch`, `build`, and/or `cache` in `/.toml` to use a restored layer. ## App Interface @@ -376,13 +374,9 @@ The lifecycle MUST skip analysis and proceed to the build phase if no such image - A reference to the previously created OCI image described above and - The final ordered group of buildpacks determined during the detection phase, -For each buildpack in the group, - -1. All `/.toml` files with `cache = true` and corresponding `/` directories from any previous build are restored to their same filesystem locations. -2. Each `/.toml` file with `launch = true` and `cache = false` that was present at the end of the build of the previously created OCI image is retrieved. -3. A given `/.toml` file with `launch = true` and `cache = true` and corresponding `/` directory are both removed if either do not match their contents in the previously created OCI image. - -Finally, the contents of `/store.toml` from the build of the previously created OCI image are restored. +For each buildpack in the group, the lifecycle +1. MUST restore `/.toml` files from the previous build as described in [Layer Types](#layer-types). +2. MUST restore `/store.toml`. After analysis, the lifecycle MUST proceed to the build phase. @@ -397,13 +391,15 @@ The purpose of build is to transform application source code into runnable artif During the build phase, typical buildpacks might: 1. Read the Buildpack Plan in `` to determine what dependencies to provide. -2. Provide the application with dependencies for launch in `/`. -3. Provide subsequent buildpacks with dependencies in `/`. -4. Compile the application source code into object code. -5. Remove application source code that is not necessary for launch. -6. Provide start command in `/launch.toml`. -7. Write a partial Bill-of-Material to `/launch.toml` describing any provided application dependencies. -8. Write a partial Bill-of-Material to `/build.toml` describing any provided build dependencies. +1. Provide the application with dependencies for launch in `/`. +1. Reuse application dependencies from a previous image by appending by `launch = true` to `/.toml`. +1. Provide subsequent buildpacks with dependencies in `/`. +1. Reuse cached build dependencies from a previous build by appending by `build = true` and `cache = true` to `/.toml`. +1. Compile the application source code into object code. +1. Remove application source code that is not necessary for launch. +1. Provide start command in `/launch.toml`. +1. Write a partial Bill-of-Material to `/launch.toml` describing any provided application dependencies. +1. Write a partial Bill-of-Material to `/build.toml` describing any provided build dependencies. The purpose of separate `/` directories is to: @@ -486,15 +482,36 @@ If generated, this build BOM MUST contain all `bom` entries in each `build.toml` #### Layers -A buildpack MAY create, modify, or delete `//` directories and `/.toml` files as specified in the [Layer Types](#layer-types) section. +##### Providing Layers +A buildpack MAY create a new layer by creating a `//` directories and `/.toml` file as specified in the [Layer Types](#layer-types) section. + +##### Reusing Layers -To decide what layer operations are appropriate, the buildpack should consider: +The lifecycle provides a mechanism for buildpacks to explicitly opt into reusing layers from a previous build. The buildpack may modify cached layers before reusing them. + +To decide whether layer reuse is appropriate, the buildpack should consider: - Whether files in the `` directory have changed since the layer was created. - Whether the environment has changed since the layer was created. - Whether the buildpack version has changed since the layer was created. - Whether new application dependency versions have been made available since the layer was created. +At the start of the build phase a buildpack MAY find: +- Partial `/.toml` files describing layers from the previous builds. The restored Layer Content Metadata SHALL NOT contain `launch`, `build`, or `cache` booleans. +- `//` directories containing layer contents that have been restored from the cache. + +The buildpack: + - MAY set `launch = true` in the restored `/.toml` file in order to include the layer in the final image. + - MAY modify `metadata` in `/.toml` + - **If** layer contents have been restored to the `//` directory + - MAY set `build = true` in the restored `/.toml` to expose to layer to subsequent buildpacks. + - MAY set `cache = true` in the restored `/.toml` to persist the layer to subsequent builds. + - MAY modify the contents of `//`. + +If the buildpack does not set `launch`, `build`, or `cache` in the restored `/.toml` the layer SHALL be ignored. + +##### Slice Layers + Additionally, a buildpack MAY specify sub-paths within `` as `slices` in `launch.toml`. Separate layers MUST be created during the export phase for each slice with one or more files or directories. This minimizes data transfer when the app directory contains a known set of files. From f95db0ee6b79136c393660f96e86550e98c6897b Mon Sep 17 00:00:00 2001 From: Emily Casey Date: Mon, 12 Oct 2020 17:57:52 -0400 Subject: [PATCH 2/5] Fix TOC Signed-off-by: Emily Casey --- buildpack.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/buildpack.md b/buildpack.md index 8bdb2684..b35c295e 100644 --- a/buildpack.md +++ b/buildpack.md @@ -27,7 +27,8 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th - [Layer Types](#layer-types) - [Launch Layers](#launch-layers) - [Build Layers](#build-layers) - - [Other Layers](#other-layers) + - [Cached Layers](#cached-layers) + - [Ignored Layers](#ignored-layers) - [App Interface](#app-interface) - [Phase #1: Detection](#phase-1-detection) - [Purpose](#purpose) @@ -42,6 +43,9 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th - [Unmet Buildpack Plan Entries](#unmet-buildpack-plan-entries) - [Bills-of-Materials](#bills-of-materials) - [Layers](#layers) + - [Providing Layers](#providing-layers) + - [Reusing Layers](#reusing-layers) + - [Slice Layers](#slice-layers) - [Phase #4: Export](#phase-4-export) - [Purpose](#purpose-3) - [Process](#process-3) From 05a09fdeb8f9b5ec7984148eeac73f2181e4ea23 Mon Sep 17 00:00:00 2001 From: Emily Casey Date: Tue, 13 Oct 2020 10:43:26 -0400 Subject: [PATCH 3/5] Apply suggestions from code review Signed-off-by: Emily Casey Co-authored-by: Joe Kutner --- buildpack.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/buildpack.md b/buildpack.md index b35c295e..08d38102 100644 --- a/buildpack.md +++ b/buildpack.md @@ -168,7 +168,7 @@ Using the [Layer Content Metadata](#layer-content-metadata-toml) provided by a b - Whether the layer directory `//` should be available to the app (via the `launch` boolean). - Whether the layer directory `//` should be available to subsequent buildpacks (via the `build` boolean). -- Whether the layer directory `//` should be persisted to subsequent builds of the same OCI image (via the `cache` boolean). +- Whether the layer directory `//` should be persisted and made available to subsequent builds of the same OCI image (via the `cache` boolean). All combinations of `launch`, `build`, and `cache` booleans are valid. When a layer declares more than one type (e.g. `launch = true` and `cache = true`), the requirements of each type apply. @@ -398,7 +398,7 @@ During the build phase, typical buildpacks might: 1. Provide the application with dependencies for launch in `/`. 1. Reuse application dependencies from a previous image by appending by `launch = true` to `/.toml`. 1. Provide subsequent buildpacks with dependencies in `/`. -1. Reuse cached build dependencies from a previous build by appending by `build = true` and `cache = true` to `/.toml`. +1. Reuse cached build dependencies from a previous build by appending `build = true` and `cache = true` to `/.toml`. 1. Compile the application source code into object code. 1. Remove application source code that is not necessary for launch. 1. Provide start command in `/launch.toml`. From 6fcc0929b0cc4f61c5d8d90364a041e375cde152 Mon Sep 17 00:00:00 2001 From: Emily Casey Date: Tue, 13 Oct 2020 10:45:32 -0400 Subject: [PATCH 4/5] Explicitly mention that bools are not restored even if they were set previously Signed-off-by: Emily Casey --- buildpack.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildpack.md b/buildpack.md index 08d38102..1ae3d6aa 100644 --- a/buildpack.md +++ b/buildpack.md @@ -501,7 +501,7 @@ To decide whether layer reuse is appropriate, the buildpack should consider: - Whether new application dependency versions have been made available since the layer was created. At the start of the build phase a buildpack MAY find: -- Partial `/.toml` files describing layers from the previous builds. The restored Layer Content Metadata SHALL NOT contain `launch`, `build`, or `cache` booleans. +- Partial `/.toml` files describing layers from the previous builds. The restored Layer Content Metadata SHALL NOT contain `launch`, `build`, or `cache` booleans even if those values were set on a previous build. - `//` directories containing layer contents that have been restored from the cache. The buildpack: From fd06da10431c5cffccad1dbfe3ec7e6cfefc0d07 Mon Sep 17 00:00:00 2001 From: Emily Casey Date: Wed, 14 Oct 2020 13:41:12 -0400 Subject: [PATCH 5/5] Clarify the intent behind renaming ignored layers Signed-off-by: Emily Casey --- buildpack.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildpack.md b/buildpack.md index 1ae3d6aa..313a1a30 100644 --- a/buildpack.md +++ b/buildpack.md @@ -223,7 +223,7 @@ Layers marked `launch = false`, `build = false`, and `cache = false` behave like At the end of each individual buildpack's build phase: - The lifecycle: - - MUST rename `//` to `/.ignore/` for all layers where `launch = false`, `build = false`, and `cache = false`. + - MUST rename `//` to `/.ignore/` for all layers where `launch = false`, `build = false`, and `cache = false`, in order to prevent subsequent buildpacks from accidentally depending on an ignored layer. The lifecycle restores all layers from previous builds as ignored layers. Buildpacks MUST set `launch`, `build`, and/or `cache` in `/.toml` to use a restored layer.