Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opt-in layer caching #151

Closed
wants to merge 5 commits into from
+55 −34
Closed

Opt-in layer caching #151

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3f72dd0
Opt-in layer caching
ekcasey Oct 12, 2020
f95db0e
Fix TOC
ekcasey Oct 12, 2020
05a09fd
Apply suggestions from code review
ekcasey Oct 13, 2020
6fcc092
Explicitly mention that bools are not restored even if they were set …
ekcasey Oct 13, 2020
fd06da1
Clarify the intent behind renaming ignored layers
ekcasey Oct 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 55 additions & 34 deletions buildpack.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th
- [Layer Types](#layer-types)
- [Launch Layers](#launch-layers)
- [Build Layers](#build-layers)
- [Other Layers](#other-layers)
- [Cached Layers](#cached-layers)
- [Ignored Layers](#ignored-layers)
- [App Interface](#app-interface)
- [Phase #1: Detection](#phase-1-detection)
- [Purpose](#purpose)
Expand All @@ -42,6 +43,9 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th
- [Unmet Buildpack Plan Entries](#unmet-buildpack-plan-entries)
- [Bills-of-Materials](#bills-of-materials)
- [Layers](#layers)
- [Providing Layers](#providing-layers)
- [Reusing Layers](#reusing-layers)
- [Slice Layers](#slice-layers)
- [Phase #4: Export](#phase-4-export)
- [Purpose](#purpose-3)
- [Process](#process-3)
Expand Down Expand Up @@ -162,9 +166,9 @@ Executable: `/bin/build <layers[EIC]> <platform[AR]> <plan[ER]>`, Working Dir: `

Using the [Layer Content Metadata](#layer-content-metadata-toml) provided by a buildpack in a `<layers>/<layer>.toml` file, the lifecycle MUST determine:

- Whether the layer directory in `<layers>/<layer>/` should be available to the app (via the `launch` boolean).
- Whether the layer directory in `<layers>/<layer>/` should be available to subsequent buildpacks (via the `build` boolean).
- Whether and how the layer directory in `<layers>/<layer>/` should be persisted to subsequent builds of the same OCI image (via the `cache` boolean).
- Whether the layer directory `<layers>/<layer>/` should be available to the app (via the `launch` boolean).
- Whether the layer directory `<layers>/<layer>/` should be available to subsequent buildpacks (via the `build` boolean).
- Whether the layer directory `<layers>/<layer>/` should be persisted and made available to subsequent builds of the same OCI image (via the `cache` boolean).

All combinations of `launch`, `build`, and `cache` booleans are valid. When a layer declares more than one type (e.g. `launch = true` and `cache = true`), the requirements of each type apply.

Expand All @@ -178,8 +182,8 @@ The lifecycle MUST include each launch layer in the built OCI image.
The lifecycle MUST also store the Layer Content Metadata associated with each layer so that it can be recovered using the layer Diff ID.

Before a given re-build:
- If a launch layer is marked `cache = false` and `build = false`, the lifecycle:
- MUST restore the entire `<layers>/<layer>.toml` file from the previous build to the same path and
- If a launch layer is marked `cache = false` and `build = false` in the previous image metadata, the lifecycle:
- MUST restore Layer Content Metadata to `<layers>/<layer>.toml`, excluding the `launch`, `build`, and `cache` booleans.
- MUST NOT restore the corresponding `<layers>/<layer>/` directory from any previous build.

After a given re-build:
Expand Down Expand Up @@ -209,21 +213,19 @@ If a cache is provided the lifecycle:
- SHOULD store the Layer Content Metadata associated with each layer so that it can be recovered using the layer Diff ID

Before the next re-build:
- The lifecycle:
- MUST either restore the entire `<layers>/<layer>.toml` file and corresponding `<layers>/<layer>/` directory from any previous build to the same paths or
- MUST restore neither the `<layers>/<layer>.toml` file nor corresponding `<layers>/<layer>/` directory.
- The lifecycle MUST do both or neither of the following:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth calling out that "both" is for cache = true and "neither" is for cache = false?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also there is that edge case where cache = false and launch = true. I don't know if that's too much detail...

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's important that the spec will be clear enough and explain all of the cases. I'll try to make it clearer.

- Restore Layer Content Metadata to `<layers>/<layer>.toml`, excluding the `launch`, `build`, and `cache` booleans.
- Restore layer contents to the `<layers>/<layer>/` directory.

#### Ignored Layers

#### Other Layers
Layers marked `launch = false`, `build = false`, and `cache = false` behave like temporary directories, available only to the authoring buildpack, existing for the duration of a single build.

The lifecycle:
- If `build = false`
- MUST NOT make the layer available to subesequent buildpacks.
- If `launch = false`
- MUST NOT include the layer or the Layer Content Metadata in the OCI image.
- If `cache = false`
- MUST NOT store the layer or the Layer Content Metadata in the cache.
At the end of each individual buildpack's build phase:
- The lifecycle:
- MUST rename `<layers>/<layer>/` to `<layers>/<layer>.ignore/` for all layers where `launch = false`, `build = false`, and `cache = false`, in order to prevent subsequent buildpacks from accidentally depending on an ignored layer.

Layers marked `launch = false`, `build = false`, and `cache = false` behave like temporary directories, available only to the authoring buildpack, that exist for the duration of a single build.
The lifecycle restores all layers from previous builds as ignored layers. Buildpacks MUST set `launch`, `build`, and/or `cache` in `<layers>/<layer>.toml` to use a restored layer.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be worth clarifying that "as ignored layers" means the types flags are set to false in the toml file, otherwise this could be interpreted to mean that the directory is also named with .ignore.


## App Interface

Expand Down Expand Up @@ -376,13 +378,9 @@ The lifecycle MUST skip analysis and proceed to the build phase if no such image
- A reference to the previously created OCI image described above and
- The final ordered group of buildpacks determined during the detection phase,

For each buildpack in the group,

1. All `<layers>/<layer>.toml` files with `cache = true` and corresponding `<layers>/<layer>` directories from any previous build are restored to their same filesystem locations.
2. Each `<layers>/<layer>.toml` file with `launch = true` and `cache = false` that was present at the end of the build of the previously created OCI image is retrieved.
3. A given `<layers>/<layer>.toml` file with `launch = true` and `cache = true` and corresponding `<layers>/<layer>` directory are both removed if either do not match their contents in the previously created OCI image.

Finally, the contents of `<layers>/store.toml` from the build of the previously created OCI image are restored.
For each buildpack in the group, the lifecycle
1. MUST restore `<layers>/<layer>.toml` files from the previous build as described in [Layer Types](#layer-types).
2. MUST restore `<layers>/store.toml`.

After analysis, the lifecycle MUST proceed to the build phase.

Expand All @@ -397,13 +395,15 @@ The purpose of build is to transform application source code into runnable artif
During the build phase, typical buildpacks might:

1. Read the Buildpack Plan in `<plan>` to determine what dependencies to provide.
2. Provide the application with dependencies for launch in `<layers>/<layer>`.
3. Provide subsequent buildpacks with dependencies in `<layers>/<layer>`.
4. Compile the application source code into object code.
5. Remove application source code that is not necessary for launch.
6. Provide start command in `<layers>/launch.toml`.
7. Write a partial Bill-of-Material to `<layers>/launch.toml` describing any provided application dependencies.
8. Write a partial Bill-of-Material to `<layers>/build.toml` describing any provided build dependencies.
1. Provide the application with dependencies for launch in `<layers>/<layer>`.
1. Reuse application dependencies from a previous image by appending by `launch = true` to `<layers>/<layer>.toml`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. Reuse application dependencies from a previous image by appending by `launch = true` to `<layers>/<layer>.toml`.
1. Reuse application dependencies from a previous image by appending `launch = true` to `<layers>/<layer>.toml`.

Copy link
Member

@natalieparellano natalieparellano Mar 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also requires that the buildpack does NOT create a <layers>/<layer> of its own... IDK if that is implied or would be worth spelling out. Personally I found myself re-reading this a couple of times...

1. Provide subsequent buildpacks with dependencies in `<layers>/<layer>`.
1. Reuse cached build dependencies from a previous build by appending `build = true` and `cache = true` to `<layers>/<layer>.toml`.
1. Compile the application source code into object code.
1. Remove application source code that is not necessary for launch.
1. Provide start command in `<layers>/launch.toml`.
1. Write a partial Bill-of-Material to `<layers>/launch.toml` describing any provided application dependencies.
1. Write a partial Bill-of-Material to `<layers>/build.toml` describing any provided build dependencies.

The purpose of separate `<layers>/<layer>` directories is to:

Expand Down Expand Up @@ -486,15 +486,36 @@ If generated, this build BOM MUST contain all `bom` entries in each `build.toml`

#### Layers

A buildpack MAY create, modify, or delete `<layers>/<layer>/` directories and `<layers>/<layer>.toml` files as specified in the [Layer Types](#layer-types) section.
##### Providing Layers
A buildpack MAY create a new layer by creating a `<layers>/<layer>/` directories and `<layers>/<layer>.toml` file as specified in the [Layer Types](#layer-types) section.

##### Reusing Layers

To decide what layer operations are appropriate, the buildpack should consider:
The lifecycle provides a mechanism for buildpacks to explicitly opt into reusing layers from a previous build. The buildpack may modify cached layers before reusing them.

To decide whether layer reuse is appropriate, the buildpack should consider:

- Whether files in the `<app>` directory have changed since the layer was created.
- Whether the environment has changed since the layer was created.
- Whether the buildpack version has changed since the layer was created.
- Whether new application dependency versions have been made available since the layer was created.

At the start of the build phase a buildpack MAY find:
- Partial `<layers>/<layer>.toml` files describing layers from the previous builds. The restored Layer Content Metadata SHALL NOT contain `launch`, `build`, or `cache` booleans even if those values were set on a previous build.
- `<layers>/<layer>/` directories containing layer contents that have been restored from the cache.

The buildpack:
- MAY set `launch = true` in the restored `<layers>/<layer>.toml` file in order to include the layer in the final image.
- MAY modify `metadata` in `<layers>/<layer>.toml`
- **If** layer contents have been restored to the `<layers>/<layer>/` directory
- MAY set `build = true` in the restored `<layers>/<layer>.toml` to expose to layer to subsequent buildpacks.
- MAY set `cache = true` in the restored `<layers>/<layer>.toml` to persist the layer to subsequent builds.
- MAY modify the contents of `<layers>/<layer>/`.

If the buildpack does not set `launch`, `build`, or `cache` in the restored `<layers>/<layer>.toml` the layer SHALL be ignored.

##### Slice Layers

Additionally, a buildpack MAY specify sub-paths within `<app>` as `slices` in `launch.toml`.
Separate layers MUST be created during the export phase for each slice with one or more files or directories.
This minimizes data transfer when the app directory contains a known set of files.
Expand Down