RFC: inline Dockerfiles for simple images

[mlin]

This is a feature I've wanted for a long time to reduce dev/maintenance overhead; looking for comments on my rough implementation! CWL has something similar, DockerRequirement::dockerFile

Motivation: sometimes / fairly often, a WDL task just depends on a combination of tools which can each be installed through apt/conda/etc. In these simple cases, it's a chore to build a custom docker image and keep it public & up-to-date. Instead we see authors piling stuff into monolithic images to be shared across many tasks (to amortize the chore), or else installing packages at runtime (which makes things slower & more prone to random failures).

Proposed solution: As an optional alternative to the runtime.docker field, admit a runtime.inlineDockerfile field with an Array[String] supplying the text lines of a "simple" Dockerfile associated with the task. Then the engine assumes responsibility to execute the task command using an image built from that Dockerfile.

Example:

task decompress_any {
  input {
    File compressed
  }

  command <<<
    case $(basename "~{compressed}") in
      *.gz )
        pigz -dc "~{compressed}" > decompressed.dat
        ;;
      *.lz4 )
        lz4 -d "~{compressed}" decompressed.dat
        ;;
      *.zst )
        zstd -d "~{compressed}" decompressed.dat
        ;;
    esac
  >>>

  runtime {
    inlineDockerfile: <<<
      FROM ubuntu:20.04
      RUN apt-get -qq update && apt-get install -y pigz lz4 zstd
    >>>
  }

  output {
    File decompressed = "decompressed.dat"
  }
}

Details:

1. Definition of "simple" Dockerfile: build is reasonably expected to succeed every time, everywhere (given Internet access), without use of build arguments nor files from "build context"
2. In other cases, developers should continue using runtime.docker and maintain/publish their docker images in the usual ways
3. Interpolations in the inlineDockerfile are discouraged (but allowed) because they would tend to interfere with reuse of the image across multiple invocations of the task
4. For local runs, the engine can simply concatenate the lines and feed them to docker build, relying on its built-in caching features
5. In a distributed setting, the engine has to ensure the image is available to whichever host will execute the task, either by building it there just-in-time, or by pushing it to a shared registry beforehand

[geoffjentry]

I like this

[jdidion]

This would require a syntax change, to allow multi-line strings with the <<< >>> tags. That is not necessarily trivial and should probably be approved first.

An alternative for inline dockerfile would be a Array[String], which is already supported without syntax changes.

inlineDockerfile: [
  "FROM ubuntu:focal",
  "RUN apt-get -qq update && apt-get install -y pigz lz4 zstd"
]

Yet another alternative would be an object value with specific keys to make it more clear what's allowed:

inlineDockerfile: {
  from: "ubuntu:focal",
  run: ["apt-get -qq update", "apt-get install -y pigz lz4 zstd"]
}

[mlin]

Thanks @jdidion -- agree the Array[String] is a better way to start here, updated

[patmagee]

@mlin thanks alot for the suggestion! This actually seems like a really great addition TBH. There is one problem with this approach though. In 2.0 the docker key will be going away in favor of the containers keyword. This is an attempt to generify the runtime so that additional container technologies are supported.

Adding inline support for just docker goes a little against the grain of the runtime overhaul. Is there any way we can make this more generic?

[mlin]

I'm fine with generalizing the name, but runtime.container is practically just an alias for runtime.docker until it has a way to specify a desired non-docker container runtime -- perhaps with more structure like @jdidion suggests below. That might be a rabbit hole though depending on the use cases people have in mind. So I think the streams merge eventually, but should proceed in parallel whilst runtime.container remains somewhat half-baked (IMO)

[illusional]

When we say Docker, don't we actually mean any OCI compliant image? We configure different engines to use Docker, Singularity, uDocker, Shifter because they all accept OCI compliant images - which the most common format is just Docker - not to be confused with defaulting the registry to DockerHub by convention.

[mlin]

When we say Docker, don't we actually mean any OCI compliant image?

I don't know -- do we? 😅 #315 doesn't say any such thing right now, which reflects my feeling that runtime.container is currently half-baked as an alternative to runtime.docker. That's fine, we should finish the baking eventually...I just think it's largely separate from the current feature idea until there's more-than-aspirational support for non-docker images/repositories

[jdidion]

What about an object value instead:

{
  "format": "docker",
  "contents: [
    line1,
    line2,
    ...
  ]
}

contents could also be a multi-line string (as proposed in #414).

This way, we don't need a separate inlineDocker key - we can just re-use container and know that if it has an object value to interpret it as inline.

This is more flexible because there might be other information we want to put into the docker image, and other container formats might require additional information.

[kislyuk]

Just wanted to add a +1 on this. Inline Dockerfiles would make it much easier for us to manage some of our workflow distribution and versioning mechanisms.

[llangit-broad]

+1 as well

…

On Mon, Mar 1, 2021 at 1:35 PM Andrey Kislyuk ***@***.***> wrote: Just wanted to add a +1 on this. Inline Dockerfiles would make it much easier for us to manage some of our workflow distribution and versioning mechanisms. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#406 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4PJP7WKRMY6D3R6XPVNLLTBPUA3ANCNFSM4SWAH24Q> .

-- Lynn Langit 714.267.5966 (CST time zone)

[jdidion]

Coming back to this - since multi-line strings are about to be approved, inlineDockerfile can be represented as a string rather than array.

Also wanted to note the addition of a similar feature to NextFlow (along with an accompanying service to build and manage inline Dockerfiles): https://www.nextflow.io/blog/2022/rethinking-containers-for-cloud-native-pipelines.html

[patmagee]

I wonder what the requirement for engines will be. IE will all engines in all environments NEED to support this feature or is it an opt in feature of the language

[jdidion]

It should be possible to specify either/both a container and an inlineDockerfile. Also, once we have hints, we can consider whether to make inlineDockerfile a hint so that it is optional for the engine to support it.

Regarding @mlin earlier comment of fully baking container, I quite like all the options that CWL provides (https://www.commonwl.org/v1.2/CommandLineTool.html#DockerRequirement). We could consider allowing container to be a string, array, or object, and when it's an object allow additional fields to be defined. That way you could do the following and the engine can decide how it chooses which one(s) to support and in which order:

runtime {
  container: {
    pull: "foo/bar:1.0",
    load: "https://myrepo.com/foobar_image.tgz",
    dockerfile: <<<
    FROM ubuntu
    ...
    >>>
  }
}

[jdidion]

@mlin it seems like there's enough support for this to propose a PR if you're up for writing it up.

[zzzseeu]

I think inlineDockerfile is a very good feature and is very useful in development and testing. In fact, WDL 1.2 has been released, but I haven't seen any relevant syntax support. Will you consider adding this feature in subsequent versions? @jdidion