Download links without a version number

[dokmic]

First of all, thanks a lot for all the hard work on the v3 release 👏

While migrating to the latest version, I noticed that it is no longer possible to have a static link to the latest tarballs. Is it possible to remove the version number from the generated files names?

For example, previously, it was possible to get the latest S6 overlay by doing this:

FROM alpine:latest

ARG TARGETARCH

RUN wget -q -O - https://github.com/just-containers/s6-overlay/releases/latest/download/s6-overlay-$TARGETARCH.tar.gz | tar zxpf - -C /

ENTRYPOINT ["/init"]

After the v3 release, it is no longer possible, and you should always explicitly specify the version:

FROM alpine:latest

ARG TARGETARCH
ARG S6_VERSION=3.0.0.2

RUN wget -q -O - https://github.com/just-containers/s6-overlay/releases/download/v$S6_VERSION/s6-overlay-noarch-$S6_VERSION.tar.xz | tar Jxpf - -C / \
  && wget -q -O - https://github.com/just-containers/s6-overlay/releases/download/v$S6_VERSION/s6-overlay-$TARGETARCH-$S6_VERSION.tar.xz | tar Jxpf - -C /

ENTRYPOINT ["/init"]

Looking at these two Dockerfile's, do you perhaps consider providing a combined tarball containing noarch and arch contents altogether? So the overlay could be installed by downloading just one link. That should save some time on some CI's and simplify the user's Dockerfile.

[skarnet]

For the combined tarball: maybe, if there's significant demand for it? This complicates the build system and makes it a bit more confusing for users to know exactly what tarball(s) they need, and isn't ideal for integration with distributions, which is why I went with separate tarballs, but if it's more comfortable for more users, we could consider it in the future.

For a static link to the latest tarballs: that's a good idea indeed, having a -latest version of the tarballs always pointing to the most recent ones. @jprjr, is it possible for the CI to do that?

[GreyXor]

This is not a good practice, there is no guarantee that the latest version works the way you want it to. But here is a small workaround

latest_version=$(curl -sL https://api.github.com/repos/just-containers/s6-overlay/releases/latest | jq -r ".tag_name")

wget https://github.com/just-containers/s6-overlay/releases/download/$latest_version/s6-overlay-noarch-"${latest_version:1}".tar.xz

[jprjr]

Regarding having the latest version always at the same link - I think the easiest way would be for the make-based build system to not include the version number in the output tarball, or I can update the CI to rename the generated tarballs before uploading.

There's not a way to make github provide a -latest URL that redirects to the current version. Releases only supports uploading binaries, there's not a way to make a symlink-type release artifact. Alternatively, I could make the CI upload a second copy of the tarball to simulate having a redirect, but then our set of tarballs grows significantly, and would likely be confusing for users looking at the list, trying to figure out what to download.

I would be concerned about the non-reproducibility factor of using a dynamic link like that. My advice is to track the version either in the Dockerfile directly, or in some other file that specifies the version used as a build argument. But that's just advice - it's not my place to force anybody to follow my perceived best practices.

@skarnet - what would you prefer - for the build system to produce version-less tarballs, or for the CI system to rename them prior to upload?

[skarnet]

@jprjr It is totally your place to provide our software according to what you think are good practices. Users already follow a ton of bad practices shoved down their throats by moronic upstreams, some good ones would be a nice change. 😛
(I often rant about how, as a programmer, I need to provide mechanism and not policy; but as distributors, distribution policy is our bailiwick.)

I really don't like the idea of only outputting version-less tarballs. I think it's a disservice to users, and makes debugging pretty difficult. If GitHub can't make symlinks, then having -latest as a copy of the latest version doesn't sound too bad - it's only one more set of tarballs, it's not going to blow up their disks. The additional set will be overwritten with every new release, won't it?

[jprjr]

So, my thought was we upload the second set of tarballs tagged -latest (or just without anything, version-less), but attached to the versioned release. They wouldn't be overwritten - release v3.x.x.0 would have two sets, release v3.x.x.1 would have two sets, and so on. So my concern is if a user looks at the web page for a particular release - it's just a lot of files for them to parse through.

Another idea is we have a long-running, continuously-updated release named v3. Only issue I can think of there is, a release has to point to a tag, so then we're essentially deleting and re-creating git tags all the time.

The last idea is, we just say no - keep the versioned tarballs as-is, @GreyXor has a decent work-around to determine the latest release already.

I don't think it's a good practice to blindly use the latest version of anything. Granted, a downstream user may have enough rigorous, automated testing to catch an upgrade-related issue before it goes to production. But I also think that kind of user is going to use a specific, tested version anyway.

[dokmic]

@jprjr @skarnet We already have a version number in the URL https://github.com/just-containers/s6-overlay/releases/download/v<version>/s6-overlay-noarch.tar.xz. The users who need to pin the S6 overlay version can stick to this URL. And at the same time, another group of users who build the image occasionally can use the magic GitHub URL pointing to the latest release. I think trimming the version number from the tarball names should fulfill everybody's needs.

The workaround provided by @GreyXor will work, but it is imperative and requires some scripting. You cannot go with that using the ADD instruction. That also requires jq and calls GitHub API, which has rates limits.

There is another downside of having a version in the tarball names. Let's say you are installing that from a shell script. In that case, it will make the logic a bit more complicated since you need to know the exact version of the downloaded tarball. Stripping the version number will enable users to fully decouple downloading and extracting logic.

Creating a magic release like v3 is not a silver bullet either because that will not work for everybody. What if somebody wants to pin until a minor version (e.g., v3.1)?

It seems like having a version number in the tarball name requires some hacks and workarounds from the user's side. And at the same time, stripping them does not deprive the option to pin the version and works in all use cases.

What do you think?

[skarnet]

@dokmic I disagree with your reasoning: users who download tarballs should always know exactly what they are installing, so they should know the version number.
That said, if the GitHub URLs already provide the version number, it is indeed unnecessary to repeat it in the tarball names, so I'm okay with trimming it. I'll strip the version number from the generated tarballs directly in the build system.

[skarnet]

I removed the version numbers from the generated tarball names. The CI should keep working as is, but just in case, @jprjr can you verify?

The new version (3.1.0.0) isn't ready yet, I'm working on it; this is one of the changes that will go into it.

[crazy-max]

For a Dockerfile perspective it would be more efficient to rely on a "distribution" s6-overlay image that would be a multi-platform one and would have all the required tooling in a scratch stage like https://github.com/crazy-max/docker-alpine-s6#usage. This way users could simply do:

ARG S6_OVERLAY_VERSION=3.1.0.0-1
FROM just-containers/s6-overlay:$S6_OVERLAY_VERSION AS s6-overlay

FROM ubuntu
COPY --from=s6-overlay / /
RUN apt-get update && apt-get install -y nginx
RUN echo "daemon off;" >> /etc/nginx/nginx.conf
CMD ["/usr/sbin/nginx"]
ENTRYPOINT ["/init"]

instead of

# Use your favorite image
FROM ubuntu
ARG S6_OVERLAY_VERSION=3.1.0.0-1

RUN apt-get update && apt-get install -y nginx xz-utils
RUN echo "daemon off;" >> /etc/nginx/nginx.conf
CMD ["/usr/sbin/nginx"]

ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz
ENTRYPOINT ["/init"]

It removes many layers and doesn't require to install extra deps like xz-utils. It also covers and natively support multi-platform images which the current example does not.

[skarnet]

That ties you to the Docker container manager, though.
We talk about Docker because it's the most widely known and used, but there are other container managers, and we intend s6-overlay to work with all of those.

We could pursue better integration with Docker in the future, though.

[crazy-max]

That ties you to the Docker container manager, though.
We talk about Docker because it's the most widely known and used, but there are other container managers, and we intend s6-overlay to work with all of those.

Oh sure I didn't mean to imply that we should only rely on an image but as a distribution alternative.

We could pursue better integration with Docker in the future, though.

Happy to help if you want.

[felipecrs]

I also think that @crazy-max's suggestion is really nice and greatly simplifies installation in Dockerfiles.