Image release pattern

[agjohnson]

We have a few things to decide on given our release pattern here and the fact that we now allow arbitrary image selection through readthedocs.yml.

What to do with latest now?

latest should probably be version 3.0 by now. I'm 👍 on making latest version 3.0 and making this the default (on a feature flag to scale in perhaps). I think we should maybe move to a versioned approach to latest though, as our docs now point people to use latest and our new latest, based on Ubuntu 18.04, could introduce unwanted changes.

This leads to another question.

How many images do users need access to?

It's helpful for users and for user to have some control over specific build image, however allowing users to select the minor version means we need to keep all these versions around, supported. I don't think it's necessary for users to have granular access to the minor versions.

This isn't a problem now, but if we semver our latest release, these versions will accumulate faster. We could end up with a list like: [(2,0), (3,0), (4,0), (4,1), (4,2)]. This seems like a lot to support, without much gain.

I think only supporting major versions in our config file makes sense, so build.image becomes an integer field.

This leads to another question

How do we handle image deprecation?

Arbitrary image selection leads to the problem that users might want to pin to version 2.0 now, or as it stands with our current config, 2.1. At some point we don't want to support this image though, so we need a pattern now on warning users that their image will be deprecated soon.

Things we do know

• We shouldn't point users to use latest tag, this puts them into a stream of build images that they need to get out of at some point. We should instruct 3.0 explicitly
• We don't want to go back to boolean image selection for the same reason
• If we want to support arbitrary images, we need UX around how to deprecate these images as well

[humitos]

Thoughts about the description of the problem and a proposal,

I don't think that supporting more that 3 images at the same time is a good idea (at the moment we have 2.0 (5.48GB) and latest (8.8GB) --using semver here will grow a lot and don't see anything good for that). Besides, what we have now (2.0.x has been changing the x without communicating this to the user and I think we didn't have to many problems on that --it has worked so far)

I would say to adopt a pattern like old-stable, stable and latest (only three images) --which at this point would be 2.0 (old-stable), 3.0 (stable, which would be our current latest) and latest (testing/devel/etc)

• People pinning on latest will always get the latest image: next year will be 4.0 and the next one will get 5.0, etc
• People pinning 2.0 now, will get 3.0 next year when 2.0 will not supported anymore
• People using 3.0 now, will get 4.0 when 3.0 is not supported anymore

Users will get always a proper version even if their yaml config is not accurate. That is, if the yaml says 2.0 but that version is deprecated / removed from the servers, they will get the current old-stable

If we follow this pattern, we don't have to much to do right now regarding the deprecation plan, since we will keep supporting all 2.0, 3.0 (latest) and we will add a new image in the near future when we have something new to test (Py3.7, to say something)

Regarding the UX, I think we need to send some kind of notification (email) to the user in a windows of 30 days each time a build is triggered.

I mentioned X.0 versions since we are using that naming currently, we I never seen a X.Y yet and I'm not sure why/how we would use that --so, maybe it's better to just removed the minor version also. I don't know.

[ericholscher]

I'm really confused by this, as it seems like there are a number of ways to handle this. I think the question that I'm struggling with are:

• What are our supported use cases for build images?

I think if we can define explicitly what cases we want to support, it will make the decision easier. My thinking is we want to support these things:

• Read the Docs maintainers pushing a new docker image, and having some users opt into it (eg. the latest) image
• Project maintainers pinning a version and having it not break for a long time

It feels like there is a tension here, because at some point if we remove old versions, it will break people. So I'm feeling kind of stuck and not knowing how to proceed. It seems like at some point we just need to stop supporting old images, but if we do that, then what is the value of even letting users define the image in the first place?

[humitos]

I think one of the idea of giving the user the ability to define the image is useful to start testing/use new features in production without breaking everything. As we have done with the form that people filled for the latest image and now with the YAML file.

"pinning a version and not break for a long time" is something that we can't guarantee 100% because we will need a lot of Gb in docker images --that's why I was proposing to use only three images and try to not break for a long time.

[ericholscher]

It feels like "pinning the latest version" is "keep the default", which wouldn't even need a setting. It seems we really want a way to "opt into dev/beta" and "opt out of broken latest until I fix it on my side".

[humitos]

It feels like "pinning the latest version" is "keep the default", which wouldn't even need a setting. It seems we really want a way to "opt into dev/beta" and "opt out of broken latest until I fix it on my side".

From this comment I understand that a boolean, like: use_beta_image solves that use case but...

With my proposal of old-stable, stable and latest (or testing) you get the same plus a "deprecation window time". So, if you are using stable and a new release is released, your stable will become old-stable and you will have some time to upgrade/change your code to make it compatible before the image you were using (now old-stable) is deprecated and deleted.

Besides, if you want to test new beta things, you can use latest (testing) in your project.

Then, we will have relative versions that will point to a different version depending on time, like:

• old-stable
• stable
• testing

and, specific numbers that will point to a specific version if it's already supported or fallback to old-stable if that specific version was deprecated/removed.

• 2, 3, etc

[ericholscher]

So stable is the default if it isn’t defined? I’m +1 on that.

[humitos]

Yes, stable is the default image to use if the user doesn't specify anything in the YAML file.

[agjohnson]

I think I've waffled on this no less than 5 times writing this reply, so I'm going to dump my thoughts before I change my mind again.

If we could support N images, versioning is 100% the way to go. For us, and for users. But without this, and with a faster release cycle, anything that relates to a version is going to present the same problems we're noting.

What is our purpose for the image

This should also be viewed as what is the users' purpose for these images I suppose. In the end, I think users really only have two use cases:

• User wants all of our features
• We fucked something up, user wants some time to adjust

This fits our ideas, except currently -- and with the plan above so far -- we're still pointing users to an unsupported image for new features. I think that is what needs to change -- we should treat the image like application code. If we support a feature, it should be a supported image. If not, we aren't pointing users to it.

So, this effectively gets rid of the latest image as we're using it. We're left with:

• latest -- our default. It's the current major release. Major releases track master closely except in the case of major version release candidate scenario. More below.
• stable -- we fucked something up. The previous major version.

There is also a change in procedure and how we do testing and release:

• Release candidates images get a feature flag and we test on a sample, there is no user opt in (or at least there is no opting in to testing image, only opting into an undocumented image 4.0 or something)
• When we've tested a release candidate sufficiently with a feature flag, we have a new major release. Users start getting this version as latest
• We never refer documentation to an unreleased version, like we are with latest currently. Users aren't told to subscribe to a future version

For example, the next few months look like this:

• Choices [1.0, 2.0] become invalid image choices, we add warning notifications
• A release of 3.0rc1 is put out, from master branch
• We feature flag test and scale up a sample size using the new image
• When we're happy, after a week or two, 3.0 is released. 3.0 image is still latest tag. Default image becomes latest
• 2.0 becomes stable. stable is documented now
• At the end of April, we prepare an 18.04 release candidate, 4.0rc1. We should be testing this locally now as core
• Eventually we feature flag on this version as the default version and scale up a sample size over the course of two weeks
• As soon as we are happy, 4.0 is released. It becomes the default
• stable points to 3.0
• [1.0, 2.0] are now deprecated choices

Key takeaways

• We treat latest like we do application code. When a feature is ready, it's in latest, we don't refer users to future images.
• Users opt in to more stable images, not less stable images
• We only have two images -- or maybe we decide on 3 (latest, stable, old-stable)?
• We shorten release and testing cycle

[ericholscher]

I think that makes sense. "Let users opt into dev features" seems like a good idea, but we never actually execute it well, so I think it's an anti-feature.

Agreed on "letting users opt into the old way when we break something" is the primary reason for this functionality. The other is "let users use custom build images", but I don't want to go down that road yet.

[humitos]

I like this pattern. It's very similar to what I was proposing with small differences (no testing image for example) and with a good and explicit pattern on how we will release a new image.

I think we can write a page documenting this process in our docs as a next step, and the following one is apply it with 3.0 to see how it goes in real life.

---

There is one thing that it's not clear to me: what the user will write in the image attribute in his yaml file: version number (2.0, 3.0) or tag name (latest, stable) or both?

Another thing is, if we are going to introduce new features in latest (the default image) are we freezing stable or are we going to backport some bug fixes? If we will modify stable for any reason, I think it makes sense to have an old-stable since we can break something else or make an incompatibility change. Otherwise, if it's freezed we don't need it, I think.

The other is "let users use custom build images", but I don't want to go down that road yet.

I think this won't be a good idea in the future either, since one of the things we are trying to avoid is the having many images on disk. So, using a custom one (from docker hub, for example) will increase the size on disk a lot.

[agjohnson]

what the user will write in the image attribute in his yaml file: version number (2.0, 3.0) or tag name (latest, stable) or both?

I'd say both are allowed, but stable and latest are what we document as valid. This way we can at least point projects to use 4.0 or something as an image, but normal usage should always be stable and latest.

are we freezing stable or are we going to backport some bug fixes

Good question. I don't think we freeze anything, we'll only apply bugfixes to stable. I think we keep things the same otherwise -- we stick with semver and major release versions dictate whether the image is stable or latest. I don't worry much about bugfix releases, just new features.

I think we can write a page documenting this process in our docs as a next step, and the following one is apply it with 3.0 to see how it goes in real life.

+1

I'd like to get 3.0 out and starting to test in a wider audience. For next week, we can release an image and notify folks to start using it -- this release process also benefits my direct case, in that i can tell users to pin to image latest now :)

[agjohnson]

Okay, i updated our contributing docs and started building the new images. The rest will be changes in ops/prod, and on our yaml config.