Time-based release process

[JukkaL]

For some time, we've had a pretty unpredictable release process, with long delays between releases. I see two main root causes for this. First, only Dropbox employees have been able to act as release managers -- mainly since we rely heavily on type checking our internal code base to avoid regressions and to validate new releases. This limits the available effort release managers have for releases (we also have our regular work responsibilities that often take precedence, unfortunately). Second, an infrequent release process tends to result in many regressions that take a lot of time to investigate and fix. This tends to make making a release a big undertaking, which also tends to reduce the frequency of releases, since fixing regressions is not a particularly fun activity.

I have some thoughts about how we might be able to make monthly releases, for example, hopefully fairly reliably. Here's a summary:

• Have at least 3-4 active maintainers that could work as release managers, including people not working at Dropbox. If each release manager owns around 1/4 of releases, for example, the workload doesn't need to be very high.
• Since the Dropbox internal codebase has been very helpful in finding regressions, I think that we should continue to use that. However, this could happen asynchronously from the mypy release process in the future, so that releases don't need to be blocked on this.
• To avoid regressions delaying a release, we can move to a policy where the default option when encountering a regression is for the release manager to revert the change, unless a fix is almost immediately available. If releases are more frequent, missing a single release does not have to be a big deal. We can merge the change again after the regression has been fixed (or after we've decided that the regressions is acceptable).
• Speeding up the release process probably increases the chances of regressions. To reduce the impact, we could also have a policy of having a bugfix release around a week after each feature release (unless no regressions have been reported). We could even have multiple bugfix releases per feature release, since bugfixes are often quite low risk.

Thoughts? Are there any volunteers for the release manager role?

[JukkaL]

A more drastic change would be to switch to very frequent (say weekly) releases, and rely on fixing/reverting major regressions after the release in a patch release. Here the assumption is that we rarely have major regressions that affect a large fraction of users, since our regular testing process (mypy primer, test suite) already works quite well.

For this to be practical, we'd need to streamline our releases a bit so that making a release only takes very little effort (e.g. auto-generate blog post draft from commit history). Having a release manager role would still be important, as somebody would have to push the button and make sure regressions get addressed in a timely manner, and make sure that blog posts go out (or maintain changelog in some other format). The role would look quite a bit different from the first proposal, though.

[JelleZijlstra]

Thanks for bringing this up! I think the slow release cadence has been a problem; it's slowed us down at typeshed and we regularly get comments from users asking when some merged PR will be released.

I like the idea of having weekly, lightweight releases. We could perhaps stop having a blog post for each release and rely on GitHub releases to autogenerate a changelog for us. And if releases are weekly, we may not even need separate bugfix releases.

We'll have to rethink the versioning scheme though, lest we end up with html5lib-style versions. Perhaps it's time to just jump to 1.0.

[jteppinette]

Perhaps there is room for improved automation whereby we can detect regressions at the pull request stage. An increased test footprint and policy to require additional testing with pull requests may help.

I am available to help in any way.

[hauntsaninja]

I'm happy to help with releases.

In terms of schedule, I'd be happy with: "monthly by default" + "whenever anyone feels like it"

Thoughts on versioning scheme:

• Empirically our releases frequently have issues that need patching, some severe, so I think bugfix releases are important. We should have a patch digit so users can set their requirements such that they automatically get bugfixes.
• While a little slow, the extra validation done and resulting fixes at Dropbox have been really valuable. It might be nice to have a "Dropbox has validated this version!" marker in the versioning scheme. Possibly a bump of major version?
• Maybe we could make more use of prereleases? Black used exclusively beta versions until just last month and most people were fine with it. (The people who weren't fine with it just wanted non-beta releases, which we would still do)

[97littleleaf11]

Vote for GitHub releases and prerelease scheme.

[JukkaL]

Thanks for the ideas! Here is a more concrete proposal that incorporates some of the above ideas.

Summary

These are the biggest changes from the existing release process:

1. Publish beta releases at least every 4 weeks, with a simple process, to make new features and bug fixes available with a predictable schedule
2. Validate against Dropbox internal code asynchronously so that stable releases don't need to block on this, and release managers don't need access to Dropbox internal code
3. Release manager shifts are time-based and cover 3 months (around 2-3 feature releases), instead of just one release
4. Release notes are posted on GitHub instead of the current blog, with less manual editing

Release manager rotation

We'd have at least three release managers (RMs) in a rotation. Each shift could be three months and would typically include 2-3 feature releases.

Beta releases

The acting release manager makes sure we have a beta release at least monthly. These can be used in typeshed, and they make it easier to start using fixes or new features not yet included in a stable release. We'd encourage users to use them so that we can find regressions sooner.

It's fine to publish beta releases more frequently. Each member of the release manager rotation (not just the currently acting RM) can publish a beta release at any point, assuming the CI build and wheel builds are green, and there are no known blocking issues (more about this below).

Stable releases

Our goal would be to have a stable feature release around every 4 weeks (and at least every 8 weeks). A stable release would be based on a beta release that has been out for at least a week, to allow time for finding regressions. When making a beta release that will be used as the basis for a stable release, the RM will announce this intention by creating a GitHub release planning issue.

The RM can cherry-pick low-risk fixes on top of the beta release during the stabilization period. If there are high-risk but important fixes, it's okay to make an additional beta release from the stable release branch.

Feature development can continue in the master branch, and it's okay to make new beta releases from the master branch while waiting for a stable feature release to go out.

Bugfix releases

After a stable feature release, we may publish one or more bugfix releases, mainly to address regressions (but any bug fixes are fine). These would often be released around 1-2 weeks after the stable release. The RM can decide when to publish additional bugfix releases and how many. Other members of the RM rotation can also make bugfix releases.

Release notes

We could probably switch to using GitHub for release notes, instead of using the current blog.

For beta releases and bugfix releases, we'd only provide auto-generated changelogs, to make it simple to make a release.

For feature releases we'd continue to write more detailed release notes. They'd include manually edited "release highlights" followed by a detailed, auto-generated changelog, to make the process light-weight.

Release numbering

We'd start with mypy 1.0. We'd follow a scheme that is influenced by semver (but we won't follow it strictly), where the major release number is increased when there is a major backward incompatible change, such as one of these:

• Removal of a significant feature (e.g. Python 2 support)
• Change of the default value of some configuration option
• Backward-incompatible change to the plugin API
• Any change that is likely to require code changes for a large fraction of users (in mypy or typeshed)

In most feature releases we'll increase the minor version number. Bugfix releases increment the third version number.

Beta versions would be numbered x.(y+1)bN, where x.y is the latest stable release or the stable release currently being released. So if 2.3 is the latest stable release, and 2.4 is being released on top of 2.3b1 (but not out yet), the next beta release would be 2.5b0 (if created from master).

Release blockers

If there's a major regression in master, we can declare an issue as a release blocker, and no beta or feature release can go out until the issue is resolved.

Dealing with regressions

The default response to a regression or release blocker would be to revert the change that introduced it. This would be a normal and expected part of our development process, and contributors shouldn't feel bad if this happens, as regressions are impossible to prevent.

The motivation is to unblock the release process as quickly as possible. Otherwise we'd risk accumulating multiple regressions that could take weeks or months to fix.

Testing with internal repos

We'd continue to test against Dropbox internal code asynchronously. If there's a significant regression, we'd either revert the offending commit, implement a forward fix (at least if it's not release blocking), or declare a release blocker. The feature releases don't need to wait for a thumbs up from Dropbox employees, but the RM is allowed to wait for this (at their discretion).

Other owners of private repos are also encouraged to test using beta releases or development releases, and they can also request reverts and declare release blockers. We'll need a detailed bug report for this.

RM responsibilities

These would be the main tasks of the acting RM:

1. Make sure there's a beta release out at least every 4 weeks (involves running a release script if nothing is seriously broken)
2. Publish a feature release every 4-8 weeks
3. Publish bugfix releases as needed
4. Make sure we sync typeshed at least once between feature releases
5. At least during and for some time after the feature release process, monitor GitHub issues for regressions
6. Help fix broken CI/builds and release blockers (possibly by reverting a change)

Discussion

I'm proposing 3-month RM shifts instead of, say, one release per shift, to reduce friction. There will be some overhead each time we change the RM, as the new RM will have to refresh their memory about the specifics of the role.

We could also consider 6-month shifts, but that could be too big of a commitment.

[KotlinIsland]

I'm very interested in volunteering as a release manager.

[ethanhs]

I hadn't had a chance to read over Jukka's proposal until now, but I really like like it!

I think the only part that concerns me is the versioning. I know we are approaching mypy 0.990 which will maybe force us to move to a different versioning scheme, but I am a bit leery to declare mypy 1.0, especially with big UX changes such as #10600, and the large number of crashes currently open.

I also do not think we should declare the plugin API stable, even if we make a mypy 1.0, as I expect changes due to potential work such as #7724. I think if we do make a mypy 1.0, one thing we could do is version the plugin API separately, and stabilize it later.

I also am not sure how effective having beta releases will be. In practice even CPython has issues of not enough testing of beta releases. It is of course easier to test beta mypy but I expect most people will just wait for the final release.

[JukkaL]

Update on the release process: I'm currently trying out a monthly release process (+ optional bugfix releases), at least initially staffed by Dropbox employees. If we can keep making all scheduled releases for, say, 4-5 feature releases, we can look at onboarding additional release managers, also from outside Dropbox. If mypy 0.960 successfully goes out in May, it would be the third successful monthly feature release.

@ethanhs

I think the only part that concerns me is the versioning. I know we are approaching mypy 0.990 which will maybe force us to move to a different versioning scheme, but I am a bit leery to declare mypy 1.0, especially with big UX changes such as #10600, and the large number of crashes currently open.

I wonder if having version numbers like 0.1000, 0.1010, etc. is supported. These would look a bit weird, but maybe it would be okay? If that doesn't work out, we can try to fix some highest-impact bugs and implement some planned disruptive changes before releasing 1.0, but with the assumption that some things would still be in flux. We can always release 2.0, 3.0, and so on if there are further major changes. We won't run out of major version numbers :-)

I think if we do make a mypy 1.0, one thing we could do is version the plugin API separately, and stabilize it later.

Agreed. It's too early to stabilize the plugin API.

I also am not sure how effective having beta releases will be. In practice even CPython has issues of not enough testing of beta releases. It is of course easier to test beta mypy but I expect most people will just wait for the final release.

I think that more frequent and predictable stable releases are more important than beta releases. We can consider supporting beta releases in the future, but I don't feel that it's the most important thing to focus on now.

@KotlinIsland

I'm very interested in volunteering as a release manager.

Thanks for the offer! As I mentioned earlier, we may start onboarding additional release managers later this year.

[pranavrajpal]

I think it would be useful to have automated (or mostly automated) beta releases that just release what's on master. As mentioned earlier, the current process of spending time putting together a release isn't really ideal for anyone (users get inconsistent delays between releases, maintainers spend a lot of time creating a release). I think we could reduce most of those problems by automating that process.

In this system, we'd have beta releases that periodically (probably somewhere between weekly and monthly) just take the current master, make sure CI and any other tests are passing, and then publish a new version. Stable releases could then be created by taking a beta release, testing that on Dropbox code or doing any other verification necessary, fixing any relevant bugs, and then releasing a stable version of that same release.

Some other related notes:

• The problem of making sure beta releases are actually used could be handled by recommending using beta releases as the default option. That would be helped by the fact that stable releases would probably be released much less often than beta releases, so people looking for new features would stick with beta releases.
• People who want the extra testing and are willing to wait longer can wait for stable releases, but there would be less people waiting for those stable releases, so this would help reduce the pressure on maintainers to create stable releases.
• A fast beta release schedule makes sure that PRs make it into releases fairly soon after being merged, so we wouldn't have any reason to block releases waiting for a PR to get merged like we do currently.
• Major regressions are usually caught by CI or testing, and minor regressions can be handled by reverting problematic PRs (as mentioned earlier), so people encountering regressions can usually just skip a release.
• Bugfix beta releases can be created whenever necessary. Most of the automation for normal beta releases should make this a relatively easy process.
• Stable releases would also become easier with this system, because creating a stable release just means figuring out how to fix any bugs, without needing to merge or cherry-pick requested features or bug fixes.
• Autogenerated changelogs generated by Github releases most likely contain enough info for release changelogs, possibly with a release highlights section added. It might be worth generating those release highlights from PR descriptions (like what sympy/sympy does) but that seems less important since writing release highlights isn't too hard.

[hauntsaninja]

I wonder if having version numbers like 0.1000, 0.1010, etc. is supported. These would look a bit weird, but maybe it would be okay? If that doesn't work out, we can try to fix some highest-impact bugs and implement some planned disruptive changes before releasing 1.0, but with the assumption that some things would still be in flux. We can always release 2.0, 3.0, and so on if there are further major changes. We won't run out of major version numbers :-)

I think version numbers like 0.1000 are supported, but they're confusing, and we should just bump to mypy 1.0 — at our current schedule, this would be September or October :-)

This is my issue for changes I'd see in an ideal 1.0 #11201
Could be nice to use the version change as an excuse to make a couple of them, particularly changes to UI / default options.

[hauntsaninja]

We've had a more regular release schedule recently, we've onboarded more people to work on releases, we have an action that sometimes automatically syncs typeshed, and we have a 1.x version scheme.