Define and document branch strategy for the spec, both development and publishing

[handrews]

This issue consolidates the various interrelated branch / merge / publish / etc. we need to sort out, as discussed in the Moonwalk 19 March and TDC 21 March meetings. This needs to cover:

• Specification development
• Specification release / publishing
• Schema development
• Schema release / publishing
• Any additional publishing currently handled through GitHub pages (the gh-pages branch) such as registries

[EDIT: Schema (#3716) and gh-pages (#3717) have been split out into their own issues.]

I've included notes from the meeting below, followed by my own thoughts.

---

@lornajane's notes from the 21 March 2024 TDC meeting:

We also talked about our git versioning/branching/development workflow and it's clear that we are in need of a change. A fruitful discussion seems to have happened this week in the Moonwalk SIG meeting. A high-level summary of what I understood would be:

• there is one file in the repository called oas.md and this is the canonical spec version (so I guess on main it's the latest stable release? We didn't talk about that)
• we maintain one branch for 3.0, where oas.md is under active development and will become the 3.0.4 release. The branch will be tagged at release time. The same pattern for a 3.1 branch where the next tag would be 3.1.1 and a 3.2 branch where we'll tag 3.2.0.
• when we tag a new version, we also copy the oas.md into the main branch under the versions structure that we have now.

The advantages of this structure are that we can diff the different versions and revisions of the same file in a sane way. There are fewer branches to keep track of so things should be easier to reason about. And it gives a simpler workflow for adding versions (hello 4.0) as we move forward.

My questions: where are we having the discussion about this that isn't in an agenda comment thread? And how do we apply changes such as tooling updates to the minor version branches without affecting the oas.md file? /cc @miqui since I know you're putting something together on this.

As a followup to getting the branches sorted, at some point we can change our use of github pages - currently it uses a branch, which was a very early implementation. We think it would be more useful to use a subfolder now.

---

My thoughts:

• main is where officially published / publishable things live; the versions and schemas directories are just fine as they are, and the tagging can happen as it does now if we want to keep doing that
• I'd recommending initializing a new dev branch purely to serve as the base for this new process. This should include the work-in-progress development directory / files that aren't published, and therefore don't appear on main, initialized from v3.0.4-dev
  • I'd then recommend new release branches, branched from dev, with the correct contents from the existing release branches pulled over, branched accordingly (whatever we decide "branched accordingly" means)
  • I'd strongly advise against anything more complicated; e.g. we do not need "feature branches" as we're not big enough, and folks should work in their forks rather than branching the main repo because the branch list here is confusing enough as it is
  • We also still need to decide if we're keeping GitHub as the cite-able source of truth or if we want the pretty HTML to take that over
• We need to think through how we want schema development to happen.
  • We create one schema per minor release, and allow them to change independent of the spec, so the workflow is different
  • We should decide when to do schema work (while we change the spec? only when we release?) and enforce it strictly
  • We need to sort out the "update $id and publish" workflow that... has some sort of problem with somethign?
  • Some schemas don't have $id which makes the publish workflow impossible right now, so...
  • We still need to figure out how to publish schema changes (see also Host schema publicly #151 and npm package not published? #2400)
• We need to think through how we want the registries to work
  • The whole point is that they're easier to change at any time than the spec, or even the schemas as there is no publishing process other than merging
  • Do we still need GitHub pages or should the registries go with the learn site?

[lornajane]

Yes, all of this. I'm not sure I have enough of a handle on the current state of play to comment on the plans for initialising - and since we need to do new releases and get those into main, I'd be tempted to branch at that point.

Do the registries update with reference to the schema versions? I think they should stay in this repo and be tied to spec changes, but it's not clear if we can update them more easily if they get changes that match spec changes, or not.

[miqui]

steps:
1st - modify developers.md to describe the new reality.

dev branch (current main) - "starting point branch" (create other branches from this one)

[miqui]

Henry - draft PR.

[miqui]

from Karen:

to summarize:

• consistent filename across branches (for spec, schema files etc)
• 3.1.1-dev branch renamed to 3.1-dev; 3.0.4-dev branch renamed to 3.0-dev; there will be a 3.2-dev branch that may also merge changes that are added to 3.1-dev
• when a version is released, it is copied into subdirectories in the main branch; web publishing done from main
• small fixes only to 3.0.x and 3.1.x; spec-impactful changes to 3.2

[miqui]

TODO: schema generation process.

When do we publish schemas?
Work in parallel or at the end of the spec change?

• current status: unknown (potentially not aligned)

Karen: PR spec change, then follow up with a new PR for schema change.

both consistent on dev branch!

[handrews]

In a recent TDC call, we decided that the spec, schema, and gh-pages aspects have different priorities, and likely different people able/willing to work on them. I've split the schema and gh-pages aspects out as follows:

• Schema development: main or branch, in parallel or at release time? #3716
• Should we-organize our GitHub pages to a separate repo (by way of a directory on main)? #3717

This issue will now only track the spec branching policy.

[handrews]

I discovered something today that AFAICT is not written down anywhere, but I wish I'd known it as it would have made some of the 3.0.4/3.1.1 archaeology a bit easier:

Each time a new vX.Y.Z-dev branch has been created, the previous versions versions/A.B.C.md file has been renamed to versions/X.Y.Z.md, which more-or-less preserves the history back to versions/2.0.md (the versions/1.2.md file does not seem to be connected to anything, which is hardly surprising given that we also don't render it as HTML).

So this means that we should create the new dev branch and rename v3.2.0-dev's versions/3.2.0.md file to src/oas.md. That will preserve the maximum history, although of course the rather chaotic cherry-picking and re-applying of fixes across different files and branches, plus the extremely aggressive squashing of commits on release, will continue to make some research impossible.

But this will preserve what we can, and once documented will make at least some diffs available that I struggled to figure out.

[handrews]

Hmm... actually I may have figured out how to preserve the positive aspects of the publication process (connecting the published version/*.md files to the commit history) while doing all release lines on a common file to allow normal git merge usage. I'll need to think about it a bit more.

[handrews]

OK, having spent more time with it now that I've figured out some key things, I think I understand how this was supposed to work. And I feel a lot better about this- this repository was set up by smart people, and I've always been puzzled as to their goal. But now it makes a lot more sense.

The original intent

Basically, it was designed for a workflow in which there were never multiple release lines.

If you never have parallel releases, it's fine to rename a file across each new branch because the actual development is (conceptually) linear, and the branch/file changes are just organizing the conceptual line and ensuring that the published spec is under the correct name.

This explains why asking for a 3.0.4 was seen as... if not strange, then at least something that was not entirely in the plan. IIRC, deciding to do a 3.0.3 after having started 3.1.0 required a bit of discussion, too. In the diagram below, you'll see that that's where the first "pseudo-merge" occurred.

Pseudo-merges

A "pseudo-merge" (a term I just made up) is a porting of commits from one file on one branch to another file on a different branch. This is what we want to eliminate, because it is not well-supported by git (it requires creative use of git format-patch and git am which are not commands most people even know exist).

In the diagram below, I have not made square commits for all of the porting between 3.0.4, 3.1.1, and 3.2.0, because that would be a dizzying mess of arrows in all directions. Instead, I've just treated it as if each release forward-pseudo-merged to the next one as it was released, which is close enough in concept.

Simplified visual

All filenames not prefixed by a directory are under versions/

Square outlines are "pseudo-merges" (treating 3.0.4 -> 3.1.1 and 3.1.1 -> 3.2.0 as having one pseudo-merge each at the end of the earlier-numbered release, instead of the extremely complicated reality).

Circular outlines are normal merges.

gitGraph TB:
  commit id:"add 1.2.md"
  commit id:"Swagger 1.2"
  commit id:"1.2 post-release fixes"
  commit id:"1.2 link and license fixes"
  commit id:"add 2.0.md"
  commit id:"Swagger 2.0"
  commit id:"Swagger 2.0 dev continued past release date"
  commit id:"Swagger 2.0 numerous fixes/expansions"
  commit id:"OAS 2.0"
  commit id:"de-facto OAS 2.0.1"
  branch OpenAPI.next order:1
  commit id:"add mostly empty 3.0.md"
  merge main type: HIGHLIGHT id:"add 3.0.md as a copy of 2.0.md"
  checkout main
  commit id:"de-facto OAS 2.0.2"
  checkout OpenAPI.next
  commit tag:"3.0.0-rc0"
  commit tag:"3.0.0-rc1"
  commit tag:"3.0.0-rc2"
  commit id:"rename 3.0.md to 3.0.0.md"
  checkout main
  merge OpenAPI.next id:"merge 3.0.0.md to main" tag:"3.0.0"
  branch v3.0.1 order:2
  commit id:"rename 3.0.0.md to 3.0.1.md"
  commit id:"3.0.1 work"
  checkout main
  merge v3.0.1 id:"merge 3.0.1.md to main" tag:"3.0.1"
  branch v3.0.2-dev order:3
  commit id:"rename 3.0.1.md to 3.0.2.md"
  commit id:"3.0.2 work"
  checkout main
  merge v3.0.2-dev id:"merge 3.0.2.md to main" tag:"3.0.2"
  branch v3.0.3-dev order:4
  checkout main
  branch v3.1.0-dev order:5
  commit id:"rename 3.0.2.md to 3.1.0.md"
  checkout v3.0.3-dev
  commit id:"rename 3.0.2.md to 3.0.3.md"
  commit id:"3.0.3 work"
  checkout main
  merge v3.0.3-dev id:"squash 3.0.3.md to main"
  branch v3.0.4-dev order:8
  checkout main
  commit id:"update README" tag:"3.0.3"
  checkout v3.1.0-dev
  merge v3.0.3-dev type:HIGHLIGHT id:"pseudo-merge from 3.0.3"
  commit tag:"3.1.0-rc0"
  commit tag:"3.1.0-rc1"
  checkout main
  merge v3.1.0-dev id:"merge 3.1.0.md to main" tag:"3.1.0"
  branch v3.1.1-dev order:6
  commit id:"rename 3.1.0.md to 3.1.1.md"
  commit id:"initial 3.1.1 work"
  checkout main
  branch v3.2.0-dev order:7
  commit id:"rename 3.1.0.md to 3.2.0.md"
  commit id:"initial 3.2.0 work"
  checkout v3.0.4-dev
  commit id:"rename 3.0.3.md to 3.0.4.md"
  commit id:"3.0.4 work"
  checkout v3.1.1-dev
  merge v3.0.4-dev type:HIGHLIGHT id:"pseudo-merge 3.0.4 into 3.1.1"
  checkout v3.0.4-dev
  merge v3.1.1-dev type:HIGHLIGHT id:"pseudo-merge 3.1.1 into 3.0.4"
  checkout main
  merge v3.0.4-dev id:"merge 3.0.4.md to main" tag:"3.0.4"
  merge v3.1.1-dev id:"merge 3.1.1.md to main" tag:"3.1.1"
  checkout v3.2.0-dev
  merge v3.1.1-dev type:HIGHLIGHT id:"pseudo-merge 3.1.1 into 3.2.0"

Loading

The good thing here is that, if you know what you're doing, there's more commit history available than previously thought. Although the squashed merges still obliterated a lot of history that would have been really helpful (and the branches that were squashed are long-deleted, unless someone has an un-pruned fork that has their last positions still available).

The tricky part will be getting 3.1.1 and 3.2.0 history onto a single file that preserves maximum history. I think we just want to branch dev off of v3.2.0-dev, rename versions/3.2.0.md to src/oas.md, and branch v3.1-dev and v3.2-dev off of that. And then there's something with release branches to publish by re-naming src/oas.md to the appropriate file on main, but that will have to wait until tomorrow for me to think through it a bit more and add another diagram.

[handrews]

OK, I have thought about how to make the switchover.

Here's a diagram with abbreviated history and the new proposal:

• This starts with the 3.2 release to show the previous (3.0.3 / 3.1.0) and current (3.0.4 / 3.1.1) releases
• The older branches are shoved over and start in the 4th column
• Note that v3.2.0-dev is not shown, as we should abandon it as explained below the diagram

gitGraph TB:
  commit id:"merge 3.0.2.md to main" tag:"3.0.2"
  branch v3.0.3-dev order:7
  checkout main
  branch v3.1.0-dev order:8
  commit id:"rename 3.0.2.md to 3.1.0.md"
  checkout v3.0.3-dev
  commit id:"rename 3.0.2.md to 3.0.3.md"
  checkout main
  merge v3.0.3-dev id:"squash 3.0.3.md to main"
  branch v3.0.4-dev order:10
  checkout main
  commit id:"update README" tag:"3.0.3"
  checkout v3.1.0-dev
  merge v3.0.3-dev type:HIGHLIGHT id:"pseudo-merge from 3.0.3"
  checkout main
  merge v3.1.0-dev id:"merge 3.1.0.md to main" tag:"3.1.0"
  branch v3.1.1-dev order:9
  commit id:"rename 3.1.0.md to 3.1.1.md"
  commit id:"initial 3.1.1 work"
  checkout main
  checkout v3.0.4-dev
  commit id:"rename 3.0.3.md to 3.0.4.md"
  checkout v3.1.1-dev
  merge v3.0.4-dev type:HIGHLIGHT id:"pseudo-merge 3.0.4 into 3.1.1"
  checkout v3.0.4-dev
  merge v3.1.1-dev type:HIGHLIGHT id:"pseudo-merge 3.1.1 into 3.0.4"
  checkout main
  merge v3.0.4-dev id:"merge 3.0.4.md to main" tag:"3.0.4"
  merge v3.1.1-dev id:"merge 3.1.1.md to main" tag:"3.1.1"
  branch dev order:1
  commit id:"rename 3.1.1.md to src/oas.md"
  branch v3.1-dev order:2
  commit id:"update src/oas.md to 3.1.2"
  checkout dev
  branch v3.2-dev order:3
  commit id:"update src/oas.md to 3.2.0"
  commit id:"add Security Scheme deprecation"
  commit id:"add Device Code authorization"
  commit id:"add oauth2MetadataUrl"
  commit id:"add ciba support"

Loading

The new approach starts from the release of 3.1.1 on main:

1. main: Create a dev branch from the 3.1.1 release tag
2. dev: Rename versions/3.1.1.md to src/oas.md
3. dev: Create v3.1-dev from the rename commit
4. v3.1-dev: Initialize 3.1.2 work by updating src/oas.md's version number
5. dev: Create v3.2-dev from the rename commit
6. v3.2-dev: Initialize 3.2.0 work by updating src/oas.md's version number
7. v3.2-dev: Apply ported changes from the old v3.2.0-dev branch, because there aren't many
   1. Security Scheme deprecated
   2. OAuth Flows device code authorization
   3. oauth2MetadataUrl
   4. CIBA (which had some markdown link errors in it, which I fixed)

Note that I just left out the Path Item $ref changes, since we already agreed to back those out. Now we don't have to do any work for that.

This gives us the proper relationship between 3.1.1 (as released) and both the v3.1-dev and v3.2-dev branches, and leaves 3.2.0 with a much cleaner revision history, showing only the changes that aren't part of 3.1.x.

I'll make a separate comment about how to release from these branches.

[handrews]

OK here's how the new approach is proposed to work, starting from the tagged release of 3.1.1 on main, using X.Y-dev branches for development lines, and X.Y.Z-rel branches for doing the release, including renaming src/oas.md to versions/X.Y.Z.md so that we preserve history while publishing to the correct file.

The following diagrams show:

• A 3.1.2 release made while 3.2.0 development continues
• 3.2.0 and 3.1.3 releases going out together, with a merge from the v3.1-dev commit that is the base for v3.1.3-rel into v3.2-dev right before creating v3.2.0-rel
• v3.1-dev and v3.2-dev updated for their next patch releases
• A new v3.3-dev branch created by either:
  • just branching from v3.2-dev (first diagram)
  • merging the 3.2.0 release to dev and branching from dev (second diagram)

No merges back to dev

We could just make each release branch off of the previous release branch:

gitGraph TB:
  commit id:"merge 3.1.1.md to main" tag:"3.1.1"
  branch dev order:1
  commit id:"rename 3.1.1.md to src/oas.md"
  branch v3.1-dev order:2
  commit id:"update src/oas.md to 3.1.2"
  checkout dev
  branch v3.2-dev order:5
  commit id:"update src/oas.md to 3.2.0"
  commit id:"add Security Scheme deprecation"
  commit id:"add Device Code authorization"
  commit id:"add oauth2MetadataUrl"
  commit id:"add ciba support"
  checkout v3.1-dev
  commit id:"a 3.1.x fix"
  branch v3.1.2-rel order:3
  commit id:"rename src/oas.md to versions/3.1.2.md"
  checkout main
  merge v3.1.2-rel tag:"3.1.2"
  checkout v3.2-dev
  commit id:"self-identifying documents"
  commit id:"hierarchical tags"
  checkout v3.1-dev
  commit id:"update src/oas.md to 3.1.3"
  commit id:"another 3.1.x fix"
  checkout v3.2-dev
  commit id:"extensible data modeling"
  commit id:"streaming JSON"
  commit id:"full use of query string"
  merge v3.1-dev id:"merge 3.1.x fixes before releasing"
  checkout v3.1-dev
  branch v3.1.3-rel order:4
  commit id:"rename src/oas.md to versions/3.1.3.md"
  checkout v3.2-dev
  branch v3.2.0-rel order:6
  commit id:"rename src/oas.md to versions/3.2.0.md"
  checkout main
  merge v3.1.3-rel tag:"3.1.3"
  merge v3.2.0-rel tag:"3.2.0"
  checkout v3.1-dev
  commit id:"update src/oas.md to 3.1.4"
  checkout v3.2-dev
  commit id:"update src/oas.md to 3.2.1"
  branch v3.3-dev order:7
  commit id:"update src/oas.md to 3.3.0"

Loading

Merge minor releases back to dev

Or we could merge the minor releases back to dev, but leave patch releases on the release line branches as they can overlap in ways that don't make sense on a single branch:

gitGraph TB:
  commit id:"merge 3.1.1.md to main" tag:"3.1.1"
  branch dev order:1
  commit id:"rename 3.1.1.md to src/oas.md"
  branch v3.1-dev order:2
  commit id:"update src/oas.md to 3.1.2"
  checkout dev
  branch v3.2-dev order:5
  commit id:"update src/oas.md to 3.2.0"
  commit id:"add Security Scheme deprecation"
  commit id:"add Device Code authorization"
  commit id:"add oauth2MetadataUrl"
  commit id:"add ciba support"
  checkout v3.1-dev
  commit id:"a 3.1.x fix"
  branch v3.1.2-rel order:3
  commit id:"rename src/oas.md to versions/3.1.2.md"
  checkout main
  merge v3.1.2-rel tag:"3.1.2"
  checkout v3.2-dev
  commit id:"self-identifying documents"
  commit id:"hierarchical tags"
  checkout v3.1-dev
  commit id:"update src/oas.md to 3.1.3"
  commit id:"another 3.1.x fix"
  checkout v3.2-dev
  commit id:"extensible data modeling"
  commit id:"streaming JSON"
  commit id:"full use of query string"
  merge v3.1-dev id:"merge 3.1.x fixes before releasing"
  checkout v3.1-dev
  branch v3.1.3-rel order:4
  commit id:"rename src/oas.md to versions/3.1.3.md"
  checkout v3.2-dev
  branch v3.2.0-rel order:6
  commit id:"rename src/oas.md to versions/3.2.0.md"
  checkout main
  merge v3.1.3-rel tag:"3.1.3"
  merge v3.2.0-rel tag:"3.2.0"
  checkout dev
  merge v3.2-dev id:"update dev with minor releases"
  branch v3.3-dev order:7
  checkout v3.1-dev
  commit id:"update src/oas.md to 3.1.4"
  checkout v3.2-dev
  commit id:"update src/oas.md to 3.2.1"
  checkout v3.3-dev
  commit id:"update src/oas.md to 3.3.0"

Loading

---

I don't have strong feelings as to which we do. I started just branching new release branches from old ones, but maybe merging minor release content back to dev before making a new minor release branch is a bit nicer.

I'm also open to entirely different proposals. This just seems to get us where we need to go while still preserving the published file names and their git history across renames.

[handrews]

If we need to do post-release work on released documents, we could do that on the vX.Y.Z-rel branches, or we could just do it on main like we did before. This is so rare, and requires such a strange situation (other people breaking links) that I'm not worried about it either way.

Note that this proposal does not address schema development, which needs to be independent of the spec releases and is being tracked in #3716.

[ralfhandl]

Why do we need the dev branch? Seems we could simplify the "No merges back to dev" picture into a "No dev" picture without losing anything.

[handrews]

@ralfhandl dev is where we establish src/oas.md as the document that we work on. It makes things easier to figure out for future people if (heaven forbid) we again lose access to all of the people who know the branch history.

Please don't make a future me have to figure out that there was never a proper start and we just renamed things from version branch to version branch. Give that future repo archaeologist a sensibly-named branch that does what it seems to and provides the basis for all of the working branchs. Please.

[handrews]

@ralfhandl another reason to have dev is that we might migrate schema development to it as well, but there would be a different set of branches from dev for schemas than for specs as they are released independently. Having a dev gives us a base for other development activities that might come up, and insulates our overall process from main, which should really only contain publishable resources plus necessary infrastructure.

dev is an insurance for flexibility