Create patch releases when significant issues are found

[squaregoldfish]

In the recent CF Conventions workshop we were shown some instances where some significant errors were discovered in a release shortly after publication. These were not rectified until the next annual release. I propose that CF considers making patch releases in such significant cases, rather than knowingly having serious issues in the latest release that go uncorrected for an extended period.

As a side effect of this, maintaining a "Kown Issues" list for the conventions will help users to know when these situations arise.

[JonathanGregory]

Dear Steve @squaregoldfish

Thanks for raising the issue of rectifying mistakes in the convention. We do actually have a provision for this in the rules for CF convention changes, as follows:

If [a] change, once implemented in the conventions, subsequently turns out to be materially flawed, meaning that data written following the convention could be somehow erroneous or ambiguous, a github issue should urgently be opened to discuss whether to revoke the change. If this is agreed by a majority of the committee, a new version of the conventions will be prepared immediately, with the second digit of the version number incremented, and will be recommended to be used instead of the flawed version. The flawed version will be deprecated by a statement in the standard document and the conformance document. However, any data written with the flawed version will not be invalidated, although it may be problematic for users. Errors or lack of clarity in wording, when the convention itself is not at fault, can be corrected by defect tickets on the usual schedule.

This would be what you call a patch release - right? We could have done this to correct either of the two errors that have so far occurred, but it was decided not to because the affected features were unlikely to have been used much in existing data, in one case because it seemed a rare case, in the other because it was brand new. I can't remember the discussion clearly, but I suspect that one consideration was that it is quite time-consuming to make a release. I think that when a mistake is corrected we should record in the convention text that this has been done, to alert users to the possible existence of erroneous data. This practice isn't mentioned in the rules (quoted above) at the moment.

I agree that a list of known issues on the website would be useful, recording when each mistake was introduced and when it was corrrected.

Best wishes

Jonathan

[JonathanGregory]

Another point is that the rules, as quoted, only envisage revoking the change to deal with the problem. Correcting it may be possible as well, which is what we did in the two cases so far. Making a correction would require an enhancement to the convention, with the usual three-week rule.

[jesusff]

I think Steve might be referring to a patch release as in semantic versioning (https://semver.org), adding a third digit specifically for this purpose (e.g. 1.10.1). According to the quote above, current rules indicate to increase the second digit (e.g. 1.11) for this patch release. This could be misleading, though, since the flawed version will go unnoticed in the version numbering series. In semantic versioning, the presence of this third number already indicates a flaw in the base version. E.g. if 1.10.1 exists, this already indicates that there was some problem with 1.10, and users should look into the release notes to see if the bug is affecting their work.

This 3-digit semantic versioning seems to have been in place for some time, since releases prior to 1.10 had it: https://github.com/cf-convention/cf-conventions/releases There was even a 1.7.3 release, indicating 3 patch releases should exist, but no release notes indicate the bugs corrected (a.k.a. known issues in the previous version). At least, these notes are not directly accesible from the github release page.

[squaregoldfish]

@jesusff has what I was thinking - I should have been more explicit.

[JonathanGregory]

Thanks for explaining, @jesusff. Version 1.7 was unusual because it was the first in AsciiDoc and probably the patch releases arose from the conversion to this new format. I don't think 1.7.[12] were really "releases" used in practice.

So the question is whether there is a reason to distinguish between a release made for enhancements (the usual) or to correct flawed versions (very unusual). As you say, the rules envisage treating them the same way (by incrementing the second digit). You are suggesting that if we add a third digit for versions correcting flaws, it provides an extra warning to those who recognise that convention. I agree that would be useful. I can't remember whether there were arguments why we didn't follow that convention when writing the rules. Perhaps someone else does.

[squaregoldfish]

Nothing in the rules states that CF uses semantic versioning, but it is mentioned in the Pull Request Template https://github.com/cf-convention/cf-conventions/blob/main/.github/PULL_REQUEST_TEMPLATE.md:

Next version in cf-conventions.adoc up to date? Versioning inspired by [SemVer](https://semver.org/).

which is why I thought that patch releases in their terms would be an obvious thing to be doing.

I think that it is useful to have the distinction built in to the version numbers: 1.10.1 states that there is nothing "new" in the specification (it's only fixing existing things) while 1.11 means there is a change in the intended behaviour (for want of a better word) of the spec.

PS. I fully agree that there is a judgement to be made regarding whether a given flaw is significant enough to merit a patch release and the related work involved. I think at least one of the examples given at the workshop would have qualified.