Initial draft of proposed Registry process

[jandrieu]

This outlines a set of proposed rules for managing the DID Method Rubric as a registry.

---

Preview | Diff

[peacekeeper]

Looks like I originally misunderstood the proposal a bit. I thought the proposal was that this would become a registry primarily for adding concrete DID method evaluations. But apparently it's a registry for adding new criteria plus some other things like example evaluations. I think this is super useful! I also like how it mentions that evaluations can happen elsewhere (see #45). +1 to this.

[iherman]

The issue was discussed in a meeting on 2021-08-31

• no resolutions were taken

View the transcript

7. DID Rubric

See github pull request did-rubric#49.

Joe Andrieu: This is the PR that has the proposed registry rules for the DID Rubric registry

Brent Zundel: is there a link to this presentation?

Joe Andrieu: I will share a link with the mailing list
… and I will share my screen to convey the key points
… the slide deck first covered what has been done
… then an evaluation of the rubric was done on the Veres One method
… this report was then shared with the community
… next they are evaluating did:web and did:ion. Both of these will be published when they are ready.
… SBA research also did some evaluations
… the main points we learned is that the Rubric is still in its infancy
… some questions were too academic
… also, structure-variable questions are needed
… also, enforcement was needed
… design was also not included
… implementations may also need to be evaluated separately
… and adversaries need to be evaluated

Ted Thibodeau Jr.: These changes are glorious. I'm very happy about how you have been doing this, and the direction Rubric things are going!

Joe Andrieu: all of these were gaps in the Rubric
… we also need better tools for community engagement
… we also need more discussion about these criteria
… the Rubric also proposes that each evaluator essentially create their own custom rubric with the criteria they need
… shared evaluations also would help
… the Rubric also needs permalinks and persistent identifiers so that references to the Rubric criteria will not break
… the proposal is to turn the Rubric into a registry where criteria can be added, updated, and curated
… updates can be done with a simple PR
… the desire is that the current DID Rubric authors recommend a starting set of rules
… so the net net is that it becomes a mechanism for continuing to improve the criteria for evaluating DID methods
… there are a number of questions about how to proceed

Brent Zundel: RE deadlines, as a WG, we have 4 more WG meetings that we can use to have this conversation
… there is a possibility the WG could be extended to address any objections
… so the last opportunity is by the end of Sept for the existing WG to approve the Note to be taken on by the new WG
… please review the proposed rules

Joe Andrieu: The proposal follows the template of the DID Spec Registries doc
… criteria must be identified and versioned
… subcomponents do not need to be versioned and permalinks
… use cases, methods, and evaluations all need to be cited
… the proposal defines what is needed in a criteria. Each proposed criteria needs at least 3 examples.
… then, for all of those fields, what is required for each field
… it also defines identifiers and how they need to work
… there is a way to provide a TR permalink
… prior criteria will still be retained in future versions in a "Retired" section so that the permalink will still work
… the versioning rules are also defined
… there is also an escalation path for disputes. However the key difference here, the editors retain the ability to curate the content
… that gives the editors more responsibility to maintain the list of curated criteria PLUS a few example evaluations
… that was done in the original Rubric, but that gave excessive visibility to six specific DID methods that they should not have

Brent Zundel: we have 10 mins left to discuss

Drummond Reed: Looking for some clarification here, I originally understood from the last call that this is about the registering of evaluations, but it seems this is about registering criteria. Is this about one or both?
… examples would be cited for showing how to evaluate a criteria, but not for registering the evaluations directly

Ted Thibodeau Jr.: I am really happy about how the Rubric is evolving. I very much like this path, and I hope it continues evolving forever.

Joe Andrieu: I'm very glad to see that. The work of evolving the Rubric has taught us a great deal, and it will continue.

Kyle Den Hartog: I also want to second its usefulness. We learned a lot when evaluating did:key.
… it was possible that my misunderstanding led the WG members astray on the last WG call
… but now that this is clarified, I think this is a great tool

Drummond Reed: what are next steps?
… what are we considering as the next steps to take for this to fit it within the working group time we have left?

Brent Zundel: Next steps are to review the PR ASAP
… the sooner we have feedback on that, the sooner we can have the Rubric in a state where the new WG can take it over as a registry

Joe Andrieu: I will send out an email to the list with the PR and ask for feedback.
… it is seven pages and it has a lot of detail
… Daniel Hardman already caught several improvements that were needed

Brent Zundel: That looks great. Looking forward to the feedback.

Kyle Den Hartog: Wanted to say thanks for accommodating this time zone for one meeting a month

Shigeya Suzuki: +1 for kdenhartog. it works for me well too!

Brent Zundel: Thanks to everyone, and especially Joe for the Rubric registry PR, and to scribes, and to wrapping it all up in Sept.
… also, please jump into the DID WG Charter issue and "make your feelings known"

---

[iherman]

My comment is purely editorial.

• Wouldn't it be cleaner if all current entries in §3 followed exactly the same structure (in terms of subtitles) as the entries listed in §2.2? This is mostly done, but the name, id, and version, are never made explicit in §3 (I realize those items are all trivially there, but it would probably be better to make it 100% explicit).
• "Each criteria must be explicitly, uniquely, and persistently identified using incremental numbers. New criteria should use the next available increment based on the highest numbered identifier in the current publication." which is fine, but we may create a race condition, so it should be made clear that the editors of the registry may have to adapt those numbers during the registration process itself.
• I would think that §3.9 should either be removed or, rather, refer to the registration process.
• Shouldn't something be said, in §2, who maintains the registry? Who has the decision power, at the end of the day, on whether a submission abides to the rules? I presume it is, officially, the maintainer of the DID Spec proper, ie, the DID WG or, in the absence thereof, the W3C Team, but that has to be made more explicit imho.
• "vioations" -> "violations" (in second bullet §2.1

[iherman]

The issue was discussed in a meeting on 2021-09-07

• no resolutions were taken

View the transcript

2.1. Initial draft of proposed Registry process (pr did-rubric#49)

See github pull request did-rubric#49.

Daniel Burnett: main point is to convert DID rubric to a registry - this is an issue that is out
… before the existing group charter ends - important to get done before we shift to a maintenance charter.

Drummond Reed: more in-depth and complex

Markus Sabadello: I misunderstood this proposal at first - it is just one part of it - just about adding additional criteria - I like it

Ivan Herman: I read it this morning - important to have it done as quickly as possible - so put in DID new charter that this will turn into an official registry document.

Drummond Reed: +1 to Ivan's point -- we want to include this in the new charter

Ted Thibodeau Jr.: I had started going over this with a fine tooth comb and made a stack of suggestions - I didn't quite understand how things were flowing as documents - main author should go through my comments and then I can do - they have been sitting there for 5 days

Daniel Burnett: If he wants this to get in he needs to be very responsive to comments

[TallTed]

Suggested change

	why that particular response was chosen.It MAY be blank.
	why that particular response was chosen. It MAY be blank.

[iherman]

The issue was discussed in a meeting on 2021-09-14

List of resolutions:

• Resolution No. 1: The DID WG will convert the DID Rubric into a registry for DID Method evaluation criteria.

View the transcript

5. Initial draft of proposed Registry process (pr did-rubric#49)

See github pull request did-rubric#49.

Brent Zundel: PR 49 has been open for a while
… good discussion so far. Summary is that PR adds text necessary to convert doc into a registry where new criteria can be added over time
… feedback received, responded to, etc.

Joe Andrieu: Would be good to get a formal approval to accept this. Still some editorial to do.
… Just got in the last Echidna fix. Let's agree to move forward with this.

Proposed resolution: The DID WG will convert the DID Rubric into a registry for DID Method evaluation criteria. (Brent Zundel)

Ted Thibodeau Jr.: +1

Drummond Reed: +1

Charles Lehner: +1

Dmitri Zagidulin: +1

Joe Andrieu: +1

Orie Steele: +1

Brent Zundel: +!

Daniel Burnett: +1

Michael Prorock: +1

Brent Zundel: +1

Markus Sabadello: +1

Ivan Herman: +1

Ryan Grant: +1

Resolution #1: The DID WG will convert the DID Rubric into a registry for DID Method evaluation criteria.

Justin Richer: +0 (don't think it's helpful but whatever)

Brent Zundel: as soon as the PR gets editorial cleanups we can merge. Thanks all!

Drummond Reed: Yes, it's a major step forward. Thanks Joe!

Joe Andrieu: thanks for your support!

Ivan Herman: to be clear, we expect new DID charter will change after formal objection is resolved. Once your PR is merged I can take out the caveat we have in the charter text, right?

Brent Zundel: yes

[jandrieu]

• Wouldn't it be cleaner if all current entries in §3 followed exactly the same structure (in terms of subtitles) as the entries listed in §2.2? This is mostly done, but the name, id, and version, are never made explicit in §3 (I realize those items are all trivially there, but it would probably be better to make it 100% explicit).

Moved to issue #56

[jandrieu]

• "Each criteria must be explicitly, uniquely, and persistently identified using incremental numbers. New criteria should use the next available increment based on the highest numbered identifier in the current publication." which is fine, but we may create a race condition, so it should be made clear that the editors of the registry may have to adapt those numbers during the registration process itself.

The section on 2.3 Identifiers says

Editors will manage any sequencing errors when accepting PRs.

Does this address your suggestion, Ivan? Or is there another place in the document we should/could highlight that point better?

[jandrieu]

• I would think that §3.9 should either be removed or, rather, refer to the registration process.

Agreed. I added a new issue to track this. We should also get rid of the "additional criteria" section or at least update it to fit with the registry.

[jandrieu]

• Shouldn't something be said, in §2, who maintains the registry? Who has the decision power, at the end of the day, on whether a submission abides to the rules? I presume it is, officially, the maintainer of the DID Spec proper, ie, the DID WG or, in the absence thereof, the W3C Team, but that has to be made more explicit imho.

Hmmm. We do say that at the end of Section 2, but it's not well highlighted. I'll break that out into its own section, 2.6.

[jandrieu]

@dhh1128 This should be ready to go. I've created a few new issues to track some good suggestions that I'd like to get in later, but those are all editorial and shouldn't hold up this merge.

Let me know if you have any questions.