Use Bikeshed pre-processor for WebDriver

[andreastt]

I would like to propose converting the specification to use the Bikeshed spec pre-processor.

Bikeshed produces specification documents that are equivalent to ReSpec that we currently use, but instead of transforming the source document at load-time (through content JavaScript) it pre-processes the source document into a complete, finished HTML document as a build step.

As many of you know I have a distaste for hard-to-explain build steps and a love for the mutable what-you-see-is-what-you-get web, so I am not the natural proponent of this kind of change. But I have come to the realisation that some of the forthcoming changes we need to make for the bi-directional protocol would be easier to tackle if we can run tasks as build steps.

One of the takeaways from TPAC 2019 in Nagoya, Japan was that we need to make WebDriver more discoverable for web authors, client maintainers, and browser developers. There are various things we could do, but I believe the most effective is to make the specification machine-readable.

With the proliferation of APIs that we expect to see with the bi-directional work, we will be moving from a human-managable set of endpoints to a reality where computer automation would be necessary to assist with ingesting changes. Having formalised schema definitions for the different interfaces, methods, and types will enable wider and earlier adoption of new APIs because the specification is offering a useful service to implementors.

Now to the point of this issue and why it means we need a pre-processor:

I picture that the schemas should be defined in separate files/artifacts, rather than be inlined in the source spec document. This would allow generation of API documentation (using Swagger or something equivalent) to serve as client maintainer documentation, MDN browser compat data and API documentation, and—most importantly!—generation of source files ready for use in browser implementations (for example into C++ or Rust) for schema validation, or local-end clients (Go/C#/Java).

With schema definitions as separate artifacts, ingesting them into the document would require a build step. We could hack together the inlining of interfaces for ReSpec with m4(1) or sed(1), but if there’s already a necessity to add a build step it would seem more obvious to adopt a spec authoring tool that already supports inclusion of files.

Here are some arbitrary examples of Bikeshed-backed specs from W3C, WHATWG, and non-affiliated:

• Encoding
• HTML
• Subresource Integrity
• URL
• WTF-8

Pros:

• Syntax is not too different from ReSpec
  • … and Bikeshed arguably has better documentation than ReSpec
• Allows interface definitions to live separately from the spec, enabling their use for other purposes:
  • MDN browser compat data
  • MDN API listings
  • API documentation generation (Swagger or similar)
  • Source code generation for browsers (C++/Rust) and clients (Go/C#/Java)
• Correctness checks can be done when PRs are created because pre-processor validates source document for parsing
• Loading https://w3c.github.io/webdriver/ should be a lot quicker
  • Compare with https://encoding.spec.whatwg.org/
  • We should experience less “juttering” because document has been pre-processed
• Spec can be viewed locally without internet connection
  • With ReSpec we rely on a bunch of remote resources from www.w3.org
• We won’t be hit by upstream ReSpec bugs
  • We rely on ReSpec from a remote server, meaning upstream bugs hit us immediately and retroactively affects already published documents

Cons:

• Requires build step to try out changes locally (view in browser)
  • Installation of Bikeshed dependency shouldn’t be necessary with internet access because we can have remote HTML generation
• Cannot make real-time changes (e.g. with DevTools) and see the transformed result immediately
• The conversion itself is a bit of busy-work, as we could hack the inlining of interfaces with ReSpec

Overall I don’t think the change should impact our workflow too much.

[foolip]

Does doing this have support among the editors and others impacted? I've done some conversions to Bikeshed before, am pretty good at vetting for unintended differences, and might volunteer if it's likely to be accepted.

[AutomatedTester]

Both @shs96c and I don't mind what tool is used. I am happy for it to be moved over, I just haven't prioritised this work

[foolip]

I'd be happy to take a stab at this. There are 11 open pull requests in https://github.com/w3c/webdriver/pulls right now, and rebasing them will be an entirely manual affair. Are there any that can be merged or closed to reduce the amount of work needed?

[christian-bromann]

@foolip it would be great if this protocol could be machine readable as it would simplify the creation of rudimental bindings that can be used to build a framework with higher level abstractions as well as for documentation. For example: the Chrome DevTools protocol is maintained as a pure json file and is used to build its documentation page as well as CDP modules like chrome-remote-interface. This can safe a lot of time and duplication for adopters building packages around the protocol.

Another example is from the WebdriverIO project where I've defined a custom JSON format of the WebDriver spec that is used to autogenerate a minimal functional binding. While I think a spec text is not necessarily suited as user documentation - being able to have the ability for type checks especially in the JS ecosystem would be very beneficial.

A while ago I tried to convert the current WebDriver protocol into an OpenAPI spec and was able to generate a page like this: https://webdriver.github.io/webdriver/. I can imagine that it should be not that difficult to build a bikeshed pre processor that reads an OpenAPI spec and coverts it into a spec document.

I would be happy to help contribute and drive the effort for a machine readable spec if you all think it is worthwhile.

[bwalderman]

I played around with OpenRPC when writing the explainer. There's an example here. I ran into some minor issues. The spec supports only one-way calls (no concept of bidi) so I ended up using tags to mark things as either "events" or "commands". The tooling support also seems pretty limited. There's a code gen tool but this supports only TypeScript and Rust at the moment.

AsyncAPI looks promising instead. It's more generic and can express both HTTP and WebSocket transports in the same file. It also has proper support for bidi communication through "channels".

[foolip]

In the initial conversion for this I'd try to have as few visual and markup changes possible, as looking for differences is the best way to catch mistakes. I wouldn't want to make any deliberate change at all, even typo fixes, in that initial change.

@christian-bromann @bwalderman I suspect you're mostly interested in machine readable for bidi, right? Can you file an issue for that? If you'd like the main webdriver spec to also be more machine readable, an issue for that would be great too.

[foolip]

I've filed #1510 and w3c/webdriver-bidi#21 for machine readable definitions for the REST and BiDi protocols, respectively.

[foolip]

I'd like to ask the input of regular WebDriver spec contributors about ID stability. Given a definition like <dfn>a thing</dfn>, ReSpec will generate id=dfn-a-thing while Bikeshed will generate id=a-thing, i.e., ReSpec prepends dfn-. IDs can be provided manually in both cases, of course.

Preserving all IDs is possible, but would make the source fairly verbose. I'd therefore like to suggest allowing a lot of IDs to change, while making an effort to not break links. One thing we can do is create a mapping of IDs to be checked client-side in JS. HTML does some stuff like this.

The most important thing is to not break links from other specs. Using reffy-reports and Bikeshed's tests there are the specs that link to WebDriver that I can find:

• https://w3c.github.io/permissions/#automation
• https://w3c.github.io/reporting/#automation
• https://w3c.github.io/sensors/#automation
• https://w3c.github.io/webauthn/#sctn-automation
• https://wicg.github.io/background-fetch/#automation
• https://immersive-web.github.io/webxr-test-api/ (mention only, no links)

My questions:

• Are there other places with incoming links that should be checked?
• If we're confident enough that the important incoming links have been accounted for, is it OK to drop a lot of IDs?
• Would taking care of the remainder with JS be OK, or will it get in the way of publishing to /TR/ later?

[marcoscaceres]

So, obviously I'm super biased (as I maintain ReSpec), but I don't see any advantage to using BS over ReSpec. ReSpec supports the same cross referencing features are BikeShed, and it's really a group's choice if they enable GitHub pages or not for a spec (you don't need to use a browser to transform the output - or let anyone in a browser view ReSpec document... you can generate a static snapshot in exactly the same manner as BikeShed does).