Further JSON processing

[annevk]

Context:

• JSON parsing not defined WICG/origin-policy#32
• Describe the manifest data structure w3c/manifest#611

Example of the complexity needed today:

• https://w3c.github.io/payment-method-manifest/#validate-and-parse

Folks involved: @mgiuca @tobie @marcoscaceres @kenchris @domenic @zcorpan @dhausknecht

Not having to deal with JavaScript seems like a big step up and something we should cover. Basically convert the output to IDL/Infra types. Note that we'll have to keep the existing hook that exposes just JavaScript, as that's what Fetch and XMLHttpRequest need.

Then there's the question if we should cover more and what kind of model we want. I basically think what we want universally is to ignore a field if it's value is an unknown type. And we don't want 123 to become "123" if the field is of type string (IDL's behavior for JavaScript).

It seems like offering some kind of validation language could be useful as otherwise everyone has to reinvent it which opens up potential differences in processing, which is unlikely what web platform consumers want.

[annevk]

And we don't want 123 to become "123" if the field is of type string (IDL's behavior for JavaScript).

Note that if we do accept this, we could probably reuse most of the IDL dictionary infrastructure here.

cc @mnot @mikewest

[domenic]

I'll want this infrastructure for https://github.com/domenic/package-name-maps and will probably give a shot at generalizing it while I work on that.

I don't think we want to allow type conversions.

[annevk]

https://www.w3.org/TR/tracking-dnt/#status-representation is another example of something browsers might consume.

[domenic]

I think there are potentially two issues here:

1. How to write normative spec text that parses JSON into spec-level data structures (infra ordered maps/lists, plus the string/number/null/boolean primitives (see Define numbers (waiting on Number / BigInt) #87))
2. How to express the expected structure of your JSON, and validate that the parsed JSON conforms to that structure.

(1) I think is straightforward: we basically use the existing parse JSON operation which gives a JS object, then add some stuff in Infra that crawls the JS object and produces spec-level data structures from it. Then no other spec ever has to worry about JS objects; Infra just abstracts that away into a simple bytes/string -> spec-level data structures operation.

(2) is trickier, and perhaps belongs in a separate issue. For simple cases, you can just check in prose; this will probably be necessary anyway in most cases (e.g. checking that the contents of a string are in the expected format).

Currently some people use Web IDL for (2), but this has pretty bad semantics. (E.g. for a field that expects a string, you could give it { foo: "bar" } instead, and your algorithm would see the string "[object Object]" due to IDL's conversion.) JSON schema seems like it should be the answer, but it's impressively ugly and unreadable... or more charitably, it's very unlike what most readers of web specs are used to.

I'm curious on others' thoughts on (2), as this does continually keep coming up.

[annevk]

You don't want to validate. If something is a number instead of a string, what you most likely want is to ignore that member and not have it in your returned data structure. As some kind of forward-compatible "error" handling.

And yeah, that would mean defining yet another schema/IDL of sorts.

[domenic]

I suppose that's true, if the field is optional anyway. If the field is required and you're expecting a string, you want to error out.

[annevk]

Yeah, at which point you kinda need a schema available while converting (or you have a schema for specification data structures so you can do post-conversion cleanup).

[tobie]

So I’ve used JSON schema for specifying and testing the data structures for Specref. I agree it’s not the best thing ever, but let’s be fair, neither is WebIDL. I got the hang of it pretty quickly, and it mostly does the job it’s designed for. I don’t know that there are many existing options that are better.

[annevk]

@tobie does JSON schema do the kind of thing we need though? We need something that's IDL-like, not DTD/RelaxNG/etc.-like.

[tobie]

@annevk JSON schema doesn't do any conversion for you, iirc.

[domenic]

For folks watching this popular thread, we've just added a parse JSON into Infra values algorithm. This gives you back Infra ordered maps/lists/booleans/numbers/strings/nulls, which your spec can operate on without having to (ab)use JavaScript abstract operations or Web IDL dictionaries.

You can see a not-quite-finished usage example of this in a parsing setting in WICG/import-maps#130 (preview), or hopefully in a day or two, in https://wicg.github.io/import-maps/.

I think there is perhaps room for a future JSON schema-based solution for validation, or standardized documentation format for JSON, but for the purpose of specifying processing models, I suspect this is going to be the best way for a while, and will closely match implementations.

[foolip]

Does this issue also track adding a variant of https://infra.spec.whatwg.org/#serialize-json-to-bytes which would take Infra values as the input instead of JS values? We might need it in w3c/webdriver-bidi#56.

[annevk]

No, this is really about doing validation in a consistent manner.

[foolip]

OK, I've filed #336 for that instead.