how unevauatedXXX interacts with older drafts

[santhosh-tekuri]

consider I have following files:

$ cat schema.json
{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$ref": "old.json",
    "unevaluatedProperties": false
}
$ cat old.json
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "properties": {
        "foo": true
    }
}
$ cat inst.json
{
    "foo": "bar"
}

if I validate inst.json with schema.json what will be the result? is it valid or invalid?

I guess draft7 does not generate annotations for properties keyword. So property foo is still considered unevaluated.

currently my validator implementation tracks what is being evaluated irrespective of draft version. I would like to know the expected behaviour.

[Relequestual]

That's a really interesting question.
Ideally, I assume, you would WANT the foo property to be counted when determining unevaluatedProperties, and that IS what at least one implementation does (I tried it on https://json-schema.hyperjump.io).

However, I'm not sure I can find supporting requirements from the specifications.

Annotations existed in draft-07, but I don't see that we defined that properties would produce annotations. We later defined that various keywords produced annotations for use by other keywords.

However, I also know (I think?) that Hyperjump doesn't use the annotations process fully for this purpose, so I'd have to nudge @jdesrosiers to explain why this works in his implementation. I'll also nudge @gregsdennis to see if he can test this with his implementation. (Jason's is the only one which allows for multiple files in a playground, and I don't have a .net envrionment.)

[Julian]

Never too late to learn Bowtie :), looks like the result if you care about all implementations that support 2020, 7, and are supported by Bowtie is:

⊙  bowtie run -i js-ajv -i rust-boon -i js-hyperjump -i go-jsonschema -i dotnet-jsonschema-net -i clojure-json-schema -i cpp-valijson -i ts-vscode-json-languageservice -i ruby-json_schemer -i go-gojsonschema -i python-jsonschema -i python-fastjsonschema -i rust-jsonschema -i lua-jsonschema -V << 'EOF' | bowtie summary --show validation
 {"description": "unevaluated across dialects", "schema": {"$schema": "https://json-schema.org/draft/2020-12/schema", "$ref": "urn:example:some-schema", "unevaluatedProperties": false}, "registry": {"urn:example:some-schema": {"$schema": "http://json-schema.org/draft-07/schema#", "properties": {"foo": true}}}, "tests": [{"description": "the instance", "instance": {}}] }
EOF
2023-03-29 10:31.01 [info     ] Will speak                     dialect=https://json-schema.org/draft/2020-12/schema
2023-03-29 10:31.02 [warning  ] Unsupported dialect, skipping implementation. [ghcr.io/bowtie-json-schema/cpp-valijson] dialect=https://json-schema.org/draft/2020-12/schema
2023-03-29 10:31.02 [warning  ] Unsupported dialect, skipping implementation. [ghcr.io/bowtie-json-schema/go-gojsonschema] dialect=https://json-schema.org/draft/2020-12/schema
2023-03-29 10:31.02 [warning  ] Unsupported dialect, skipping implementation. [ghcr.io/bowtie-json-schema/lua-jsonschema] dialect=https://json-schema.org/draft/2020-12/schema
2023-03-29 10:31.02 [warning  ] Unsupported dialect, skipping implementation. [ghcr.io/bowtie-json-schema/python-fastjsonschema] dialect=https://json-schema.org/draft/2020-12/schema
2023-03-29 10:31.03 [warning  ] Unsupported dialect, skipping implementation. [ghcr.io/bowtie-json-schema/ruby-json_schemer] dialect=https://json-schema.org/draft/2020-12/schema
2023-03-29 10:31.03 [warning  ] Unsupported dialect, skipping implementation. [ghcr.io/bowtie-json-schema/clojure-json-schema] dialect=https://json-schema.org/draft/2020-12/schema
2023-03-29 10:31.03 [error    ]                                [ghcr.io/bowtie-json-schema/js-ajv] case=unevaluated across dialects message=Error: no schema with key or ref "http://json-schema.org/draft-07/schema#" schema={'$schema': 'https://json-schema.org/draft/2020-12/schema', '$ref': 'urn:example:some-schema', 'unevaluatedProperties': False} seq=1
2023-03-29 10:31.04 [error    ]                                [ghcr.io/bowtie-json-schema/rust-boon] case=unevaluated across dialects message=unsupported scheme in urn:example:some-schema schema={'$schema': 'https://json-schema.org/draft/2020-12/schema', '$ref': 'urn:example:some-schema', 'unevaluatedProperties': False} seq=1
2023-03-29 10:31.04 [info     ] Finished                       count=1
                                                                                                           Bowtie                                                                                                            
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Schema                                                      ┃                                                                                                                                                             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ {                                                           │                                                                                                                                                             │
│  "$schema": "https://json-schema.org/draft/2020-12/schema", │              JsonSchema.Net                       ajv               hyperjump-jsv      jsonschema                      jsonschema         vscode-json-la…   │
│  "$ref": "urn:example:some-schema",                         │   Instance   (dotnet)           jsonschema (go)   (javascript)      (javascript)       (python)          boon (rust)   (rust)             (typescript)      │
│  "unevaluatedProperties": false                             │  ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────  │
│ }                                                           │   {}         valid              valid             error             valid              valid             error         valid             invalid            │
│                                                             │                                                                                                                                                             │
└─────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[jdesrosiers]

I also know (I think?) that Hyperjump doesn't use the annotations process fully for this purpose

This works in my implementation precisely because I don't use annotations to implement this behavior. It doesn't matter that draft-07 properties doesn't produce annotations because I use a mechanism other than annotations to determine which properties are evaluated.

I expect that implementations that do use annotations for implementing this behavior are using the same properties implementation for draft-07 as they are in 2020-12. That means the keyword produces the 2020-12 annotations even if used in a draft-07 schema. While that's probably not technically correct, I expect that's what's happening in most of these implementations and why we are seeing consistent "valid" results.

[gregsdennis]

Conversely, this validates in my implementation because I always use annotations. Annotations as a form of output weren't really a think at draft 7. While there's not really a requirement to collect them in the post-7 way, there's certainly no restriction not to.

My implementation is driven by annotations (partly as an experiment to verify the mechanics of the spec, but also because I'd have to track those kinds of things under the hood anyway). If I don't collect annotations, my implementation doesn't work.

[Julian]

I've thought about this sort of case recently while writing referencing tests. I don't personally put a lot of weight in what implementations do relative to what the spec says they should do, so since I don't think this sort of case is well-defined in the spec, nor likely to have been explicitly considered, I think to me the answer should be "it's undefined behavior in the current spec".

Morally, if I think about what I believe we should write in a future spec, I think the answer should be the instance you showed should be invalid with foo having been unevaluated. But the "more difficult" moral question to me would be referencing a draft 2019 schema where unevaluatedProperties is defined, and ask the same question -- and there I still think the answer should be "unevaluated" / "invalid" -- but clearly there are multiple possible opinions (invalid, invalid in your case but valid in the 2019 case I mentioned, and valid in all cases).

[Relequestual]

I'd say this is an emergent situation I hadn't considered.

[jdesrosiers]

That's an interesting question that I don't think I've ever considered. The "unevaluated" keywords could have been defined in terms of their behavior only and not coupled to annotations. Using annotations to implement these keywords is an implementation detail, it's not an intrinsic part of the behavior of these keywords.

If the spec did describe the behavior independent of annotations, I would say that properties declared in a draft-07 schemas should apply. However, since the spec does couple this behavior to annotations, I would say that the technically correct behavior is that properties declared in a draft-07 schema should not be counted as evaluated properties because a draft-07 properties keyword doesn't produce the annotations that the spec requires for this functionality.

Assuming that analysis is correct, the technically correct behavior isn't very helpful or intuitive. I think people would expect that the draft-07 properties would be counted. I think the solution is to redefine the "unevaluated" keywords to decouple them from annotations. That should make their behavior compatible with any dialect including custom keywords that might define properties (Example: requiredProperties).

However, that doesn't provide a clear answer to how 2019-09 and 2020-12 should be interpreted. I weigh intention very highly, sometimes above what the spec technically says. I don't think there was intention to exclude other keywords that declare properties, whether it's properties from older dialects or custom keywords. I think that exclusion is an accident of the way the behavior was defined. Therefore, I'd say that implementations should count properties declared in other dialects as evaluated, but it wouldn't be technically incorrect not to.

[gregsdennis]

I agree that we need to remove that language and let implementation decide how to do it. I'm going to stick with the annotations in my implementation, though, because I'd be tracking the data internally anyway. It just makes sense to use annotations as the mechanism to do it.

I would say that the technically correct behavior is that properties declared in a draft-07 schema should not be counted as evaluated properties because a draft-07 properties keyword doesn't produce the annotations that the spec requires for this functionality.

While I see where you're coming from, the spec also doesn't forbid annotations because they weren't a thing yet. With knowledge of the later specs, it could be inferred that draft-7-properties should also produce annotations.

That's probably a very biased interpretation, though.

Therefore, I'd say that implementations should count properties declared in other dialects as evaluated, but it wouldn't be technically incorrect not to.

So what you're saying is, "this should probably be valid, but we can't put a test on it."

[jdesrosiers]

So what you're saying is, "this should probably be valid, but we can't put a test on it."

Right. My argument for that conclusion was a bit flimsy, so I'm glad you came up with a stronger one.