Support Draft 2020-12 and refactor schema retrieval (#931)

* Fix ref * Refactor * Refactor * Refactor * Refactor * Fix test * Refactor * Add test * Ensure correct meta schema * Support dynamic ref * Fix anchor * Refactor * Update docs * Refactor * Fix * Refactor * Update test suite * Refactor * Fix * Support custom meta schema * Refactor * Support format assertion configuration * Refactor * Refactor * Refactor * Update docs * Allow yaml to be optional * Support vocabularies * Refactor * Refactor * Refactor * Remove deprecations * Update uri schema loader * Refactor * Refactor * Add get instance customizer * Add content validation * Update compatibility doc * Refactor enumObject is failing * Fix * Refactor * Refactor package * Refactor config * validation context set in constructor * Add javadoc * Only normalize standard json schema dialects * Refactor * update comment * Fix dynamic ref circular dependency * Update docs and readme * Update example * Fix 877 * Fix rebase * Add tests * Shift package * Add convenience methods for validation * Refactor * Update docs * Update docs * Update docs * Refactor schema loaders * Update format routines and tests and fixes * Update doc * Update doc * Fix spacing
networknt · Jan 26, 2024 · 7f1ec11 · 7f1ec11
1 parent f3825f3
commit 7f1ec11
Show file tree

Hide file tree

Showing 167 changed files with 4,890 additions and 2,921 deletions.
diff --git a/README.md b/README.md
@@ -12,7 +12,19 @@
 [![Javadocs](http://www.javadoc.io/badge/com.networknt/json-schema-validator.svg)](https://www.javadoc.io/doc/com.networknt/json-schema-validator)
 
 
-This is a Java implementation of the [JSON Schema Core Draft v4, v6, v7, v2019-09 and v2020-12(partial)](http://json-schema.org/latest/json-schema-core.html) specification for JSON schema validation. In addition, it also works for OpenAPI 3.0 request/response validation with some [configuration flags](doc/config.md). For users who want to collect information from a JSON node based on the schema, the [walkers](doc/walkers.md) can help. The default JSON parser is the [Jackson](https://github.com/FasterXML/jackson) that is the most popular one. As it is a key component in our [light-4j](https://github.com/networknt/light-4j) microservices framework to validate request/response against OpenAPI specification for [light-rest-4j](http://www.networknt.com/style/light-rest-4j/) and RPC schema for [light-hybrid-4j](http://www.networknt.com/style/light-hybrid-4j/) at runtime, performance is the most important aspect in the design. 
+This is a Java implementation of the [JSON Schema Core Draft v4, v6, v7, v2019-09 and v2020-12](http://json-schema.org/latest/json-schema-core.html) specification for JSON schema validation.
+
+In addition, it also works for OpenAPI 3.0 request/response validation with some [configuration flags](doc/config.md). For users who want to collect information from a JSON node based on the schema, the [walkers](doc/walkers.md) can help. The default JSON parser is the [Jackson](https://github.com/FasterXML/jackson) that is the most popular one. As it is a key component in our [light-4j](https://github.com/networknt/light-4j) microservices framework to validate request/response against OpenAPI specification for [light-rest-4j](http://www.networknt.com/style/light-rest-4j/) and RPC schema for [light-hybrid-4j](http://www.networknt.com/style/light-hybrid-4j/) at runtime, performance is the most important aspect in the design.
+
+## JSON Schema Draft Specification Compatibility
+
+Information on the compatibility support for each version, including known issues, can be found in the [Compatibility with JSON Schema versions](doc/compatibility.md) document.
+
+## Upgrading to new versions
+
+Information on notable or breaking changes when upgrading the library can be found in the [Upgrading to new versions](doc/upgrading.md) document. This library can contain breaking changes in minor version releases.
+
+For the latest version, please check the [Releases](https://github.com/networknt/json-schema-validator/releases) page.
 
 ## Why this library
 
@@ -42,23 +54,67 @@ The OpenAPI 3.0 specification is using JSON schema to validate the request/respo
 
 Following the design principle of the Light Platform, this library has minimum dependencies to ensure there are no dependency conflicts when using it. 
 
-Here are the dependencies:
+The following are the dependencies that will automatically be included when this library is included.
 
 ```xml
 <dependency>
+    <!-- Used for logging -->
+    <groupId>org.slf4j</groupId>
+    <artifactId>slf4j-api</artifactId>
+    <version>${version.slf4j}</version>
+</dependency>
+
+<dependency>
+    <!-- Used to process JSON -->
     <groupId>com.fasterxml.jackson.core</groupId>
     <artifactId>jackson-databind</artifactId>
     <version>${version.jackson}</version>
 </dependency>
 
 <dependency>
-    <groupId>org.slf4j</groupId>
-    <artifactId>slf4j-api</artifactId>
-    <version>${version.slf4j}</version>
+    <!-- Used to process YAML -->
+    <groupId>com.fasterxml.jackson.dataformat</groupId>
+    <artifactId>jackson-dataformat-yaml</artifactId>
+    <version>${version.jackson}</version>
+</dependency>
+
+<dependency>
+    <!-- Used to validate RFC 3339 date and date-time -->
+    <groupId>com.ethlo.time</groupId>
+    <artifactId>itu</artifactId>
+    <version>${version.itu}</version>
 </dependency>
 ```
 
-**Note**: Up to version [1.0.81](https://github.com/networknt/json-schema-validator/blob/1.0.81/pom.xml#L99), the dependency `org.apache.commons:commons-lang3` was included as a runtime dependency. Starting with [1.0.82](https://github.com/networknt/json-schema-validator/releases/tag/1.0.82) it is not required anymore.
+The following are the optional dependencies that may be required for certain options.
+
+These are not automatically included and setting the relevant option without adding the library will result in a `ClassNotFoundException`.
+
+```xml
+<!-- This is required when setting setEcma262Validator(true)  -->
+<dependency>
+    <!-- Used to validate ECMA 262 regular expressions -->
+    <groupId>org.jruby.joni</groupId>
+    <artifactId>joni</artifactId>
+    <version>${version.joni}</version>
+    <optional>true</optional>
+</dependency>
+```
+
+The YAML dependency can be excluded if this is not required. Attempting to process schemas or input that are YAML will result in a `ClassNotFoundException`.
+
+```xml
+<dependency>
+    <groupId>com.networknt</groupId>
+    <artifactId>json-schema-validator</artifactId>
+    <exclusions>
+        <exclusion>
+            <groupId>com.fasterxml.jackson.dataformat</groupId>
+            <artifactId>jackson-dataformat-yaml</artifactId>
+        </exclusion>
+    </exclusions>
+</dependency>
+```
 
 #### Community
 
@@ -68,37 +124,110 @@ This library is very active with a lot of contributors. New features and bug fix
 
 The library supports Java 8 and up. If you want to build from the source code, you need to install JDK 8 locally. To support multiple version of JDK, you can use [SDKMAN](https://www.networknt.com/tool/sdk/)
 
-## Dependency
+## Usage
+
+### Adding the dependency
 
 This package is available on Maven central. 
 
-Maven: 
+#### Maven: 
 
 ```xml
 <dependency>
     <groupId>com.networknt</groupId>
     <artifactId>json-schema-validator</artifactId>
-    <version>1.0.87</version>
-
-    <!-- Only required for versions < 1.0.82. See README.md -->
-    <exclusions>
-        <exclusion>
-            <groupId>org.apache.commons</groupId>
-            <artifactId>commons-lang3</artifactId>
-        </exclusion>
-    </exclusions>
+    <version>1.2.0</version>
 </dependency>
 ```
 
-Gradle:
+#### Gradle:
 
 ```java
 dependencies {
-    implementation(group: 'com.networknt', name: 'json-schema-validator', version: '1.0.87');
+    implementation(group: 'com.networknt', name: 'json-schema-validator', version: '1.2.0');
 }
 ```
 
-For the latest version, please check the [release](https://github.com/networknt/json-schema-validator/releases) page. 
+### Validating inputs against a schema
+
+The following example demonstrates how inputs is validated against a schema. It comprises the following steps.
+
+* Creating a schema factory with the default schema dialect and how the schemas can be retrieved. 
+  * Configuring mapping the `$id` to a retrieval URI using `schemaMappers`.
+  * Configuring how the schemas are loaded using the retrieval URI using `schemaLoaders`.
+    For instance a `Map<String, String> schemas` containing a mapping of retrieval URI to schema data as a `String` can by configured using `builder.schemaLoaders(schemaLoaders -> schemaLoaders.schemas(schemas))`. This also accepts a `Function<String, String> schemaRetrievalFunction`.
+* Creating a configuration for controlling validator behavior.
+* Loading a schema from a schema location along with the validator configuration.
+* Using the schema to validate the data along with setting any execution specific configuration like for instance the locale or whether format assertions are enabled.
+
+```java
+// This creates a schema factory that will use Draft 2012-12 as the default if $schema is not specified in the schema data. If $schema is specified in the schema data then that schema dialect will be used instead and this version is ignored.
+JsonSchemaFactory jsonSchemaFactory = JsonSchemaFactory.getInstance(VersionFlag.V202012, builder -> 
+    // This creates a mapping from $id which starts with https://www.example.org/ to the retrieval URI classpath:schema/
+    builder.schemaMappers(schemaMappers -> schemaMappers.mapPrefix("https://www.example.org/", "classpath:schema/"))
+);
+
+SchemaValidatorsConfig config = new SchemaValidatorsConfig();
+// By default JSON Path is used for reporting the instance location and evaluation path
+config.setPathType(PathType.JSON_POINTER);
+// By default the JDK regular expression implementation which is not ECMA 262 compliant is used
+// Note that setting this to true requires including the optional joni dependency
+// config.setEcma262Validator(true);
+
+// Due to the mapping the schema will be retrieved from the classpath at classpath:schema/example-main.json. If the schema data does not specify an $id the absolute IRI of the schema location will be used as the $id.
+JsonSchema schema = jsonSchemaFactory.getSchema(SchemaLocation.of("https://www.example.org/example-main.json"), config);
+String input = "{\r\n"
+    + "  \"main\": {\r\n"
+    + "    \"common\": {\r\n"
+    + "      \"field\": \"invalidfield\"\r\n"
+    + "    }\r\n"
+    + "  }\r\n"
+    + "}";
+
+Set<ValidationMessage> assertions = schema.validate(input, InputFormat.JSON, executionContext -> {
+    // By default since Draft 2019-09 the format keyword only generates annotations and not assertions
+    executionContext.getConfig().setFormatAssertionsEnabled(true);
+});
+```
+
+### Validating a schema against a meta schema
+
+The following example demonstrates how a schema is validated against a meta schema.
+
+This is actually the same as validating inputs against a schema except in this case the input is the schema and the schema used is the meta schema.
+
+```java
+JsonSchemaFactory jsonSchemaFactory = JsonSchemaFactory.getInstance(VersionFlag.V202012, builder -> 
+    // This creates a mapping to load the meta schema from the library classpath instead of remotely
+    // This is better for performance and the remote may choose not to service the request
+    // For instance Cloudflare will block requests that have older Java User-Agent strings eg. Java/1.
+    builder.schemaMappers(schemaMappers -> 
+        schemaMappers.mapPrefix("https://json-schema.org", "classpath:").mapPrefix("http://json-schema.org", "classpath:"))
+);
+
+SchemaValidatorsConfig config = new SchemaValidatorsConfig();
+// By default JSON Path is used for reporting the instance location and evaluation path
+config.setPathType(PathType.JSON_POINTER);
+// By default the JDK regular expression implementation which is not ECMA 262 compliant is used
+// Note that setting this to true requires including the optional joni dependency
+// config.setEcma262Validator(true);
+
+// Due to the mapping the meta schema will be retrieved from the classpath at classpath:draft/2020-12/schema.
+JsonSchema schema = jsonSchemaFactory.getSchema(SchemaLocation.of(SchemaId.V202012), config);
+String input = "{  \n"
+    + "  \"type\": \"object\",  \n"
+    + "  \"properties\": {    \n"
+    + "    \"key\": { \n"
+    + "      \"title\" : \"My key\", \n"
+    + "      \"type\": \"invalidtype\" \n"
+    + "    } \n"
+    + "  }\n"
+    + "}";
+Set<ValidationMessage> assertions = schema.validate(input, InputFormat.JSON, executionContext -> {
+    // By default since Draft 2019-09 the format keyword only generates annotations and not assertions
+    executionContext.getConfig().setFormatAssertionsEnabled(true);
+});
+```        
 
 ## [Quick Start](doc/quickstart.md)
 
@@ -110,9 +239,7 @@ For the latest version, please check the [release](https://github.com/networknt/
 
 ## [YAML Validation](doc/yaml.md)
 
-## [Schema Mapping](doc/schema-map.md)
-
-## [Customized URIFetcher](doc/cust-fetcher.md)
+## [Customizing Schema Retrieval](doc/schema-retrieval.md)
 
 ## [Customized MetaSchema](doc/cust-meta.md)
 
@@ -130,15 +257,6 @@ For the latest version, please check the [release](https://github.com/networknt/
 
 ## [Validating RFC 3339 durations](doc/duration.md)
 
-
-## Known issues
-
-I have just updated the test suites from the [official website](https://github.com/json-schema-org/JSON-Schema-Test-Suite) as the old ones were copied from another Java validator. Now there are several issues that need to be addressed. All of them are edge cases, in my opinion, but need to be investigated. As my old test suites were inherited from another Java JSON Schema Validator, I guess other Java Validator would have the same issues as these issues are in the Java language itself.
-
-[#7](https://github.com/networknt/json-schema-validator/issues/7)
-
-[#5](https://github.com/networknt/json-schema-validator/issues/5)
-
 ## Projects
 
 The [light-rest-4j](https://github.com/networknt/light-rest-4j), [light-graphql-4j](https://github.com/networknt/light-graphql-4j) and [light-hybrid-4j](https://github.com/networknt/light-hybrid-4j) use this library to validate the request and response based on the specifications. If you are using other frameworks like Spring Boot, you can use the [OpenApiValidator](https://github.com/mservicetech/openapi-schema-validation), a generic OpenAPI 3.0 validator based on the OpenAPI 3.0 specification. 

diff --git a/doc/compatibility.md b/doc/compatibility.md
@@ -1,3 +1,13 @@
+## Compatibility with JSON Schema versions
+
+This implementation does not currently generate annotations.
+
+The `pattern` validator by default uses the JDK regular expression implementation which is not ECMA-262 compliant and is thus not compliant with the JSON Schema specification. The library can however be configured to use a ECMA-262 compliant regular expression implementation.
+
+### Known Issues
+* The `anyOf` applicator currently returns immediately on matching a schema. This results in the `unevaluatedItems` and `unevaluatedProperties` keywords potentially returning an incorrect result as the rest of the schemas in the `anyOf` aren't processed.
+* The `unevaluatedItems` keyword does not currently consider `contains`.
+
 
 ### Legend
 
@@ -8,27 +18,27 @@
 |   🔴   | Not implemented       |
 |   🚫   | Not defined           |
 
-### Compatibility with JSON Schema versions
+### Keywords Support
 
 | Keyword                    | Draft 4 | Draft 6 | Draft 7 | Draft 2019-09 | Draft 2020-12 |
 |:---------------------------|:-------:|:-------:|:-------:|:-------------:|:-------------:|
-| $anchor                    | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
-| $dynamicAnchor             | 🚫 | 🚫 | 🚫 | 🚫 | 🔴 |
-| $dynamicRef                | 🚫 | 🚫 | 🚫 | 🚫 | 🔴 |
-| $id                        | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 |
+| $anchor                    | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
+| $dynamicAnchor             | 🚫 | 🚫 | 🚫 | 🚫 | 🟢 |
+| $dynamicRef                | 🚫 | 🚫 | 🚫 | 🚫 | 🟢 |
+| $id                        | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
 | $recursiveAnchor           | 🚫 | 🚫 | 🚫 | 🟢 | 🚫 |
 | $recursiveRef              | 🚫 | 🚫 | 🚫 | 🟢 | 🚫 |
-| $ref                       | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 |
-| $vocabulary                | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
+| $ref                       | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
+| $vocabulary                | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
 | additionalItems            | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
 | additionalProperties       | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
 | allOf                      | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
 | anyOf                      | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
 | const                      | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
 | contains                   | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
-| contentEncoding            | 🚫 | 🚫 | 🔴 | 🔴 | 🔴 |
-| contentMediaType           | 🚫 | 🚫 | 🔴 | 🔴 | 🔴 |
-| contentSchema              | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
+| contentEncoding            | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
+| contentMediaType           | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
+| contentSchema              | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
 | definitions                | 🟢 | 🟢 | 🟢 | 🚫 | 🚫 |
 | defs                       | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
 | dependencies               | 🟢 | 🟢 | 🟢 | 🚫 | 🚫 |
@@ -67,7 +77,49 @@
 | uniqueItems                | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
 | writeOnly                  | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
 
-### Semantic Validation (Format)
+#### Content Encoding
+
+Since Draft 2019-09, the `contentEncoding` keyword does not generate assertions. As the implementation currently does not collect annotations this only generates assertions in Draft 7.
+
+#### Content Media Type
+
+Since Draft 2019-09, the `contentMediaType` keyword does not generate assertions. As the implementation currently does not collect annotations this only generates assertions in Draft 7.
+
+#### Content Schema
+
+The `contentSchema` keyword does not generate assertions. As the implementation currently does not collect annotations this doesn't do anything.
+
+#### Pattern
+
+By default the `pattern` keyword uses the JDK regular expression implementation validating regular expressions. 
+
+This is not ECMA-262 compliant and is thus not compliant with the JSON Schema specification. This is however the more likely desired behavior as other logic will most likely be using the default JDK regular expression implementation to perform downstream processing.
+
+The library can be configured to use a ECMA-262 compliant regular expression validator which is implemented using [joni](https://github.com/jruby/joni). This can be configured by setting `setEcma262Validator` to `true`.
+
+This also requires adding the `joni` dependency.
+
+```xml
+<dependency>
+    <!-- Used to validate ECMA 262 regular expressions -->
+    <groupId>org.jruby.joni</groupId>
+    <artifactId>joni</artifactId>
+    <version>${version.joni}</version>
+</dependency>
+```
+
+### Format
+
+Since Draft 2019-09 the `format` keyword only generates annotations by default and does not generate assertions.
+
+This can be configured on a schema basis by using a meta schema with the appropriate vocabulary.
+
+| Version               | Vocabulary                                                    | Value             |
+|:----------------------|---------------------------------------------------------------|-------------------|
+| Draft 2019-09         | `https://json-schema.org/draft/2019-09/vocab/format`          | `true`            |
+| Draft 2020-12         | `https://json-schema.org/draft/2020-12/vocab/format-assertion`| `true`/`false`    | 
+
+This behavior can be overridden to generate assertions on a per-execution basis by setting the `setFormatAssertionsEnabled` to `true`.
 
 | Format                | Draft 4 | Draft 6 | Draft 7 | Draft 2019-09 | Draft 2020-12 |
 |:----------------------|:-------:|:-------:|:-------:|:-------------:|:-------------:|

diff --git a/doc/config.md b/doc/config.md
@@ -45,14 +45,6 @@ The default value is true in the SchemaValidatorsConfig object.
 
 For more details, please refer to this [issue](https://github.com/networknt/json-schema-validator/issues/183). 
 
-
-* uriMappings
-
-Map of public, typically internet-accessible schema URLs to alternate locations; this allows for offline validation of schemas that refer to public URLs. This is merged with any mappings the sonSchemaFactory 
-may have been built.
-
-The type for this variable is `Map<String, String>`. 
-
 * javaSemantics
 
 When set to true, use Java-specific semantics rather than native JavaScript semantics.