The CLI jar falsely reports true RAML conformance with some invalid files

[dcrossleyau]

• I'm submitting a ...

  • [*] bug report
  • feature request
  • question
  • other

• What is the current behavior?
  The CLI jar incorrectly reports a broken RAML file as "conforms=true"

• What is the expected behavior?
  Should say "conforms=false"

• Please tell us about your environment:

  • AMF Version: amf-4.4.0-SNAPSHOT
  • AMF Distribution: Java CLI
  • Operating System: macOS

• Other information

I have found a number of situations with validation of RAML 1.0 documents, which does properly report "conforms=false" when using "webapi-parser" and "amf-client-js" and "raml-js-parser-2". However the "AMF CLI" java tool falsely reports these as "conforms=true".

I have packaged a minimal example as test-pets.tar.gz

Using these tools the file is properly reported as invalid:
raml-js-parser-2 1.1.67
amf-client-js 4.3.0
webapi-parser 0.5.0

However, with today's CLI jar (built with Java 11 and sbt-1.3.13):

java -jar amf-4.4.0-SNAPSHOT.jar validate \
  pets/invalid-included-rtype-broken-key.raml \
  --format-in "RAML 1.0" -mime-in "application/raml+yaml" -p "RAML"

reports "conforms=true" ... but should not be so.

Here are the main files that are included in the archive.
Note the deliberate error at the second-last line, where petsSchemaBrokenKey should instead be petsSchema

invalid-included-rtype-broken-key.raml

#%RAML 1.0

title: invalid, included resourceType, broken rtype key

types:
  petsSchema: |
    {
      "$schema": "http://json-schema.org/draft-04/schema#",
      "id": "petsSchema",
      "type": "object",
      "properties": {
        "pets": {
          "id": "petsList",
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "genus": {
                "type": "string"
              },
              "quantity": {
                "type": "integer"
              }
            },
            "required": [
              "genus",
              "quantity"
            ]
          }
        },
        "totalRecords": {
          "type": "integer"
        }
      },
      "required": [
        "pets",
        "totalRecords"
      ]
    }

resourceTypes:
  collection: !include raml-shared/rtypes/collection.raml

/pets:
  type:
    collection:
      rtypeCollection: petsSchemaBrokenKey
      exampleCollection: !include examples/valid-pets.json

raml-shared/rtypes/collection.raml

#%RAML 1.0 ResourceType
  
description: Collection of <<resourcePathName>> in stock.
get:
  description: Get a list of <<resourcePathName>>.
  responses:
    200:
      body:
        application/json:
          type: <<rtypeCollection>>
          example: <<exampleCollection>>

[BlockLucas]

Hi @dcrossleyau

I've created an internal ticket to check this case: https://www.mulesoft.org/jira/browse/APIMF-2536

We track down the behavior change to 4.0.4, maybe is still useful for you to use AMF 4.0.3 in the meantime.

See 4.0.3 output:

{
  "@type": "http://www.w3.org/ns/shacl#ValidationReport",
  "http://www.w3.org/ns/shacl#conforms": false,
  "http://www.w3.org/ns/shacl#result": [
    {
      "@type": "http://www.w3.org/ns/shacl#ValidationResult",
      "http://www.w3.org/ns/shacl#resultSeverity": {
        "@id": "http://www.w3.org/ns/shacl#Violation"
      },
      "http://www.w3.org/ns/shacl#focusNode": {
        "@id": "file:///Users/lblock/Downloads/test-pets/invalid-included-rtype-broken-key.raml#/web-api/end-points/%2Fpets/get/200/application%2Fjson/any/schema/unresolved"
      },
      "http://www.w3.org/ns/shacl#resultMessage": "Unresolved reference 'petsSchemaBrokenKey'",
      "http://www.w3.org/ns/shacl#sourceShape": {
        "@id": "http://a.ml/vocabularies/amf/core#unresolved-reference"
      },
      "http://a.ml/vocabularies/amf/parser#lexicalPosition": {
        "@type": "http://a.ml/vocabularies/amf/parser#Position",
        "http://a.ml/vocabularies/amf/parser#start": {
          "@type": "http://a.ml/vocabularies/amf/parser#Location",
          "http://a.ml/vocabularies/amf/parser#line": 10,
          "http://a.ml/vocabularies/amf/parser#column": 16
        },
        "http://a.ml/vocabularies/amf/parser#end": {
          "@type": "http://a.ml/vocabularies/amf/parser#Location",
          "http://a.ml/vocabularies/amf/parser#line": 10,
          "http://a.ml/vocabularies/amf/parser#column": 35
        }
      }
    }
  ]
}

Regards!

[arielmirra]

Hi @dcrossleyau!
We fixed this wrong behavior of the CLI. The fix should be public and available in the next release of AMF, version 4.4.1. I'm closing this issue.
Thanks!