Evaluate styleguide compliance validation tooling

[pvdbosch]

In GitLab by @pvdbosch on May 17, 2019, 19:24

Provide tooling support to validate the compliance of OpenAPI to the styleguide, for checks that can be automated.

There was a PoC "styleguide-validationrules", but it was put on hold.
We're currently looking at using spectral.

Unsolved issues encountered when trying out the styleguide-validationrules:

• NPE on external $ref with relative path - workaround by prefixing with "./". Issue reported on the maven plugin.
• SemanticError at paths./health.get: All 4xx and 5xx should link to "#/definitions/Problem". For the health resource we may return 503 with non-problem body. This exception to the rule should be added.
• SemanticError at b2b-env.ksz-bcss.fgov.be:4523: Should be in lowercase - for validation of host value. Seems like a bug.
• SemanticError at $.info.version: The version number should be 'x.y'. A version number "1.0.0" fails, while it should be supported as patch version is optional

[pvdbosch]

In GitLab by @pvdbosch on May 17, 2019, 19:50

I also got some weird error on problem-v1.yaml:

[ERROR] DefinitionSemanticError at InvalidParamProblem.invalidParams.items: reference InvalidParam cannot be found

[ERROR] DefinitionSemanticError at InvalidParamProblem: reference Problem cannot be found

I can't find the cause of this. Error seems to disappear when removing the allOf construct in InvalidParamProblem, but allOf doesn't cause problems when used in the main swagger file.

[pvdbosch]

Investigating if we'd rewrite the validation rules using spectral as tool, which is supported in stoplight studio.

[pvdbosch]

I removed the styleguidevalidation-rules (+ maven plugin) from the guide.

[pvdbosch]

Someone at Smals is working on this.

There's also a tool by the developers of the openapi-generator:

• https://openapitools.github.io/openapi-style-validator/
• https://github.com/OpenAPITools/openapi-style-validator

Haven't checked yet how good it fits our needs.

[pvdbosch]

@wsalembi , Italy has worked out on online OpenAPI validation tool based on Spectral.
They reached out to us in belgif/rest-security#21 ; it may be interesting to see what they developed and if it's interesting to work together on some aspects.

Links:

• the checker: https://italia.github.io/api-oas-checker/
• rules: https://italia.github.io/api-oas-checker/spectral-full.doc.html
• GitHub repo: https://github.com/italia/api-oas-checker

[pvdbosch]

feedback from @wsalembi and colleagues:

• Spectral: hard to bypass default checks, some guidelines cannot be expressed in spectral (e.g. check consistency between multiple elements in the OpenAPI file), another flavor of regexp that doesn't allow to express all types of lookups.
• openapi-style-validator: hardcoded rules, not really extensible. Not mature enough.
• Now working on a Java-based validation tool that uses drools rules, which allows full flexibility.

[pvdbosch]

EC provides a validation framework:
https://joinup.ec.europa.eu/collection/interoperability-test-bed-repository/solution/interoperability-test-bed/detailed-information#capabilities

Doesn't support REST guide validation rules, but framework might be extensible enough to fit it in. To investigate; consider reaching out?

[pvdbosch]

Progress has been made on the Java-based validation tool that can be executed from a maven plugin. @wsalembi will share the tool and its code with the WG participants for early feedback. The tools allows for extension with custom rules or deactivation of built-in rules.

[pvdbosch]

first versions of validation tool are released, internally on git/artifactory gcloud. CBSS is interested to take a look at it, and perhaps contribute.

(there's also an openapi diff tool in development)

[pvdbosch]

@wsalembi will check if development of validation tool could be moved to GitHub (under Open Source license ASL 2.0). Binaries could be published then to maven central. Setup of github project can be done by CBSS participants.

@wsalembi will already share the binaries within the WG manually in the meantime.

[jpraet]

I also found these:

• https://github.com/zalando/zally
• https://github.com/daveshanley/vacuum

[pvdbosch]

open-sourcing of the Smals tool will be followed up in #168; closing this issue