Tekton Type system scope and design doc

[xihaLong]

Tekton Types Design Doc (starter)

Type system and any methods of extensibility to consider on top of the Types (including as-plugin or as an external project entirely). There are already JSON Schema validators. Can Tekton let you provide your own validator, or should Tekton ignore validations? This has somewhat of an impact on what happens in Arrays, Templates, and reuse of Pipelines/Resources/Tasks

Tracking discussion and followup on Types for Tekton.
Is there anything limiting reuse if Types are arbitrary strings that simply need to match? How would our intended Duck Typing of resources function? Duck Typing of parameters?

Specifically interesting is the question of type validation. Not interested to re-implement any existing language for type validation. Need to leave the doors open to a 3rd party providing a rich system of type validations for Pipeline Distributors and 3rd-party product creators that rely on Tekton under-the-hood.

Meeting notes (copied on 2019-07-10 at 17:08 GMT) :
"Would we want Task authors to be able to add further validation? E.g. the string must be X chars or less, must be at least 50
Would want ppl to be able to use Tasks right out of the box, ideally should contain the validation needed?
Some ideas:
Can look at the type system that Puppet (Lyra) is using
People will make types as complex as they possibly can!
Would be cool to make a proof of concept that uses this
pcore library from lyra: https://github.com/lyraproj/pcore
example workflow with types: https://github.com/ahpook/lyra-local/blob/master/workflows/azure_wordpress.yaml
JSON schema?
Don’t want to create a new validation language!
Could also look into OPA language rego"
https://www.openpolicyagent.org/
https://www.openpolicyagent.org/docs/latest/how-do-i-write-policies/

"Simon has tackled this problem a few times as well, and Eric can share the puppet approach"

Intention is to list out a few alternative options in a design doc for review at next week's meeting (7/17), along with recommended choice
@ Eric Sorenson and @ Simon Kaegi

[xihaLong]

@ahpook @skaegi

[xihaLong]

reading https://puppet.com/blog/lyra-brings-puppet-concepts-and-more-cloud-native

[xihaLong]

reading https://blog.openpolicyagent.org/securing-the-kubernetes-api-with-open-policy-agent-ce93af0552c3

[xihaLong]

filed #1078 and #1079

[xihaLong]

created https://docs.google.com/document/d/1pH9wyVaGVLjnhi5tOa2TSq7d1j5HTTeCbCmosRo4jWE document on OPA

[xihaLong]

asked about Types again at the weekly working group meeting

PipelineResources may turn into updates to the TaskRun/PipelineRun (hanging the results from the run directly), as one potential future direction. Related conversation in #544

[thallgren]

I've done some work on a more go-like type system "dgo" (dynamic go) where the type syntax is designed to be close to the go syntax. If something like pcore was of interest, then I think this can be even more so: https://github.com/lyraproj/dgo

[dibyom]

Can this issue be closed in favor of #1393? @bobcatfish @skaegi ?

[skaegi]

At least for built-in type validation in Tekton I'd suggest that we do not want to stray far from the platform and follow what api machinery is suggesting -- e.g. open api v3 / json schema / jsonpath -- and essentially follow the versions and support what they spec.

This does not prevent other forms of validation, it just means at least for now it must be layered above Tekton. For example OPA is handled at a higher layer for us. We might eventually support pluggable custom validators but I think we could still do that in the context of JSON schema.

@dibyom let's finalize our thoughts for beta in #1393 and then if we're happy I can open a separate issue that breaks out the schema and jsonpath support and we could continue the typing discussion there. wdyt?

[dibyom]

Sounds good to me @skaegi
I'll close out this issue and we can continue the discussion in #1393
/close

[tekton-robot]

@dibyom: Closing this issue.

In response to this:

Sounds good to me @skaegi
I'll close out this issue and we can continue the discussion in #1393
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.