[PROPOSAL] Build a test configuration for workflow templates that allows them to be easily sanity tested

[dbwiddis]

What/Why

What are you proposing?

The OpenSearch API Specification repo includes a Test Suite that allows for easy YAML-based test writing. We should use that functionality to test our workflow substitution templates automatically.

What users have asked for this feature?

I wrote detailed samples here and it seems having a link in the API docs to one authoritative source is better than duplicating it?

The spec should be that source. The problem with the documentation as written is that there's no guarantee that it's correct, so our documentation contains a ton of bugs and typos. With running samples in this repo we will be able to export both the reference documentation (opensearch-project/documentation-website#7700) and working samples in 8 different programming languages that are verified to be correct and work.

Originally posted by @dblock in opensearch-project/opensearch-api-specification#440 (comment)

OK, to be fair, dB's comment was regarding a thorough test/demo of Search Pipeline processors, but it's a great idea to apply here. I've already suggested that we should have spec-based tests for our Workflow Step API implementations:

• [FEATURE] Add test(s) to compare workflow steps configuration to API specification #651

And there's an intent to add specs for plugins (although we got left off the list, 😢):

• [FEATURE] Add missing API specs opensearch-api-specification#168

What problems are you trying to solve?

A recurring developer pain point when new templates are being written is "sanity testing". Just this week I spent several hours manually running several new templates to make sure they produced expected results.

This may have worked for our initial release when the templates had fixed content and we could count the number of templates needed on two hands.

However, we now have at least 17 substitution templates which also have some required parameters.

And that number may grow.

Manual sanity testing is not scalable. According to this high tech calculator, the ~6 hours I spent for one month's worth of new templates to setup, test, debug, and report results, would warrant a 2-week sprint's worth of effort to automate.

What is the developer experience going to be?

Similar to the current requirements when updating an OpenSearch API:

• Write the code and any needed unit and integration tests (we are already doing that)
• Write a documentation PR for our public documentation (we are already doing that)
• Write an OpenAPI spec for the new addition (we just started doing that).

The tests are yaml-based and probably duplicate the sequence of steps that the developer has already written for unit testing. But it gains the benefit of an end-to-end "integration" style test.

Are there any security considerations?

Tests shouldn't use any stored credentials. The test setup linked already addresses this.

Are there any breaking changes to the API

The whole purpose of doing this is to detect/avoid breaking changes!

What is the user experience going to be?

Users will be able to consult the OpenSearch API Spec not only to get accurate API requirements, but also see example implementations, streamlining and possibly automating their own development!

The time you save may not just be your own.

Are there breaking changes to the User Experience?

This adds to the user experience.

Why should it be built? Any reason not to?

It may require a bit of effort but the benefits far outweigh the time investment.

What will it take to execute?

• Document the Flow Framework API in a spec file
• Document each substitution template in that spec file, with required parameters, similarly to how Search Pipeline processors are presently documented
• Write test files in YAML to exercise each template and validate a correct response (e.g. deployed resources)

Any remaining open questions?

1. Why haven't we done this already?
2. When can we get started?

[dbwiddis]

Closing in favor of #833