Systematic configuration of Flyte workflows

[cameronraysmith]

Following up on tangential discussions in

#3397 (comment)
flyteorg/flytekit#2299

It might be helpful to consider adding documentation and/or plugins that support a systematic approach to configuring Flyte workflows.

The above discussion/PR focus on the related set of

• omegaconf (again see Add flytekit-omegaconf plugin flytekit#2299),
• hydra, and
• hydra-zen tools,

but there may certainly be others worth discussing as well.

[cameronraysmith]

From flyteorg/flytekit#2299 (comment)

I think it's a good idea to support omegaconf's dictconfig directly because it is a foundational layer, but would just note that you get something like this almost for free if you use hydra-zen because it already provides a bridge from omegaconf's types to dataclasses.

Do you have a minimal working example that shows how a DictConfig can be passed to a Flyte task using hydra-zen? 🙏

Unfortunately I do not yet have an example I would consider minimal and the examples I do have do not intentionally construct DictConfig objects. Rather, they use hydra-zen along with a helper to construct serializable Dataclasses from function interfaces to configure workflows.

I have not managed to crystallize my experience into a flyte(kit) PR precisely because it is not yet clear to me what is minimal with respect to what is already included in Flyte's support for utilizing serializable Dataclasses in task and workflow interfaces. As such one question I have had is whether one would need to add anything to the flytekit source at all, or instead, simply document how this can be done without any modification to flytekit.

A relatively simple, but not yet minimal and now slightly outdated, example emanates from
https://github.com/sciexp/flytezen/blob/v1.0.3/src/flytezen/workflows/lrwine.py

A more complicated example where the same approach is employed is

Warning

this workflow currently makes incorrect use of dynamic workflows but I link it here because it separately illustrates how to configure workflows with hydra-zen and serializable Dataclasses

• workflow: https://github.com/pinellolab/pyrovelocity/blob/v0.2.0/src/pyrovelocity/workflows/main_workflow.py
• configuration: https://github.com/pinellolab/pyrovelocity/blob/v0.2.0/src/pyrovelocity/workflows/main_configuration.py
  • construction of Dataclass interfaces for functions wrapped by Flyte tasks
  • helper to create a Dataclass from a callable

In this case there is no user-modified yaml. With hydra-zen, this is taken care of by function/method default parameters and associated default values for Dataclass fields. When a workflow is executed, potentially with overrides from the command-line interface, hydra will issue a yaml receipt, as it usually does, containing all that would be required to configure a future execution in an equivalent manner including any parameter values that were overridden when issuing the relevant execution command.

I think to work toward a minimal approach, it would be best to either eliminate or better codify and test usage of something like the helper to create a serializable Dataclass from a callable that is used in constructing serializable Dataclass interfaces for functions that are wrapped by workflow tasks. Simplification in favor of the desirable elimination of this helper altogether might become clear if sciexp/flytezen#5 is resolved in such a manner that the Dataclasses already constructed by hydra-zen can be made serializable and avoid these intermediate helper steps that are redundant with hydra-zen's functionality modulo the need to have all the Dataclasses it constructs inherit from one of the various serializable forms (i.e. mashumaro's DataClassJSONMixin or equivalent).