Node Persistent Storage

[pryce-turner]

Motivation

Many workflows rely on large, static assets that have to be fetched from S3 for every pod that uses them. Having the ability to cache certain FlyteFiles or FlyteDirectories on node would be a massive speedup for certain tasks.

Proposed Implementation

A very simple initial implementation would be to provision a storage class, persistent volume and persistent volume claim for this purpose:

• The PV would be a local volume type and use any node affinity appropriate (e.g. cache=true label). It should also be given as much space as possible since unused space on a pvc remains unallocated on the underlying filesystem.
• The storage class would not use any provisioner, as dynamic provisioning is not supported for local PVs. It should use waitForFirstConsumer as the binding mode to delay binding until a pod is actually using it
• A PVC with the same space allocation should use the same storage class for binding the PV, the name would also be used in the PodTemplate

Once the above is setup, any task using a PodTemplate with the correct PVC name should be allowed to interact with the node cache:

• A default kubernetes.client.models.V1Volume could exist in flytekit with the default PVC name as a loose coupling between task and underlying infra
• A flytekit CacheFile class should exist which accepts a FlyteFile and handles it in a way that parallel executions of the same task do not clobber the cache
• subPaths can be used in the PodTemplate when mounting to separate out the cache for different parts of a workflow

Limitations

• no scheduling considerations; task pods will land randomly on nodes with available resources, whether the cache is relevant or not. As long as 1 other task needs to access the cache on the same node, that's a speedup over refetching from S3.
• worker nodes are ephemeral, many tasks need to be run at the same time or in a long series to get the most benefit

[hamersaw]

This looks great! A few things that I think could use some clarity:
(1) The flytekit UX seems to have some thought here, IMO it would be very helpful to get some pseudo-code mocks for what this would look like.
(1a) I'm interested in seeing UX of CacheFile with the PodTemplate for mounting PVC. These seems like a pain point as an "in-between" Flyte native integration and something that Flyte can support.
(2) It's not clear to me how creating the local volumes integrates into Flyte's execution model. If I recall, during the demo these were manually created with labels matching specific k8s nodes to ensure placement. Is the plan that FlytePropeller creates these (and cleans up) during workflow execution?

[pryce-turner]

(1) Please see this initial implementation: https://github.com/unionai-oss/cs-hackathon-1123/blob/2d78486979f983f5518db87a678187b31158da63/workflows/caching.py#L49

(1a) It's a bit of a fine line for sure. I've designed it so the CacheFile class fails on init if the podtemplate mountpath isn't present, the exception also bubbles up to the console with a helpful error message. We'll also have to handle the case where those PV/PVCs don't exist and the task pod just hangs waiting for them to get scheduled.

(2) I'll delegate to folks with more cluster knowledge how to setup and teardown those objects. I just used hostname because it was a label that already existed, nodeSelectorTerms could take any arbitrary label that's on a worker node. We could perhaps constrain it to instance_type and only allow scheduling on instances that have instance storage to avoid going over the network to an EBS volume. As far as automating the provisioning of those PV/PVCs, I figure we could just put them on every worker node, wether they get consumed or not shouldn't really have any impact.

[pryce-turner]

Going in a different direction more in line with artifacts.