feat: directory checkpoint storage [DET-9594]

[ioga]

Description

Introduce a new checkpoint storage type directory which writes the checkpoints and tensorboard files to a local directory.

Intended use cases are 1) using k8s PVC for ckpt storage 2) better representing detached mode storage paradigm

See included docs for the usage details.

Test Plan

Automated test for bind-mounted storage on agent RM is included:

1. bind mount a directory into the experiment
2. configure the same directory for storage using the new option
3. make sure the data is saved to the mounted directory

k8s pod spec setup:

1. make a PVC inside a k8s cluster, e.g. on GKE.
2. mount it into an experiment using the pod_spec.
3. make sure the data is on the PVC

For example, using a GKE cluster, make a yaml for PVC and create it using kubectl create -f <filename>

# pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard-rwo

Run an experiment with the following extra content in the config, which'll mount and use the PVC. Since our PVC is a simple GKE PVC, it can only be mounted on a single node, so this must be a single slot experiment (or multi-slot but single-node, if you're brave enough).

# appended to your exp config yaml
checkpoint_storage:
  type: directory
  container_path: /tmp/somepath2/

environment:
  pod_spec:
    apiVersion: v1
    kind: Pod
    spec:
      volumes:
        - name: pvc-demo-vol
          persistentVolumeClaim:
            claimName: pvc-demo
      containers:
        - name: determined-container
          volumeMounts:
            - name: pvc-demo-vol
              mountPath: /tmp/somepath2

Then, run a notebook using the same PVC to check it's there:

# pvc-notebook.yaml
environment:
  pod_spec:
    apiVersion: v1
    kind: Pod
    spec:
      volumes:
        - name: pvc-demo-vol
          persistentVolumeClaim:
            claimName: pvc-demo
      containers:
        - name: determined-container
          volumeMounts:
            - name: pvc-demo-vol
              mountPath: /tmp/somepath2

det notebook start --config resoureces.slots=0 --config-file pvc-notebook.yaml

In the notebook in /tmp/somepath2, you should see a bunch of uuids for the checkpoints.
Inside the notebook, you can run det e list-checkpoints <experiment id>, find a checkpoint uuid, and then det checkpoint download <checkpoint uuid>. this should "download" the checkpoint into the current directory.

Commentary (optional)

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

[netlify]

✅ Deploy Preview for determined-ui ready!

Name	Link
🔨 Latest commit	03ec7aa
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/654136459fa021000852c8b4
😎 Deploy Preview	https://deploy-preview-8255--determined-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[stoksc]

backend +1

[mpkouznetsov]

👍

[mpkouznetsov]

LGTM

[tara-det-ai]

tensorboard should be TensorBoard in all instances in the text

[tara-det-ai]

.. warning::

When downloading checkpoints (e.g., using det checkpoint download), Determined assumes the same
directory is present locally at the same container_path.

[tara-det-ai]

.. warning::

When downloading checkpoints (e.g., using det checkpoint download), Determined assumes the same
directory is present locally at the same container_path.

[tara-det-ai]

suggested edits

[ioga]

merging over a lint-react flake