Support production level volume types for model data

[lianhao]

Currently, our k8s installation (either helm-charts or manifest) have hostPath volume type to store the model data to save model download time. This is NOT ready for production environment, and is NOT suitable for scale outing with multiple nodes.

In production environment, users often use persistent volume(PV/PVC) to store those kind of data. We should give users the options to choose which volume type they want to use.

For PVs, it should have annotation "pv.beta.kubernetes.io/gid: " so non root user in pod can have write access to the volume mounted from this PV. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#access-control as reference.

[endomorphosis]

I agree with this, and this is one of the reasons why previous to OPEA I built huggingface model manager, that pulls models from ipfs / s3 / huggingface, and evicts models as needed.

[daisy-ycguo]

@lianhao can this issue be closed?

[lianhao]

nope, we didn't have time to resolve this in M3. should target for next release

[moting9]

hostpath is easy for getting started, maybe we can have a "sample" dir in the repo, and add diverse nfs, s3 and so on backend pv/pvc to store model files for user reference.