Review the concept of connections

[viravit]

ODAHU Connections Manager allows to manage access to variety of external systems like VCS (git), Docker Registry, Object Storages (AWS S3, GCP GCS, etc.. ) But Connections Manager is not designed as a dataset registry/catalog that allows to manage multiple versions of datasets needed during ML multiple experiments.

The dataset management is to be a separate service/component that has features :

• API, UI and CLI to register dataset. The data cab be located in external Data Storage / DB.
• Pluggable Data Storages / DBs.
• Metadata (version, notes, labels, comments, schema, etc) is to be associated with datasets
• Access credentials is to be associated with datasets. (reuse connections manager service - TBD)
• Dataset access permissions for user and user groups
• ODAHU Model Training Services is to consume data during training job execution from/through dataset registered in ODAHU.