Add multitenancy support for MLFlow

[brendangadd]

No description provided.

[sylus]

Can probably do something similar to Minio and with our own operator similar to this one:

https://github.com/AICoE/mlflow-tracking-operator

A neat idea would be as soon as a jupyter notebook launches you get a corresponding mlflow or something like that.

[chritter]

@sylus That's a great direction. One step further would be even better: if each namespace had its own MLFlow server instance running. We want to compare models produced in different server instances, especially when doing parallel runs with GPUs. Could this be realized through an operator?

[sylus]

I have now created an operator:

https://github.com/StatCan/mlflow-operator

[sylus]

Have some really cool things that this enables us to do.

@zachomedia what do we think about giving every user their own mlflow based on their kubeflow profile?

[chritter]

@sylus This is great news. I would like to use mlflow to log a couple of models for a project.

[sylus]

Okay so this works now and have made it pretty similar to how the minio setup is.

https://github.com/statcan/mlflow-operator

Currently I have added a daaas and tenant1 folder for the first instance of MLFLOW. From this point on is just a straight copy of the folder to make additional instances and the following 3 additional tasks:

• Minor tweak to naming of kustomization.yaml for that instance
• Add the 3 additional envvars to CI + ./params.sh (will need the Azure AD group to grant auth against)
• Add the redirect url to Azure AD

https://github.com/StatCan/mlflow-operator/blob/master/.github/workflows/release.yml#L86

Right now both k8s and daaas-ml AD groups have access to the following environment:

http://daaas-tenant1-mlflow.covid.cloud.statcan.ca/

For programmatic access we will need to add custom header support to MLFLOW but for right now can use curl:

curl -v --cookie "oidc-cookie-XXXXX=XXXXX" https://daaas-tenant1-mlflow.covid.cloud.statcan.ca/api/2.0/preview/mlflow/experiments/list

Note: You can get the cookie when you first login to MLFLOW in the browser and then use dev tools.

I filed an issue for the mlflow python sdk here: mlflow/mlflow#3012

[sylus]

Subsequent issues can be now be filed against this but this is done.