[feature] Allow manifests to set external Object Store to deploy Kubeflow Pipelines

[rimolive]

Feature Area

What feature would you like to see?

Following this Slack thread about using an external minio, users should be able to deploy kfp using an external Object Store instance.

What is the use case or pain point?

Manifests are tied to the bundled minio instance, which is very old, and does not fulfill the users requirements for a secure kfp installation

Is there a workaround currently?

It is possible to either tweak the manifests or configure manually each kfp Deployment to point to an external objStore, but it is undocumented and even experienced users wouldn't be able to do this without spending a lot of time and effort.

---

Love this idea? Give it a 👍.

[HumairAK]

Related: #9689

/assign @HumairAK

[tarekabouzeid]

Hi @rimolive ,
I am also using manifests installation and connecting kubeflow to an external minio cluster. I got it to work only for logs for KFP and artifacts. But when i tried to integrate the KFP to use external minio cluster for caching and using it as pipelineroot, then i got alot of issues in argo not being able to read the "workflow-controller-configmap" which contains "artifactRepository" having my minio configuration.

I spent alot of time on this yesterday and probably will continue working on this during this week, if i get it to work i will update you.

Below are my findings so far:

pipeline logs:

I0422 18:09:38.818730      37 cache.go:139] Cannot detect ml-pipeline in the same namespace, default to ml-pipeline.kubeflow:8887 as KFP endpoint.
I0422 18:09:38.818738      37 cache.go:116] Connecting to cache endpoint ml-pipeline.kubeflow:8887
I0422 19:30:08.287624 39 object_store.go:306] Cannot detect minio-service in the same namespace, default to minio-service.kubeflow:9000 as MinIO endpoint.
[KFP Executor 2024-04-22 18:09:39,451 INFO]: Got executor_input:
{
    "inputs": {},
    "outputs": {
        "artifacts": {
            "metadata_path": {
                "artifacts": [
                    {
                        "type": {
                            "schemaTitle": "system.Artifact",
                            "schemaVersion": "0.0.1"
                        },
                        "uri": "minio://mlpipeline/v2/artifacts/my-example/1d863805-b5b3-4fad-aa65-32b72288bef4/par-download-data/metadata_path"
                    }
                ]
            },
            "real_data_path": {
                "artifacts": [
                    {
                        "type": {
                            "schemaTitle": "system.Artifact",
                            "schemaVersion": "0.0.1"
                        },
                        "uri": "minio://mlpipeline/v2/artifacts/my-example/1d863805-b5b3-4fad-aa65-32b72288bef4/par-download-data/real_data_path"
                    }
                ]
            }
        },
        "outputFile": "/tmp/kfp_outputs/output_metadata.json"
    }
}

workflow controller pod logs:

time="2024-04-22T18:44:22.339Z" level=info msg="Processing workflow" namespace=kubeflow-user-test01 workflow=my-example-5pcmp
time="2024-04-22T18:44:22.345Z" level=info msg="Get configmaps 404"
time="2024-04-22T18:44:22.345Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2024-04-22T18:44:22.345Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository

" I tried to create artifact-repositories configmap manually but also didnot work"

content of : workflow-controller-configmap
artifactRepository -->

archiveLogs: true

s3:
  endpoint: "mys3.example:443"
  bucket: "kubeflow-ml-pipeline"
  keyFormat: "artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}"
  insecure: false
  accessKeySecret:
    name: mlpipeline-minio-artifact
    key: accesskey
  secretKeySecret:
    name: mlpipeline-minio-artifact
    key: secretkey

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.