Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline minio artifact secret - named mlpipeline-minio-artifact; workflow expects minio-artifact #193

Closed
jlewi opened this issue Jul 3, 2019 · 2 comments · Fixed by #198
Closed

Pipeline minio artifact secret - named mlpipeline-minio-artifact; workflow expects minio-artifact #193

jlewi opened this issue Jul 3, 2019 · 2 comments · Fixed by #198
Labels
priority/p0

Comments

@jlewi
Copy link
Contributor

jlewi commented Jul 3, 2019

I tried running the pipelines sample exit on run and the containers failed to start.

Warning  FailedMount  67s (x12 over 9m22s)  kubelet, gke-kf-vmaster-n03-kf-vmaster-n03-cpu-9b8ee2fb-bxb6  MountVolume.SetUp failed for volume "minio-artifact" : secrets "minio-artifact" not found

I have a secret named "mlpipeline-minio-artifact"

mlpipeline-minio-artifact

My guess is kustomize is adding mlpipeline as a common prefix.

We recently upgraded the pipelines to the latest image. Not sure if that changed things.
@yanniszark yanniszark mentioned this issue Jul 3, 2019
Closed
IronPan added a commit to IronPan/manifests that referenced this issue Jul 3, 2019
@IronPan
Update argo artifact repo secret name
bbe3f16 
This is to align with https://github.com/kubeflow/kubeflow/blob/8a01606dc43b217a221eaf2fea2a8207d72ad75d/kubeflow/argo/prototypes/argo.jsonnet#L13

Fix kubeflow#193
IronPan added a commit to IronPan/manifests that referenced this issue Jul 3, 2019
@IronPan
Update argo artifact repo secret name
0c9137b 
This is to align with https://github.com/kubeflow/kubeflow/blob/8a01606dc43b217a221eaf2fea2a8207d72ad75d/kubeflow/argo/prototypes/argo.jsonnet#L13

Fix kubeflow#193
This was referenced Jul 3, 2019
Closed
Merged
k8s-ci-robot pushed a commit that referenced this issue Jul 3, 2019
@IronPan @k8s-ci-robot
Update argo artifact repo secret name (#198)
1868812 
* Update argo artifact repo secret name

This is to align with https://github.com/kubeflow/kubeflow/blob/8a01606dc43b217a221eaf2fea2a8207d72ad75d/kubeflow/argo/prototypes/argo.jsonnet#L13

Fix #193

* Update argo-base_test.go

* Update argo-overlays-istio_test.go
@SoheilSalehian
Copy link

SoheilSalehian commented Apr 13, 2021 •
edited
Loading

Hi @IronPan I am still getting this on GCP running from OSX, did you ever get a fix for this?

Here's the error in the pod:

Warning  FailedMount  6s (x5 over 14s)  kubelet, gke-kubeflow-staging-new-pool-1123efe7-cq83  MountVolume.SetUp failed for volume "mlpipeline-minio-artifact" : secret "mlpipeline-minio-artifact" not found

{Version:kustomize/v4.0.5 GitCommit:9e8e7a7fe99ec9fbf801463e8607928322fc5245 BuildDate:2021-03-08T20:53:03Z GoOs:darwin GoArch:amd64}

@edwardzjl
Copy link

Hi @IronPan I am still getting this on GCP running from OSX, did you ever get a fix for this?

Here's the error in the pod:

Warning  FailedMount  6s (x5 over 14s)  kubelet, gke-kubeflow-staging-new-pool-1123efe7-cq83  MountVolume.SetUp failed for volume "mlpipeline-minio-artifact" : secret "mlpipeline-minio-artifact" not found

{Version:kustomize/v4.0.5 GitCommit:9e8e7a7fe99ec9fbf801463e8607928322fc5245 BuildDate:2021-03-08T20:53:03Z GoOs:darwin GoArch:amd64}

I got the same error, the reason is secret mlpipeline-minio-artifact is in namespace "kubeflow" and the pod requires this secret is in per-user namespace (kubeflow-user-example-com for example)

I found a temporary fix in this comment

@HumairAK HumairAK mentioned this issue May 3, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority/p0
Projects
None yet
Milestone
No milestone
3 participants
@jlewi @SoheilSalehian @edwardzjl