Pipeline UI can't display s3 Outputs

[kdubovikov]

Describe the bug
I can't download any of the pipeline step outputs due to my bucket residing in different region than us-east-1.

Steps To Reproduce

1. Launch any pipeline
2. Go to Run UI
3. Try to download any output
4. Observe the error: Failed to get object in bucket [bucket] at path [path]: S3Error: The authorization header is malformed; the region 'us-east-1' is wrong; expecting '[actual bucket region]'

Expected behavior
A clear and concise description of what you expected to happen.

Environment

• Kubernetes version
  1.21

• Using EKS (yes/no), if so version?
  Yes, 1.21

• Kubeflow version

• AWS build number

• AWS service targeted (S3, RDS, etc.)
  S3

Screenshots

[kdubovikov]

Actually, it seems to be an issue with the Kubeflow UI itself. If we look here, https://github.com/kubeflow/pipelines/blob/e8abec24fed4c4f8be6f527207b1cec9811ce3e7/frontend/server/minio-helper.ts#L50, the region parameter is not being passed and it is not a part of the artefact API as well. However, MINIO Client API allows you to specify region when creating a client: https://docs.min.io/docs/javascript-client-api-reference.html.

I have filed a bug in Kubeflow Pipelines repo to address this kubeflow/pipelines#8111

[kdubovikov]

Just confirmed that this is not a Kubeflow Pipelines issue. Here, we see that API container actually requires the following variable names:

• AWS_REGION
• AWS_SECRET_ACCESS_KEY
• AWS_ACCESS_KEY_ID

If I add them manually by changing the deployment:

kubectl edit deployment ml-pipeline-ui-artifact -n kubeflow-user-example-com

I see that

Everything works as expected. That means that easy fix would amount to the following:

• I see that AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID are already added here:

  kubeflow-manifests/awsconfigs/apps/pipeline/s3/aws-configuration-patch.yaml

  Line 18 in 259792a

  - name: AWS_ACCESS_KEY_ID

• That means that we only need to add AWS_REGION parameter to the kustomization for everything to work

[kdubovikov]

Created a PR to fix this

[kdubovikov]

I have re-opened kubeflow/pipelines#8111 since this fix does not seem to solve the issue

[AlexandreBrown]

Hey @kdubovikov there is a workaround for viewing S3 artifacts #117 (comment)
Proper support is a WIP from what I understood.

[kdubovikov]

Thanks @AlexandreBrown. Closing this issue as a duplicate