[frontend] Artifact API Does not support S3 Regions

[kdubovikov]

Artifact fetching API does not seem to support region for s3 resources. If we look here,

pipelines/frontend/server/minio-helper.ts

Line 50 in e8abec2

return new MinioClient({ ...config, accessKey, secretKey, sessionToken });

, the region parameter is not being passed and it is not a part of the artefact API as well. However, MINIO Client API allows you to specify region when creating a client: https://docs.min.io/docs/javascript-client-api-reference.html. I think API should fetch default region using MINIO_REGION environment variable which should be passed as a part of ml-pipeline-ui-artifact k8s deployment env map.

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  kubeflow-manifests

• KFP version:
  1.5.1

Steps to reproduce

1. Launch any pipeline
2. Go to Run UI
3. Try to download any output
4. Observe the error: Failed to get object in bucket [bucket] at path [path]: S3Error: The authorization header is malformed; the region 'us-east-1' is wrong; expecting '[actual bucket region]'

Expected result

Materials and Reference

[kdubovikov]

My bad, I've found that it actually uses AWS_REGION variable here:

pipelines/frontend/server/configs.ts

Line 129 in e8abec2

region: AWS_REGION || 'us-east-1',

. Will add that to deployment to test if it works

[kdubovikov]

It works as expected if I specify AWS_REGION, AWS_SECRET_ACCESS_KEY, and AWS_ACCESS_KEY_ID

[kdubovikov]

Actually, I think that there is still an issue when we use multi-profile setup. The problem lies in the ml-pipeline-ui-artifact deployment which is automatically created for each profile.

Here,

pipelines/manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/sync.py

Line 304 in a0a8f1d

"name": "MINIO_ACCESS_KEY",

we sync only MINIO_* variables. However, when using s3, this code suggests that we should provide AWS_REGION, AWS_SECRET_ACCESS_KEY, and AWS_ACCESS_KEY_ID.

pipelines/frontend/server/configs.ts

Line 129 in e8abec2

region: AWS_REGION || 'us-east-1',

That means that we should add those variables to pipelines-profile-controller. When I add them manually to the ml-pipeline-ui-artifact UI fetches all s3 artifacts in the Pipeline UI as expected.

Another possible solution would be to add MINIO_REGION variable mapping here istead of modifying pipelines-profile-controller

pipelines/frontend/server/configs.ts

Line 139 in e8abec2

minio: {

. Although, I am not entirely sure that it will work.

[rawc0der]

Hi @kdubovikov @chensun,

Also running into this problem after switching Artifact Store persistence to S3.

Wondering if you've seen the latest MinioJS v7.0.27 with Assume Web Identity Role support minio/minio-js#960 ?

This could be part of the fix, without needing any workaround if I'm not mistaken.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.