Remove IAM user usage in Kubeflow and replace that with IAM role

[goswamig]

We want kubeflow to be completely off the usage of static credentials but only using IRSA.

1. figure out the IAM user usage in kubeflow component.
2. Replace them with IAM role using IRSA.

[revolutionisme]

@goswamig Is there any update on this? IIRC, we could use IRSA for setting up kubeflow pipelines separately but this seems to be not possible with Kubeflow and this is a critical requirement for us as we are restricted to creating static users on our AWS accounts!

[icereed]

This is a very important feature to us since static IAM users are a security issue because the credentials can leak.
Highly appreciated if this is being worked on. 👍

[muthurajr]

@revolutionisme @icereed A patch for the minio deployment using s3 gateway can be applied as a temporary solution.

An example,

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  template:
    spec:
      serviceAccountName: minio
      containers:
        - name: minio
          image: minio/minio:RELEASE.2022-02-01T18-00-14Z
          args:
            - gateway
            - s3
          volumeMounts:
          - name: data
            $patch: delete
      volumes:
        - name: data
          $patch: delete

Here "minio" is a eks service account with IAM role attached, and container image needs update as well.

[goswamig]

a lot of discussion around this kubeflow/pipelines#3405

[goswamig]

Current status:

So there are multiple places where we can use IRSA for S3 access and these are blocked by Minio feature gap of support

Kubeflow pipeline

KFServing/KServing

Tensorboard on KubeFlow

Probably many other places...

There are places where we use IRSA today since its not blocked minio
for e.g.
in Kubeflow pipeline component

There are places where we can use IRSA
for e.g. Notebook pod where individual users should have limited access from their notebook pod.

Regarding Kubeflow pipeline

We needed to add support for IRSA on following components

1. Argo-workflow: This support is already available from argo-workflow 2.5.0, we had version
2. Minio-go for backend: There was a PR which was closed prematurely, we need to revive it and drive it to completion.
3. Minio-js for frontend: Once Minio-js-7.0.27 is available with IRSA feature, we need to include the changes something like mentioned in this issue

[surajkota]

Created an issue to support this in Kserve: kserve/kserve#2113

[goswamig]

minio-js-7.0.27 was released few days back https://github.com/minio/minio-js/tree/7.0.27

[surajkota]

PRs open for kserve/kserve#2113
kserve/kserve#2377
kserve/website#177

[ryansteakley]

tracking kubeflow/pipelines#8502

[aaj-synth]

Is there an update on this issue?

[ryansteakley]

@aaj-synth its been resolved, IRSA has been usable since 1.7 release in AWS distro