Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(backend): isolate artifacts per namespace/profile/user using only one bucket #7725

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open

feat(backend): isolate artifacts per namespace/profile/user using only one bucket #7725

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c9c3a9e
Seperate artifacts per namespace/profile/user
juliusvonkohout May 13, 2022
1218655
sync.py
juliusvonkohout May 13, 2022
fb03d90
Create Dockerfile.pipelines-profile-controller
juliusvonkohout May 13, 2022
8ca1101
Merge branch 'kubeflow:master' into patch-22
juliusvonkohout May 13, 2022
e8a7122
Update deployment.yaml
juliusvonkohout May 13, 2022
e99ab6a
Update sync.py
juliusvonkohout May 13, 2022
b681570
more secure iam policy
juliusvonkohout May 14, 2022
27c6795
enable lifecycle policy
juliusvonkohout May 14, 2022
f1ab99b
Version 9 is more compatible
juliusvonkohout May 14, 2022
9f3efc4
typographical error
juliusvonkohout May 14, 2022
8b15871
Update sync.py
juliusvonkohout May 14, 2022
8e15e56
Update Dockerfile.pipelines-profile-controller
juliusvonkohout May 14, 2022
eff65ea
Merge branch 'kubeflow:master' into patch-22
juliusvonkohout May 18, 2022
1c292a9
Update composite-controller.yaml
juliusvonkohout May 18, 2022
6bdffd5
Merge branch 'kubeflow:master' into patch-22
juliusvonkohout Aug 23, 2022
cbc549a
move changes to the minio distribution
Aug 23, 2022
5743208
rename distribution
Aug 23, 2022
a869701
rename distribution
Aug 23, 2022
63ffae9
fix kustomize build
Aug 23, 2022
1b13e66
remove unecessary changes
Aug 23, 2022
42a3a1a
remove wrong files
Aug 23, 2022
f6f3755
Update composite-controller-patch.yaml
juliusvonkohout Aug 26, 2022
1485d6e
Update sync.py
juliusvonkohout Aug 26, 2022
4988713
Update OWNERS
juliusvonkohout Aug 29, 2022
a4906e0
Update sync.py
juliusvonkohout Nov 3, 2022
b195034
Merge branch 'kubeflow:master' into patch-22
juliusvonkohout Nov 3, 2022
5674951
Update sync.py
juliusvonkohout Nov 4, 2022
82883b5
Merge branch 'kubeflow:master' into patch-22
juliusvonkohout Nov 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions backend/Dockerfile.pipelines-profile-controller
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
FROM docker.io/minio/mc:RELEASE.2019-08-14T20-49-49Z as minio-cli

FROM python:3.7
# curl -o mc 'https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2019-08-14T20-49-49Z'
COPY --from=minio-cli /usr/bin/mc /app/mc
RUN pip3 install 'boto3==1.22.*' 'minio==7.1.*' 'kubernetes==23.3.*'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My advice is to install these python library and minio cli client during runtime. New image has to pass license review and has to be served under new registry path. I would avoid that if possible.

Copy link
Member Author

@juliusvonkohout juliusvonkohout Jun 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could make a runtime check and install them if they are not already there (self-built images), but we need them for the minio configuration. Another option could be to use the minio python client only. Anyway having downloads at runtime makes us dependent on the network and availability of pypi.org and the minio mc download site. I would prefer a static image that always works.

By the way did you read "@zijianjoy The minio mc license was changed quite late minio/mc@1402987 so that should be fine. We are using a way older mc, exactly the same date as your current minio-license-compatible image. The python SDK is still Apache 2.0 https://github.com/minio/minio-py/blob/master/LICENSE.

which minio alternative do you have in mind for the next years? ceph-rook?"

Boto3 is also apache 2.0 https://github.com/boto/boto3/blob/develop/LICENSE together with the kubernetes library https://github.com/kubernetes-client/python/blob/master/LICENSE

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you were to create a new image, then it needs to be integrated into our test and release process. Here're some pointers on top of my head, there could be more.
https://github.com/kubeflow/pipelines/blob/master/.cloudbuild.yaml
https://github.com/kubeflow/pipelines/blob/master/.release.cloudbuild.yaml


ENTRYPOINT ["python3" ]
3 changes: 3 additions & 0 deletions manifests/kustomize/env/multi-user-minio/OWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
approvers:
- juliusvonkohout
- TobiasGoerke
3 changes: 3 additions & 0 deletions manifests/kustomize/env/multi-user-minio/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
This is a distribution for standalone Kubernetes clusters (Minikube, Openshift, etc.) that use MinIO as storage backend.
It isolates the users on the MinIO S3 storage and has other security fixes.
We will continue to harden this distribution with more security advancements.
40 changes: 40 additions & 0 deletions ...ze/env/multi-user-minio/argo-workflow-controller/workflow-controller-configmap-patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: workflow-controller-configmap
data:
containerRuntimeExecutor: emissary
artifactRepository: |
archiveLogs: true
s3:
endpoint: "minio-service.$(kfp-namespace):9000"
bucket: "$(kfp-artifact-bucket-name)"
keyFormat: "private-artifacts/{{workflow.namespace}}/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}"
insecure: true
accessKeySecret:
name: mlpipeline-minio-artifact
key: accesskey
secretKeySecret:
name: mlpipeline-minio-artifact
key: secretkey
executor: |
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 0.01
memory: 32Mi
limits:
cpu: 0.5
memory: 512Mi
args:
- --loglevel
- debug
- --gloglevel
- "6"
workflowDefaults: |
spec:
securityContext:
fsGroup: 1000
fsGroupChangePolicy: "OnRootMismatch"
runAsUser: 1000
runAsGroup: 0
8 changes: 8 additions & 0 deletions manifests/kustomize/env/multi-user-minio/kubeflow-pipelines-profile-controller/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
FROM docker.io/minio/mc:RELEASE.2019-08-14T20-49-49Z as minio-cli

FROM python:3.7
# curl -o mc 'https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2019-08-14T20-49-49Z'
COPY --from=minio-cli /usr/bin/mc /app/mc
RUN pip3 install 'boto3==1.22.*' 'minio==7.1.*' 'kubernetes==23.3.*'

ENTRYPOINT ["python3" ]
12 changes: 12 additions & 0 deletions ...io/kubeflow-pipelines-profile-controller/kubeflow-pipelines-profile-controller-patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubeflow-pipelines-profile-controller
spec:
template:
spec:
serviceAccountName: meta-controller-service
containers:
- name: profile-controller
image: mtr.devops.telekom.de/ai/pipelines-profile-controller:2.0.0-alpha.3

Loading