Update manifests to enable authorization check mechanisms for katib-UI in kubeflow mode #2041

apo-ger · 2022-11-29T10:11:57Z

What this PR does / why we need it:
This is a follow-up PR from: #1983

Currently, in kubeflow mode, any workload/pod can set the kubeflow-userid header and communicate with the backend of the Katib UI. Since the code for securing the backend is merged we need to provide a way to enable this feature.

Changes to install-with-kubeflow manifests:

Enable Istio sidecar injection for katib-ui component
Introduce AuthorizationPolicies:
- katib-ui: Allow traffic only from the istio-ingressgateway towards the UI backend. We can trust IGW as we are sure it's setting the user header correctly.
Enable authz checks for katib-ui backend (set APP_DISABLE_AUTH ENV var to false)
Update ClusterRole to provide RBAC persmissions for the katib-ui so it can create SubjectAccessReview objects

Refs: #1547

apo-ger · 2022-11-29T10:49:16Z

@kimwnasptd @andreyvelich @johnugeorge In this PR I introduce the required manifest changes only for the katib-ui component. We could go one step further and enable istio for the rest of the components as well.

This would require additional AuthorizationPolicies to allow traffic but without the kubeflow-userid header set (to avoid impersonations) for certain components (e.g DB-manager, #2022).

Furthermore, in some cases we could apply narrower policies for hardening the security even more (e.g allow only katib-db-manager to talk to katib-mysql for storing/fetching metric logs).

As mentioned in #1983 (comment) and #1983 (comment) we need to test sidecar changes carefully.

Should we outline a full list of requirements for this effort?

johnugeorge · 2022-12-13T18:15:35Z

@apo-ger We don't have a mechanism to test these changes. Did you test all workflows from Kubeflow dashboard?

apo-ger · 2022-12-19T13:11:29Z

Hi @johnugeorge, apologies for the late reply.

We don't have a mechanism to test these changes. Did you test all workflows from Kubeflow dashboard?

Enabling Istio sicecar injection for all Katib components would require us to introduce appropriate AuthorizationPolices. More specifically:

katib-controller: Allow traffic to the webhook port in order to let the K8s api server send traffic to this endpoint.
- When an Experiment CR is submitted, the Katib Experiment mutating and validating webhook is called to set the default values for the Experiment CR and validate the CR.
katib-ui: Allow traffic only from the istio-ingressgateway towards the UI backend
- Add authorization mechanisms in new Katib UI backend #1983
- We can trust IGW as we are sure it's setting the user header correctly.
katib-mysql: The db manager's persistence layer. Allow traffic only from db-manager.
katib-db-manager: Allow ALL traffic, but without the kubeflow-userid header set, as Katib jobs are currently not supporting Istio sidecar injection.
- This is because trial pods are part of a Job, see: Support Istio-sidecar handling #1638

I have tested that the above AuthorizationPolicies are sufficient

johnugeorge · 2022-12-20T16:28:08Z

Can you fix lint?

Also, which rule is responsible for katib-mysql? The db manager's persistence layer. Allow traffic only from db-manager.

manifests/v1beta1/components/controller/controller.yaml

andreyvelich · 2022-12-21T17:37:48Z

@apo-ger Please can we post-pone changes for other components (katib-controller, katib-db-manager, katib-mysql) to the next release ?
I think we should be very careful with enabling istio proxy for Katib Controller, etc. Since we don't have any E2Es to verify this change.

E.g. When Early Stopping is enabled, early stopping container queries data from the Katib DB: https://github.com/kubeflow/katib/blob/master/pkg/earlystopping/v1beta1/medianstop/service.py#L151.
What do you think @johnugeorge ?
/hold for the review

johnugeorge · 2023-01-20T19:03:36Z

@apo-ger Can you update with Katib-UI changes in this PR and take other changes in a separate PR?

Changes to install-with-kubeflow manifests: * Enable istio sidecar injection for katib-ui component * Add AuthorizationPolicy to allow only istio-ingressgateway to talk to katib-ui [user traffic]. * Set APP_DISABLE_AUTH ENV var to false when in kubeflow-mode to enable authorization checks in UI's backend * Extend the RBAC persmissions of katib-ui so it can crate SAR objects when in kubeflow-mode Signed-off-by: Apostolos Gerakaris <[email protected]>

Introduce authn/authz checks in the backend Signed-off-by: Apostolos Gerakaris <[email protected]>

apo-ger · 2023-01-23T06:54:13Z

I updated this PR to keep only the changes related to the katib-ui component as mentioned in #2041 (comment)

pkg/new-ui/v1beta1/backend.go

Signed-off-by: Apostolos Gerakaris <[email protected]>

manifests/v1beta1/installs/katib-with-kubeflow/kustomization.yaml

tenzen-y · 2023-01-23T07:32:24Z

@apo-ger Thanks for submitting this PR.
Can you update the PR description? The correct description helps to track changes.

Signed-off-by: Apostolos Gerakaris <[email protected]>

tenzen-y · 2023-01-23T07:59:13Z

LGTM
/assign @johnugeorge @andreyvelich

johnugeorge · 2023-01-23T17:17:31Z

/lgtm

tenzen-y · 2023-01-24T16:05:10Z

@andreyvelich PTAL.

andreyvelich

Thanks a lot for these updates @apo-ger!
/lgtm
/approve

google-oss-prow · 2023-01-24T16:21:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, apo-ger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tenzen-y · 2023-01-24T16:25:05Z

/hold cancel

google-oss-prow bot requested review from anencore94 and tenzen-y November 29, 2022 10:12

google-oss-prow bot added the size/M label Nov 29, 2022

apo-ger force-pushed the feature-authz-kf-katib branch from b46c954 to 3d2c9d4 Compare November 29, 2022 11:05

apo-ger changed the title ~~Upgrade manifests to enable authorization check mechanisms for katib-UI in kubeflow mode~~ Update manifests to enable authorization check mechanisms for katib-UI in kubeflow mode Nov 29, 2022

apo-ger force-pushed the feature-authz-kf-katib branch 2 times, most recently from 699066e to 4590203 Compare December 13, 2022 17:57

apo-ger force-pushed the feature-authz-kf-katib branch from 4590203 to a7a137f Compare December 21, 2022 14:32

google-oss-prow bot added size/L and removed size/M labels Dec 21, 2022

apo-ger force-pushed the feature-authz-kf-katib branch from a7a137f to d4101cd Compare December 21, 2022 14:35

johnugeorge reviewed Dec 21, 2022

View reviewed changes

manifests/v1beta1/components/controller/controller.yaml Outdated Show resolved Hide resolved

apo-ger force-pushed the feature-authz-kf-katib branch 2 times, most recently from 0d793ad to 0e86177 Compare December 21, 2022 15:53

google-oss-prow bot added the do-not-merge/hold label Dec 21, 2022

apo-ger added 2 commits January 23, 2023 08:36

UI(back): Secure /katib/fetch_trial/ route

d3728da

Introduce authn/authz checks in the backend Signed-off-by: Apostolos Gerakaris <[email protected]>

apo-ger force-pushed the feature-authz-kf-katib branch from 0e86177 to d3728da Compare January 23, 2023 06:47

google-oss-prow bot added size/M and removed size/L labels Jan 23, 2023

tenzen-y reviewed Jan 23, 2023

View reviewed changes

pkg/new-ui/v1beta1/backend.go Outdated Show resolved Hide resolved

review: Remove duplicate dependencies

20aff71

Signed-off-by: Apostolos Gerakaris <[email protected]>

tenzen-y reviewed Jan 23, 2023

View reviewed changes

manifests/v1beta1/installs/katib-with-kubeflow/kustomization.yaml Outdated Show resolved Hide resolved

review: Move patch to a separate file

aad7db8

Signed-off-by: Apostolos Gerakaris <[email protected]>

apo-ger force-pushed the feature-authz-kf-katib branch from 5b572d5 to aad7db8 Compare January 23, 2023 07:47

google-oss-prow bot assigned andreyvelich and johnugeorge Jan 23, 2023

google-oss-prow bot added the lgtm label Jan 23, 2023

andreyvelich approved these changes Jan 24, 2023

View reviewed changes

google-oss-prow bot added the approved label Jan 24, 2023

google-oss-prow bot removed the do-not-merge/hold label Jan 24, 2023

google-oss-prow bot merged commit 0421327 into kubeflow:master Jan 24, 2023

kimwnasptd mentioned this pull request Mar 17, 2023

[0.15-rc.1] 401 error: user header not present #2126

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update manifests to enable authorization check mechanisms for katib-UI in kubeflow mode #2041

Update manifests to enable authorization check mechanisms for katib-UI in kubeflow mode #2041

apo-ger commented Nov 29, 2022 •

edited

Loading

apo-ger commented Nov 29, 2022 •

edited

Loading

johnugeorge commented Dec 13, 2022

apo-ger commented Dec 19, 2022 •

edited

Loading

johnugeorge commented Dec 20, 2022

andreyvelich commented Dec 21, 2022 •

edited

Loading

johnugeorge commented Jan 20, 2023

apo-ger commented Jan 23, 2023

tenzen-y commented Jan 23, 2023

tenzen-y commented Jan 23, 2023

johnugeorge commented Jan 23, 2023

tenzen-y commented Jan 24, 2023

andreyvelich left a comment

google-oss-prow bot commented Jan 24, 2023

tenzen-y commented Jan 24, 2023

Update manifests to enable authorization check mechanisms for katib-UI in kubeflow mode #2041

Update manifests to enable authorization check mechanisms for katib-UI in kubeflow mode #2041

Conversation

apo-ger commented Nov 29, 2022 • edited Loading

apo-ger commented Nov 29, 2022 • edited Loading

johnugeorge commented Dec 13, 2022

apo-ger commented Dec 19, 2022 • edited Loading

johnugeorge commented Dec 20, 2022

andreyvelich commented Dec 21, 2022 • edited Loading

johnugeorge commented Jan 20, 2023

apo-ger commented Jan 23, 2023

tenzen-y commented Jan 23, 2023

tenzen-y commented Jan 23, 2023

johnugeorge commented Jan 23, 2023

tenzen-y commented Jan 24, 2023

andreyvelich left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Jan 24, 2023

tenzen-y commented Jan 24, 2023

apo-ger commented Nov 29, 2022 •

edited

Loading

apo-ger commented Nov 29, 2022 •

edited

Loading

apo-ger commented Dec 19, 2022 •

edited

Loading

andreyvelich commented Dec 21, 2022 •

edited

Loading