Notebook cannot implicitly connect to/launch runs in the Pipelines client

[ca-scribner]

Currently in a notebook, this does not work:

import kfp
client = kfp.Client()

reporting:

ERROR:root:Failed to read a token from file '/var/run/secrets/kubeflow/pipelines/token' ([Errno 2] No such file or directory: '/var/run/secrets/kubeflow/pipelines/token').
WARNING:root:Failed to set up default credentials. Proceeding without credentials...

I think this is because, when switching to multi-user isolation for pipelines, we missed adding this to the PodDefaults for each user.

As a temporary workaround, used can from a terminal on cluster create a PodDefault in their namespace:

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: access-ml-pipeline
  namespace: "<YOUR_USER_PROFILE_NAMESPACE>"
spec:
  desc: Allow access to Kubeflow Pipelines
  selector:
    matchLabels:
      access-ml-pipeline: "true"
  volumes:
    - name: volume-kf-pipeline-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 7200
              audience: pipelines.kubeflow.org      
  volumeMounts:
    - mountPath: /var/run/secrets/kubeflow/pipelines
      name: volume-kf-pipeline-token
      readOnly: true
  env:
    - name: KF_PIPELINES_SA_TOKEN_PATH
      value: /var/run/secrets/kubeflow/pipelines/token

which will then enable new notebook servers to select KFP access in the Configurations section of the new notebook page.

Another hack that will let you use kfp from an existing notebook is to copy your session cookie from your browser's development tools, then do:

cookies = 'authservice_session=<your cookie from your browser>'
client = kfp.Client(
    host="http://10.64.140.43.nip.io/pipeline",
    cookies=cookies,
)

[Barteus]

Workaround with PodDefault works for me, thanks!

[ca-scribner]

kubeflow/pipelines#5138 discusses this problem and some possible fixes. The fixes are not merged and the discussion seems stale but I'll see if I can revive anything.

[ca-scribner]

We can do kubeflow/pipelines#6629 ourselves (we already run a modified version of the sync.py file in our profile-controller charm anyway). That effectively puts the PodDefault above in every user's namespace, meaning that all users will see the configuration button for pipelines on the notebook creation page by default.

Users still need to select that configuration button to get pipeline access in their notebook, but the situation would be better than it is now. Fixing/closing this issue should include a change to our documentation to instruct users about the button

[ca-scribner]

Nevermind, we can also make the trivial patch to our notebook spawner default config from kubeflow/kubeflow#6160 and the button is checked by default!

[ca-scribner]

Closed by canonical/kfp-operators#49 and canonical/notebook-operators#24