add hub.config passthrough and use it for all auth config #1943

consideRatio · 2020-12-14T03:35:59Z

Closes #1871.
Closes #1379 by updating the documentation examples to be directly mapped to LDAP config.

PR Status

Review -> merge remain, I don't know of anything I'd like to do further at this point for this PR.

The deprecation logic is tested against all the auth config examples in the docs we had, and it worked out fine. Automating testing seems overkill as we shouldn't make changes to it over time.
An e2e test of the deprecation and that the new configuration is valid has been done successfully.

Review suggestions

For me, the key part to review is the deprecation mechanisms. Secondary, we can focus on the documentation language etc, but I think we should iterate on that later as this PR is required for a release that I'd like to make. Below are some review points anyhow.

1. Updates to schema.yaml in 5d5778f
2. Changes in jupyterhub_config.py e4b64ce 72e3967
3. Adding of null safe lookup utility in ee323db
4. Handling of auth.state.cryptoKey in 9ed09c0
5. Transform of configuration in authentication.md 245e86c
@minrk I replaced replaced env var based configuration as well, because I recall your refactoring in oauthenticator made us no longer need to use env vars + traitlet config but could instead rely on only traitlet config of token_url etc. I have also verified this worked for a deployment of mine. Does it sound okay to you?
6. The documentation added before the "Configuring authenticator classes" section - PR preview

Summary

This Helm chart's configuration of authenticator classes such as GitHubOAuthenticator were as observed by the influx of issues not so straightforward for users, and it wasn't so straightforward for maintainers either. We had to keep up with adding mappings between the helm charts configuration and the authentication classes configuration.

In this PR, we leave that system behind in favor of a more general system. In this transition users having configured authentication will fail directly when they run helm upgrade and be provided with an useful error message to help with the transition.

Transition message

config.yaml

auth:
  key1: val1
  key2:
    key3: val3
  type: github
  google:
    clientSecret: test-secret
    clientId: test-id
    callbackUrl: callback-url
  scopes:
    - profile
    - email
  state:
    enabled: true
  admin:
    access: true
    users:
      - cat
  whitelist:
    users:
      - kitten1
  allowedUsers: # maps to the same as whitelist.users
    - kitten2

helm upgrade

The JupyterHub Helm chart's auth config has been reworked and requires changes.

The new way to configure authentication in chart version 0.11.0+ is printed
below for your convinience. The values are not shown by default to ensure no
secrets are exposed, run helm upgrade with --set global.safeToShowValues=true to
show them.

hub:
  config:
    Authenticator:
      admin_users: '***'
      allowed_users: '***'
      enable_auth_state: '***'
    JupyterHub:
      admin_access: '***'
      authenticator_class: oauthenticator.github.GitHubOAuthenticator
    OAuthenticator:
      client_id: '***'
      client_secret: '***'
      oauth_callback_url: '***'
      scope: '***'
    WarningUnrecognizedConfig:
      key1: '***'
      key2.key3: '***'

For more details, please see the updated auth config documentation at:
https://zero-to-jupyterhub.readthedocs.io/en/latest/administrator/authentication.html

helm upgrade --set global.safeToShowValues=true

...

hub:
  config:
    Authenticator:
      admin_users:
      - cat
      allowed_users:
      - kitten1
      enable_auth_state: true
    JupyterHub:
      admin_access: true
      authenticator_class: oauthenticator.github.GitHubOAuthenticator
    OAuthenticator:
      client_id: test-id
      client_secret: test-secret
      oauth_callback_url: callback-url
      scope:
      - profile
      - email
    WarningUnrecognizedConfig:
      key1: val1
      key2.key3: val3

...

doc/source/conf.py

manics

I made a first pass through the docs, I'll go through the code and schema changes later.

doc/source/administrator/authentication.md

consideRatio · 2020-12-20T23:13:44Z

@manics thank you for your review!

I want to clarify that I have not really changed the text or structure of the documentation about how to configure the specific authenticators, but I'll give it a pass given your review comments. The reason for it being listed as an addition is the rST -> markdown conversion I did to not need to work with rST.

consideRatio · 2020-12-21T10:39:36Z

@manics I've done a full pass of the auth docs following the questions you raised.

Working with Helm templates isn't funny... This has taken a lot of hours to complete.

While these were devleoped, I relied on the following tests to ensure they functioned as I wanted. __A values.yaml file__ ``` test: key1: val1 dict1: key1: val1 emptyDict: {} emptyList: [] empty: ``` __A Helm template file__ ``` {{- if ne (include "jupyterhub.digToJson" (list "test.key1" .Values)) "\"val1\"" }}{{ "digToJson test 1" | fail }}{{ end }} {{- if ne (include "jupyterhub.digToJson" (list "test.dict1" .Values)) "{\"key1\":\"val1\"}" }}{{ "digToJson test 2" | fail }}{{ end }} {{- if ne (include "jupyterhub.digToJson" (list "test.noDict" .Values)) "null" }}{{ "digToJson test 3" | fail }}{{ end }} {{- if ne (include "jupyterhub.digToJson" (list "test.noDict1.noDict2" .Values)) "null" }}{{ "digToJson test 4" | fail }}{{ end }} {{- if ne (include "jupyterhub.digToJson" (list "test.emptyDict" .Values)) "{}" }}{{ "digToJson test 5" | fail }}{{ end }} {{- if ne (include "jupyterhub.digToJson" (list "test.emptyList" .Values)) "[]" }}{{ "digToJson test 6" | fail }}{{ end }} {{- if ne (include "jupyterhub.digToJson" (list "test.empty" .Values)) "null" }}{{ "digToJson test 7" | fail }}{{ end }} {{- if not (include "jupyterhub.digToTrue" (list "test.key1" .Values)) }}{{ "digToTrue test 1" | fail }}{{ end }} {{- if not (include "jupyterhub.digToTrue" (list "test.dict1" .Values)) }}{{ "digToTrue test 2" | fail }}{{ end }} {{- if (include "jupyterhub.digToTrue" (list "test.noDict" .Values)) }}{{ "digToTrue test 3" | fail }}{{ end }} {{- if (include "jupyterhub.digToTrue" (list "test.noDict1.noDict2" .Values)) }}{{ "digToTrue test 4" | fail }}{{ end }} {{- if (include "jupyterhub.digToTrue" (list "test.emptyDict" .Values)) }}{{ "digToTrue test 5" | fail }}{{ end }} {{- if (include "jupyterhub.digToTrue" (list "test.emptyList" .Values)) }}{{ "digToTrue test 6" | fail }}{{ end }} {{- if (include "jupyterhub.digToTrue" (list "test.empty" .Values)) }}{{ "digToTrue test 7" | fail }}{{ end }} ```

I updated the logic for checking if we had a valid configuration as well using the Helm template helpers I introduced as I concluded it was needed to remain sane while checking nested config that may be null at any place. I deleted the entry in secret.yaml as it is only to be used to set an env variable for use by JupyterHub in the hub pod, which in turn is fine to delete because its only used by JupyterHub's CryptKeeper class which is responsible for managing encrypt/decrypt operations for JupyterHub.

choldgraf · 2021-01-07T21:43:32Z

I wonder if @yuvipanda would be particularly interested in this, since we're doing a lot of auth config in 2i2c and this would probably affect us, no? I would review it myself, but I don't think I'm qualified to do so from a tech perspective :-/

minrk

This is awesome work! I left some minor comments in the docs, but so happy for this to move forward.

jupyterhub/templates/_helpers-auth-rework.tpl

minrk · 2021-01-11T09:10:16Z

jupyterhub/values.yaml

+  config:
+    JupyterHub:
+      admin_access: true
+      authenticator_class: dummyauthenticator.DummyAuthenticator


shortcut works and is perhaps easier to understand:

Suggested change

authenticator_class: dummyauthenticator.DummyAuthenticator

authenticator_class: dummy

We shouldn't be using dummyauthenticator anymore, as dummy ships with jupyterhub now.

Ah! So I'll remove jupyterhub-dummyauthenticator from the requirements.txt, and we deprecate the repsitory?

Yes, I believe so. This change is still needed to avoid the current crash loop backoffs because you've removed the dummyauthenticator package but are still using it in config.

Resolved by 5b0a011

doc/source/administrator/authentication.md

Co-authored-by: Min RK <[email protected]>

doc/source/administrator/authentication.md

minrk · 2021-01-12T08:14:28Z

Great work, @consideRatio!

jupyterhub/zero-to-jupyterhub-k8s#1943 Merge pull request #1943 from consideRatio/pr/auth-rework

consideRatio · 2021-01-12T08:18:10Z

Thanks @minrk and @manics for your review efforts!!! :Heart: ❤️ ❤️

meeseeksmachine · 2021-01-14T17:11:38Z

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/jupyterhub-helm-chart-0-11-0-released/7521/1

Breaking changes to fix: - All authenticator config has changed, and must be modified - jupyterhub/zero-to-jupyterhub-k8s#1943 - pdb has been disabled - we should file an issue to investigate if we want it - jupyterhub/zero-to-jupyterhub-k8s#1938 - Any networkpolicy changes we might need - https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/CHANGELOG.md#breaking-changes-1

bsikander · 2022-01-10T15:42:37Z

Two cents from my side while upgrading to 0.11.1.

When I tried to upgrade, helm suggested the configuration changes as mentioned in the PR. So, I just copy pasted the output in my CLI and placed in my config. Hub kept of giving 500 internal server errors and after digging deep inside, i found that hub was not recognising my provided authenticator configs and was using the dummy default authenticator. The root cause of the problem was that I had two hub.* tags in my yaml. One for hub.extraConfig.* and other one that CLI suggested hub.config.*. When I manually merged the two seperate yaml sections into one like below, it started to recognize.

hub:
  config:
     ........
  extraConfig:
     ........

Apart from this, dynamically setting the configurations worked also

hub:
    extraConfig:
        authenticator:
            c.JupyterHub.authenticator_class = "oauthenticator.azuread.AzureAdOAuthenticator"
            c.JupyterHub.admin_access = True
            c.AzureAdOAuthenticator.client_id = ""
            c.AzureAdOAuthenticator.client_secret = ""
            c.AzureAdOAuthenticator.tenant_id = ""
            c.AzureAdOAuthenticator.oauth_callback_url = ""
            c.Authenticator.admin_users = {'usera', 'userb'}

consideRatio marked this pull request as draft December 14, 2020 03:36

consideRatio force-pushed the pr/auth-rework branch 2 times, most recently from 295c66f to b1a954a Compare December 19, 2020 03:56

consideRatio added breaking maintenance labels Dec 19, 2020

consideRatio commented Dec 20, 2020

View reviewed changes

doc/source/conf.py Show resolved Hide resolved

consideRatio changed the title ~~WIP: Auth config rework~~ Auth config reworked for comprehension and maintenance sustainability Dec 20, 2020

consideRatio requested review from manics, minrk and choldgraf and removed request for manics December 20, 2020 19:06

consideRatio marked this pull request as ready for review December 20, 2020 19:10

consideRatio added this to the 0.11.0 milestone Dec 20, 2020

manics reviewed Dec 20, 2020

View reviewed changes

consideRatio mentioned this pull request Dec 25, 2020

Changelog for 0.11.0 #1926

Merged

2 tasks

consideRatio force-pushed the pr/auth-rework branch from 20bf806 to 5ad8aa7 Compare December 29, 2020 23:36

consideRatio mentioned this pull request Jan 4, 2021

docs: 100% MyST Markdown #1974

Merged

consideRatio added 10 commits January 4, 2021 11:27

auth rework: initial work on config transition message

11e57b9

Working with Helm templates isn't funny... This has taken a lot of hours to complete.

auth rework: remove note about hmacauthenticator, 1.0 is released

7f2b8d2

auth rework: locate under hub.config

6e161ab

auth rework: censor values by default

936ec21

auth rework: remove old logic in jupyterhub_config.py

cb7577b

auth rework: add new logic in jupyterhub_config.py

4a9115d

auth rework: improve reliability of help text logic

cef2be6

auth rework: update docs with new config examples

3a6cfc5

docs: add basic hub.baseUrl entry to not make linkcheck fail

24e3ce8

consideRatio force-pushed the pr/auth-rework branch from 51c65b0 to 24e3ce8 Compare January 4, 2021 10:43

docs: please linkcheck

4d4eca1

consideRatio requested a review from yuvipanda January 10, 2021 06:53

minrk reviewed Jan 11, 2021

View reviewed changes

consideRatio and others added 5 commits January 11, 2021 13:08

Apply language suggestions from code review

b19d01b

Co-authored-by: Min RK <[email protected]>

docs: update authenticator class basics

b740fcf

Co-authored-by: Min RK <[email protected]>

docs: add details on provided config example

3610078

Co-authored-by: Min RK <[email protected]>

auth rework: favor XYZOAuthenticator over OAuthenticator

e0948f9

auth rework: use jupyterhub's built in dummy authenticator

2e3a73a

consideRatio force-pushed the pr/auth-rework branch from 898447d to 2e3a73a Compare January 11, 2021 12:44

consideRatio requested a review from minrk January 11, 2021 12:45

auth rework: final fix for jh embedded dummy authenticator

5b0a011

minrk reviewed Jan 12, 2021

View reviewed changes

doc/source/administrator/authentication.md Outdated Show resolved Hide resolved

fix CILogon yaml example

6c44594

minrk approved these changes Jan 12, 2021

View reviewed changes

minrk changed the title ~~Auth config reworked for comprehension and maintenance sustainability~~ add hub.config passthrough and use it for all auth config Jan 12, 2021

minrk merged commit b116faa into jupyterhub:master Jan 12, 2021

consideRatio pushed a commit to jupyterhub/helm-chart that referenced this pull request Jan 12, 2021

[jupyterhub] Automatic update for commit 0.10.6-n212.h5b0a0118

c469c3b

jupyterhub/zero-to-jupyterhub-k8s#1943 Merge pull request #1943 from consideRatio/pr/auth-rework

consideRatio mentioned this pull request Jan 16, 2021

If auth.whitelist.users is non-empty GitHub orgs are ignored #521

Closed

chinmaychandak mentioned this pull request Jan 27, 2021

Not able to get Open Signup working on Kubernetes jupyterhub/nativeauthenticator#132

Closed

yuvipanda mentioned this pull request Feb 4, 2021

Bump hub version to 0.11 2i2c-org/infrastructure#194

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add hub.config passthrough and use it for all auth config #1943

add hub.config passthrough and use it for all auth config #1943

consideRatio commented Dec 14, 2020 •

edited

Loading

manics left a comment

consideRatio commented Dec 20, 2020

consideRatio commented Dec 21, 2020

choldgraf commented Jan 7, 2021 •

edited

Loading

minrk left a comment

minrk Jan 11, 2021

consideRatio Jan 11, 2021

minrk Jan 11, 2021

consideRatio Jan 11, 2021

minrk commented Jan 12, 2021

consideRatio commented Jan 12, 2021

meeseeksmachine commented Jan 14, 2021

bsikander commented Jan 10, 2022 •

edited

Loading

	authenticator_class: dummyauthenticator.DummyAuthenticator
	authenticator_class: dummy

add hub.config passthrough and use it for all auth config #1943

add hub.config passthrough and use it for all auth config #1943

Conversation

consideRatio commented Dec 14, 2020 • edited Loading

PR Status

Review suggestions

Summary

Transition message

manics left a comment

Choose a reason for hiding this comment

consideRatio commented Dec 20, 2020

consideRatio commented Dec 21, 2020

choldgraf commented Jan 7, 2021 • edited Loading

minrk left a comment

Choose a reason for hiding this comment

minrk Jan 11, 2021

Choose a reason for hiding this comment

consideRatio Jan 11, 2021

Choose a reason for hiding this comment

minrk Jan 11, 2021

Choose a reason for hiding this comment

consideRatio Jan 11, 2021

Choose a reason for hiding this comment

minrk commented Jan 12, 2021

consideRatio commented Jan 12, 2021

meeseeksmachine commented Jan 14, 2021

bsikander commented Jan 10, 2022 • edited Loading

consideRatio commented Dec 14, 2020 •

edited

Loading

choldgraf commented Jan 7, 2021 •

edited

Loading

bsikander commented Jan 10, 2022 •

edited

Loading