Automatic HTTPS setup for BinderHub

[consideRatio]

Goal

To make it easy to setup BinderHub's Helm chart to easily use HTTPS, which means that network traffic between the user and binderhub will be encrypted.

History

We have used kube-lego to acquire certificates, but it was deprecated and could not comply with a new requirement by Let's Encrypt that we interact with. We have considered kube-lego's successor called cert-manager, but we considered doing so came with a bit too much overhead. This is why we now want a more lightweight solution.

Here is a related issue about not being able to use kube-lego: pangeo-data/pangeo-binder#127

Theory 101

For a BinderHub user to establish a secure communication (HTTPS) with a BinderHub server at binder.example.com, some things need to happen first.

1. Choose a CA (Let's Encrypt)
   We need a common trusted party, a Certificate Authority (CA). Let's Encrypt is such CA that is well trusted and free to use.

2. Prove domain ownership -> acquire signed domain certificate
   The CA can give away a signed domain certificate acting as proof the domain owner needs later, but only to a domain owner that can prove its ownership of the domain to the CA. This is where the ACME protocol is useful. BinderHub can ask the CA for a http01 challenge to prove it. During the http01 challenge, BinderHub will need to respond in a specific way to binder.example.com, which help the CA be confident it is control of the server responding to binder.example.com.

3. Encrypt HTTP / Decrypt HTTPS
   BinderHub needs to encrypt/decrypt all outgoing/incoming traffic. This is called TLS termination and can be done in a standalone manner by a TLS termination proxy or by the webserver serving the BinderHub content. This step requires the certificate we acquired in the previous step.

Additional reading

• https://letsencrypt.org/how-it-works/
• https://howhttps.works/https-ssl-tls-differences/

Z2JH's solution

I want to reuse as much as possible from the Z2JH solution implemented in jupyterhub/zero-to-jupyterhub-k8s#1539 by @yuvipanda. Assuming Z2JH was configured to use this solution, the following would happen.

1. Z2JH would redirect traffic from its original HTTP only entrypoint to a Traefik (v2) proxy.
2. A sidecar container running alongside the Traefik container in a pod would mount a certificate from a secret if there was one that Traefik could use.
3. If no signed domain certificate was available to Treafik, Traefik would acquire a signed domain certificates from Let's Encrypt through a http01 challenge.
   1.If a new signed domain certificate was acquired, the sidecar container would store it to a secret.
4. The Traefik proxy would use the signed domain certificate to terminate TLS and send direct traffic towards the HTTP entrypoint.

Z2JH's Kubernetes Service: proxy-public

BinderHub's planned solution

I plan to either reference or duplicate this code, and make minor changes to help BinderHub specifically.

• The docker image built in Z2JH running as a sidecar container to sync the signed domain certificate with a Kubernetes secret can be reused smoothly. This is because we are using Z2JH as a Helm chart dependency (aka. subchart), and that makes us have access to the image's name and tag from BinderHub's Helm templates which render into Kubernetes resources.
• I intend to duplicate all Kubernetes resources needed though, as I don't see a simple enough way to extract to motivate the added complexity to the project of doing it.
• I intend to reuse the configuration of the jupyterhub subchart, so instead of referencing a BinderHub helm chart configuration about what Traefik image to use, I'll reference .Values.jupyterhub.proxy.traefik.image.name for example.

BinderHub's Kubernetes Service: binder

The Auto HTTPS part is new, the other is left unchanged.

[choldgraf]

cc @betatim who was just on an issue where I mentioned Erik working on security stuff. Tim - this is a "meta" repository for the "Jupyter Meets the Earth" project and some conversation about JupyterHub/Binder-related things might happen here, though in general we will try to keep issues etc in the proper JupyterHub repository. Just wanted to let you know about this repo's existence :-)

[yuvipanda]

This is awesome content, @consideRatio!

I want to suggest an alternate approach. If you see https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/jupyterhub/templates/proxy/autohttps/configmap.yaml#L104, we are defining just one router + backend, pointing to CHP. However, it is trivial to add some templating there for extra routers and extra backends. So the z2jh config could take a list of domain names + their backends (as service names or just DNS entries), and add them to the traefik config. This will make traefik aquire HTTPS certs for all of them, and do some routing as well. This should cause less work, and make it really easy for users of z2jh to use for other external services they might have.

What do you think of this?

[betatim]

👋

Two points:

1. cert-manager on mybinder.org (finally) just worked (for staging, haven't had time to enable it for prod)
2. I support Chris' suggestion that if you want to discuss architecture/plan changes for binderhub it is probably best to move the conversation there with enough time for the people who hang out there to chime in and give input

A question: is it possible to use the traefik that does LE for JupyterHub to also cover other Ingress objects in the k8s namespace? Thinking of mybinder.org where we have several ingress objects that needs certs but aren't part of the binderhub or z2jh chart. The reason for moving forward with cert-manager there was that I think we need something like it tocover all these "extra" ingress objects.

[consideRatio]

@yuvipanda @betatim @choldgraf this topic was to a large extent a rubber ducking exercise that got out of hand and ended up being technically relevant for repos like z2jh and binderhub. I'll ensure to bring discussion there going onwards.

---

I want to suggest an alternate approach.

@yuvipanda sounds good to me! I'm going for it!

1. cert-manager on mybinder.org (finally) just worked (for staging, haven't had time to enable it for prod)

@betatim 🎉 yepp! nginx-ingress + cert-manager is the more reliable and scalable solution that for example could avoid causing disruptions thanks to the ingress proxy pods are HA and the acquisition of certificates can work well still. This would be the more lightweight default solution that can help a new binderhub admin quickly get up and running.

A question: is it possible to use the traefik that does LE for JupyterHub to also cover other Ingress objects in the k8s namespace? Thinking of mybinder.org where we have several ingress objects that needs certs but aren't part of the binderhub or z2jh chart. The reason for moving forward with cert-manager there was that I think we need something like it tocover all these "extra" ingress objects.

@betatim yepp I think Traefik can do this, to configure itself to route traffic arriving to it given ingress resources in Kubernetes, and I think it can also acquire certificates for these resources. But, unlike a nginx-ingress + cert-manager setup, Traefik's open source version cannot work in HA and acquire certificates for the ingress routes.

[consideRatio]

Branched out work and discussion

Z2JH PR: Configuration sustainable from a maintainer perspective

Helm charts are hard to maintain because it requires to plan for everything that someone may want to modify, which is impossible. This PR is about setting us up to avoid that in preparation for the coming related PRs.

Z2JH PR: Testing infrastructure for autohttps

This infrastructure doesn't modify our current tests, but it allows us to mimic having Let's Encrypt in our CI system and the process of acquiring HTTPS certificate as well as using them.

Future: Z2JH PR: Make CI system test with HTTPS

Future: BinderHub PR: Use Z2JH's infra to get HTTPS

[consideRatio]

Status update

This is still ongoing. The effort so far...

• Created a Helm chart for a local ACME server (Pebble), for testing of automatic TLS certificate acquisitions. See https://github.com/jupyterhub/pebble-helm-chart
• Reworking of the CI system for Z2JH, which involves to test automatic TLS certificate acquisition DX: k3s/k3d instead of kind & CI: autohttps testing jupyterhub/zero-to-jupyterhub-k8s#1664
• Fixing of an issue, perhaps the most stubborn issue I've experienced in my life, caused by two intermittent issues that in combination made me struggle a lot.
  • A Traefik upstream issue: acme: sometimes two instead of one ACME order is placed, which sometimes leads to challenge failure traefik/traefik#6871
  • A Z2JH issue: CI/CD: Autohttps reliability issues when acquiring certs with Pebble (unknown behavior with Let's Encrypt) jupyterhub/zero-to-jupyterhub-k8s#1716, acme: unreliable (80% ok) with a fast responding local ACME server traefik/traefik#7033
• Proof of poncept usage of the infrastructure to verify its on the right track for BinderHub: [WIP] Automatic HTTPS and tests jupyterhub/binderhub#1101

What remains is to update jupyterhub/binderhub#1101 now that the pieces are in place, which also requires a bit of CI updates of the BinderHub infrastructure as well.

[consideRatio]

Status update

I have not dropped the ball on this, but my attention isn't fully focused on it. Here is the current status of things. Since last status update...

• Updated CI infrastructure of BinderHub making it easier to integrate additional logic to test HTTPS, this was quite extensive work that I hope also will help the CI system easier to understand and maintain. CI / Local dev refresh and Helm2->3 transition jupyterhub/binderhub#1188 Dependency freezer script, bumped image deps, and doc-requirements.txt cleanup jupyterhub/binderhub#1180
• Helped out with the migration of TravisCI to GitHub actions that became merited when TravisCI stopped providing free credits for the jupyterhub organization. [MRG] Add basic GitHub Actions workflow for testing jupyterhub/binderhub#1209
• Made decisions on network policy configuration to avoid locking in against changing configuration API: netpol: allowedIngressPorts and interNamespaceAccessLabels config added with defaults retaining 0.9.1 current behavior jupyterhub/zero-to-jupyterhub-k8s#1842
• Released JupyterHub Helm chart 0.10.0 0.10.1 0.10.2 with changes needed (Changelog)
• Explore ways BinderHub can configure the JupyterHub Helm chart sustainable from a configuration maintenance perspective, we don't want to introduce the need for users to specify things multiple times for example.

Future

• Help jupyterhub/binderhub land so binderhub starts depending on JupyterHub 0.10.2: Bump Z2JH to 0.10.6 jupyterhub/binderhub#1184
• Create a PR configuring BinderHub to use the system available and testing it using the Pebble Helm chart as ACME server and with network policies.

[consideRatio]

Status update

I lost momentum waiting for step-along-the-way PRs to be merged and didn't get back on it. I have jupyterhub/binderhub#1179 open still, but it isn't a road block I think.

This work can continue at this point.