Deprecate kube-lego and document cert-manager usage

[clkao]

Note that gce ingress is currently having a regression preventing first-time cert-manager request fail (when the tls secret is unset): kubernetes/ingress-gce#388, we might want to hold off until gce is fixed to avoid confusion.

Closes #475

[consideRatio]

Woooooo!! I'm enthusiastic about this :D

[betatim]

Can you add the command that people should run with this yaml? Can't we integrate this into the z2jh chart?

[consideRatio]

We can I think - we can declare it as a dependency.

I've done that for the z2jh-chart itself and built on top of it.

For reference: https://github.com/consideRatio/z2jh-extended/blob/master/z2jh-extended/requirements.yaml

[clkao]

It is very likely the users already run or will run cert-manager cluster-wide, if using it. having it as a hard dependency and the issuer managed by us seems to complicate things.

[clkao]

@betatim just added the full command

[minrk]

I want to make sure that enabling letsencrypt remains the simplest and first recommended way to enable HTTPS. It shouldn't be moved to advanced topics. This https-enabling config that faces the users should ideally stay the same and the kube-lego->certmanager transition should be hidden from the user as much as possible.

What's the impediment to doing this now?

[betatim]

Addressing this is mandatory and in my eyes what needs to be done before we can merge this.

[clkao]

Happy to add the simple command back to the security section.

[jhamman]

Is there a specific blocker to moving this PR forward? We're running into some related challenges that we'd like to use cert-manager for.

[consideRatio]

kubucations's videos about cert-manager + lets encrypt was great for me to watch, that made me grasp a lot of things about cert-manager.

I have now tried cert-manager, and will summarize what I've picked up:

1. Have a cert-manager in your cluster, some namespace to look for issuer/certificate resources all over.
2. Have a issuer and certificate resource in some namespace
3. cert-manager creates a pod, updates an existing service to redirect traffic from a certain port to that pod in order to meet a "i can confirm i own this domain challenge" by lets encrypt.
4. When that is done, everything is returned to normal, and cert-manager will do it again when renewal is needed.

Conclusion:

1. our chart can provide an option to install the needed issuer / certificate with a configurable switch, allowing cert-manager to handle the TLS setup if cert-manager is running in the cluster.
2. i've learned that one can also optionally install chart dependencies

Suggestions:

1. We let the user declare if they would want a certificate/issuer created to be utilized by some cert-manager
2. We let the user declare if they would want a cert-manager chart installed as a dependency

---

@clkao @betatim @minrk @yuvipanda, what do you think? Especially curious regarding the optional chart dependency solution.

[clkao]

An optional dependency makes it easier for people to get started, by just editing a single values file, but that does mean we have to support it and follow upstream. IMO this is more worthwhile when the dependencies are very coupled, such as a database that you want to share secrets with.

For cert manager i think asking users to run an additional helm install and configure the issuer should be fine.

Another scenario: if we make this an optional dependency and do not support all options, and the users wish to migrate to a standalone cert-manager (for example if users decide to switch from http01 to dns01 domain validation), we'll need to figure out how to disconnect the helm dependency and preventing the old tls secret being deleted in the process.

[consideRatio]

@clkao it will be fine to loose the tls secret though right? It will just force a renewal ahead of time i think.

Adding cert-manager as an optional dependency, makes the users able to control it directly through provided config.yaml. So we would support all options, I think.

Example of config.yaml below:

proxy:
  secretToken: asdfasdfasdf

cert-manager:
  someCertManagerKey: andValue

[minrk]

I think an additional helm install is fine, but right now, it requires not just a helm install, but an additional raw kubectl create command for the issuer (which I don't see in the docs), with coordination of values that must occur in the cert-manager install, the issuer object, and the hub ingress (maybe?). I don't think that's clear enough yet for z2jh. The automatic https deployment should be as easy as possible, and not require referring to any advanced instructions. Users should be able to deploy a cluster with https, with only:

• specify hosts, lestencrypt email in config.yaml
• possible additional verbatim copy/paste, such as helm install stable/cert-manager

I'd really like to avoid any kubectl create calls, if we can, since that doesn't get cleaned up by things like helm del.

What about:

1. create the default issuer in our chart by default if https.hosts (harmless if unused, right?)
2. include the appropriate helm install command to deploy cert-manager to work with our issuer

Alternately, and I don't know what the up/downsides are, ship a cert-manager in our chart. What are the downsides, for instance, of having multiple cert-manager instances in a cluster? Is it a problem? If not, having the simples autohttps config deploy cert-manager, but make sure the chart works with a central cert-manager might be ideal.

[consideRatio]

@minrk to make cert-manager get a secret with TLS certificates etc in it for a certain domain, you need both an Issuer (or ClusterIssuer) and a Certificate resource

You can automate the creation of a Certificate resource if you have an Ingress resource with a tls field in its spec though, because cert-manager will find it and use it to create the Certificate.

To conclude so far, we need an Issuer of assume the usage of a predefined ClusterIssuer, and we need a Certificate or a Ingress resource with tls fields in its spec. Then we get ourselves a TLS-secret automatically created for us to consume.

If we declare an ingress object though, that specific cloud provider needs to create an associated LoadBalancer and be able to terminate TLS etc. These things are currently in flux, and many different cloud providers have implemented this to varying degree.

The GitLab helm chart deploys their own ingress controller (pods are spawned and running) in the namespace along with their deployment, that makes them work better on all kinds of cloud provider.

---

My ideas right now.

• Not happy about deploying the kubernetes official ingress controller called ingress-nginx if we need to have the proxy pod running anyhow.
• So then we should only need a cert-manager installed.
• Then our chart should support:
  1. Using a predefined ClusterIssuer
  2. Dynamic optional creation of an namespaced Issuer (think of these as representing an letsencrypt account + letsencrypt server)
  3. Dynamic optional creation of a namespaced Certificate
  4. Proxy to consume a secret of a given name (it already does!)
  5. The Ingress object to support custom annotation (it already does!)

Hmmm... Something like this.

[yuvipanda]

My general hope is to keep stringing kube-lego around until we get the traefik based proxy going - although I'll defer to others wether this is tenable. That should simplify everything here. In the meantime, additional docs on using cert-manager instead of our automatic issuer seems like a good thing to do?

[yuvipanda]

https://github.com/jupyterhub/outreachy/blob/master/ideas/traefik-jupyterhub-proxy.rst has info on the traefik proxy.

[stevegore]

I'm currently having issues with kube-lego with very straight-forward implementation of the chart. Any chance this PR could be merged soon? @clkao @consideRatio

[betatim]

The blocker is mentioned in this comment #755 (comment). The user facing config should remain as simple as it is today and be unchanged for the "just use letsencrypt" use case.

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/server-becomes-stale-in-a-matter-of-days-even-hours/1920/7

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/how-to-get-csr-for-ssl-cert/2505/2

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/how-to-get-csr-for-ssl-cert/2505/2

[supern8ent]

I just wanted to point out that running

helm install stable/cert-manager \
  --name cert-manager \
  --namespace cert-manager \
  --set ingressShim.defaultIssuerName=letsencrypt-prod \
  --set ingressShim.defaultIssuerKind=ClusterIssuer

deploys but also warns,

**This Helm chart is deprecated**.
All future changes to the cert-manager Helm chart should be made in the
official repository: https://github.com/jetstack/cert-manager/tree/master/deploy.
The latest version of the chart can be found on the Helm Hub: https://hub.helm.sh/charts/jetstack/cert-manager.

[supern8ent]

I tried the instructions in Advanced.md and it doesn't seem to be working. The cert-manager logs suggest that that part is working, but the system continues to serve on http, not https.

As I read the instructions, all that is needed in config.yaml is

proxy:
  secretToken: <token>
ingress:
  annotations:
    kubernetes.io/tls-acme: "true"
  tls:
   - hosts:
      - <hostname>
     secretName: jupyterhub-tls

I tried that. I also tried adding

https:
  enabled: true
  type: manual

And I tried adding

  enabled: true
  hosts:
    - <mydomain.com>

in the ingress section.

[abal09]

I tried the instructions in Advanced.md and it doesn't seem to be working. The cert-manager logs suggest that that part is working, but the system continues to serve on http, not https.

As I read the instructions, all that is needed in config.yaml is

proxy:
  secretToken: <token>
ingress:
  annotations:
    kubernetes.io/tls-acme: "true"
  tls:
   - hosts:
      - <hostname>
     secretName: jupyterhub-tls

I tried that. I also tried adding

https:
  enabled: true
  type: manual

And I tried adding

  enabled: true
  hosts:
    - <mydomain.com>

in the ingress section.

proxy:
  secretToken: <token>
ingress:
  enabled:true
  annotations:
    kubernetes.io/tls-acme: "true"
  tls:
   - hosts:
      - <hostname>
     secretName: jupyterhub-tls

Add enabled=true underneath the ingress

[akaszynski]

It should be noted that the --name flag is now mandatory for helm3

[yuvipanda]

We no longer use kube-lego, with #1539. I've personally never been able to get cert-manager to work despite trying several times - it feels very heavy for our common use case. How do people think we should proceed with this PR now?

[stevegore]

I use cert-manager in production across a number of sites and it's been great, especially in the last few releases.

[consideRatio]

While I think cert-manager is a great utility, I'm less happy about embedding a dependency on it in this helm chart, this is an evolved opinion. I have learned from experience that it is hard to install/upgrade CRDs and we would run into that using cert-manager as a dependency.

@yuvipanda have implemented a great replacent of kube-lego that is lightweight and suitable for anyone wanting to enable HTTPs quickly, that also doesn't exclude use of cert-manager if one have that utility in the cluster.

@clkao do you agree to close this PR? I'm very thankful for your work on it, and it made us have relevant discussion and helped me learn a lot more about this topic.

[consideRatio]

Closing as 0.9.0-beta.3 has the new autohttps with a traefik pod that acquires a cert and syncs it to a k8s secret.

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/jupyterhub-upgrade-steps-for-https-acmev2/4029/1