Support for External-DNS

[codeagencybe]

@mysticaltech

Do you happen to know if and how it could add support for external DNS from https://github.com/kubernetes-sigs/external-dns ?

This can help automating the DNS part for the custom domains set, and hence improve the SSL part from cert manager via DNS01 resolved (instead of HTTP01).
https://github.com/ixoncloud/cert-manager-webhook-cloudns

[mysticaltech]

@codeagencybe I try to limit the number of external packages to a strict minimum, as the goal of this project is just to deploy an optimized cluster. The only exception was Longhorn (as it requires specific OS packages to work) and Rancher (which in turn requires cert-manager) as it allows for the installation of many other packages.

The best way is to install this after the cluster is up, either by using Rancher itself, or the helm cli or by applying a HelmChart definition as explained here, and used as a template here (for inspiration on the format).

If you use the latter option, you can also pass this via kustomization if you want to add a bunch of extra helm packages, all automated following the terraform apply -auto-approve command success.

[codeagencybe]

@mysticaltech

Sure that is understandable. I was just looking for a way to easy link the base domain and rancher domain directly to my DNS provider.
In case we want to spin up a new cluster, it's easier because you don't know in advance which IP Hetzner is giving you for the LB so you can't pre-configure your DNS to point A record to the IP.
Having this in place at the time of deploying, automatically, would make it a bit easier.

I already found some terraform module for my DNS provider, so I'm looking into this route.
Then I can add just another TF module, handle config in there and call it at some point during the process when the LB is generated so it links the A record with my DNS provider (ClouDNS)

[mysticaltech]

@codeagencybe Awesome; if you can make this work in a configurable way, I would be happy to review a PR if that's a route you would like to take.

Or share your extra module in the examples folder if you are willing to make it open source, along with a small mention in the Readme in the examples section, a PR for that would be great too!! 😎

[mysticaltech]

@codeagencybe Also moving this one to discussions, as that's where I prefer to have enhancement proposals that are not super urgent.

[captnCC]

I'm right now in the process of setting up a similar setup, but using Hetzner's DNS Service.
This includes using the terraform resources from timohirt/hetznerdns to setup the A records (and AAAA when the output of the LB IPv6 address is released), installing the webhook service to resolve DNS01 challenges against the Hetzner DNS via its helm chart and installing the ClusterIssuer.

But at this point emerges for me the question, what the best way to install the additional helm charts is. As main components like the cert-manager are installed in the deployment by deploying the yaml files to the server, it would seem logical to do the same with the webhook provider.
Right now I deploy them via the Kubernetes provider in terraform.

Is it of interest to provide a managed "all-in-one"-solution in the Hetzner infrastructure?

[codeagencybe]

I'm also still puzzled about what the correct way is for dealing with DNS automation.
There is a project in Kubernetes about external-dns, but it seems this is also not quite what I expected so far.
And then there is also the SSL part via Cert Manager, you can't request certs if your DNS is not properly configured.

I'm using ClouDNS myself (https://cloudns.net), not to advertise for them, but they have an integration ready made for Cert Manager so it can deal with DNS01 resolved and auto-create the TXT records for certification verification.

I was hoping that external-dns would solve this problem also for general DNS stuff like A records, CNAME records, ...

At this moment, I'm probably just gonna go with a custom script that I fire off at a certain "early" event during deployment so the required A/CNAME/.... records are created in ClouDNS via API.
And then let Cert Manager deal with their part of the puzzle. But I don't know if this is a resilient solution either.

Ultimately, I just want a new application to be "ready" automatically after deploying from Rancher including the DNS stuff required without me having to fiddle with DNS settings manually each time.
If somebody knows a better way, I'm all ears but so far I think a custom API script for DNS automation is probably the easiest and faster solution.

[captnCC]

Just to recap on your deployment strategy, you build the cluster via terraform and later deploy multiple applications via Rancher in the form of Helm charts or similar?

If that's the case, a combination of external-dns and cert-manager should do the trick. In terraform you would need to install the external-dns via Helm using as a helm_release and add your ClusterIssuer like described in the docs. Just as a side note, the ClusterIssuer needs to be created via a local helm chart as well because the Kubernetes provider seems to be broken when the cluster is setup the first time.

[codeagencybe]

Yep

1. we deploy a cluster with Terraform incuding longhorn + Rancher
   The DNS/domain for this I do manually to get "mycluster.mydomain.tld" and "rancher.mydomain.tld"

2. applications we will deploy via Rancher (either web ui or yaml or helm whatever is easiest)
   My longterm goal is to allow new clients to just fill out an order form to start a subscription for eg Wordpress, Magento, ...
   In the form they have to type their desired domain to be linked to their new Wordpress website (or something else)
   Once they complete the form, I already can register their domain as a new DNS zone in ClouDNS. I already have this automated since few years.
   But to make it more and better then I can call the Kubernetes API which will deploy the app and handle the domain stuff so it points automatically to the correct LB service and then handle the Let's encrypt certificates.

[mysticaltech]

Hey folks, interesting discussion! For me personally, I use Cloudflare via external-dns to automate all DNS matters; it works quite well. You just need to configure your Cloudflare API token via a config map and your records will automatically be taken care of if a resource like the ingress definitions has the right annotations, letting external-dns/Cloudflare know it needs to map it. There are other discussions on the subject in this repo, just search for it.

About installing other helm charts automatically. The easiest is just to create HelmChart definition yaml file (if you do not know what that is, just search for it too in this repo, it was explained at multiple locations). It's rancher's way of dealing with helm charts, and what is used by default in k3s.

AND you could move these files to a special folder on the "init" node, and have k3s automatically install them (during it's own installation). Basically, the logic for that would be:

1. (By the user) Pre-cluster setup, create a directory in your root folder, where your kube.tf lives, name it extra-manifests.
2. (By the code) During the init, before k3s is installed, with terraform - copy all yaml files from extra-manifest to /var/lib/rancher/k3s/server/manifests on the node via the File provisioner (if extra-manifests directory exists and is not empty). And probably before that, if not already present, it would be good to create the folders in the path of the destination either via a remote-exec like "mkdir -p" or a pure terraform equivalent if available.

Please see more details on that auto-deploy folder here, basically, it supports both normal yaml manifests and HelmCharts definitions. And according to this, it can even be done after k3s install. Hence removing the need to create this directory /var/lib/rancher/k3s/server/manifests before copying. It seems that once k3s is running, it will regularly scan this folder for new files to deploy.

PR is welcome to add support for the extra-manifest folder!

---

As for full support of DNS out of the box along with SSL webhooks. Why not @captnCC, timohirt/hetznerdns seems perfect for the job! As long as it's off by default and configurable via variables. Again, good PRs with clear code are most welcome! :)

_Please note that if you need to change the initial Helm values of cert-manager, we can add support for a cert-manager_values.yaml file in the root folder, the same as was done for Rancher, Longhorn, Nginx, and Cilium. Just let me know; I have done it so many times that I should be able to add support for this quickly.

In any case, I will surely add support for that in the next release, as it's always good to give options. But if you feel adventurous and want to observe how it was done for the other HelmCharts above, please shoot a PR, too (as much as possible, one PR per feature, please)!_

[captnCC]

@mysticaltech To keep this thread on topic, I branched out both idea into dedicated discussion #328 and #329