Using Kubernetes resources right after creating the cluster

[captnCC]

Hello folks out there,
I'm currently playing around with this module to create clusters and deploy applications on it.

The straight forward path seemed to be to use the cluster module together with the Kubernetes provider.

So I take the retuned kubeconfig from the cluster module and put it into the Kubernetes provider:

provider "kubernetes" {
  host                   = module.cluster.kubeconfig.host
  client_certificate     = module.cluster.kubeconfig.client_certificate
  client_key             = module.cluster.kubeconfig.client_key
  cluster_ca_certificate = module.cluster.kubeconfig.cluster_ca_certificate
}

This works well after the cluster is created, but on the first run most resources will fail with an cert error
Failed to create deployment: Post "https://142.132.239.194:6443/apis/apps/v1/namespaces/default/deployments": x509: certificate is valid for 10.255.0.101, 10.43.0.1, 127.0.0.1, ::1, not 142.132.239.194.
Weirdly enough in some instances a part of the resources would succeed while the rest fails. Even adding a delay of 3 minutes before creating the resources didn't resolve that issue.

As base of the infrastructure I needed some additional components to be installed via helm. So I take the same config and put it in the helm provider:

provider "helm" {
  kubernetes {
    host                   = module.cluster.kubeconfig.host
    client_certificate     = module.cluster.kubeconfig.client_certificate
    client_key             = module.cluster.kubeconfig.client_key
    cluster_ca_certificate = module.cluster.kubeconfig.cluster_ca_certificate
  }
}

And to my surprise this just works fine.

Has anyone faced the same issue and what is your workaround? And the more general question: Keep infra and application in the same place or split it up in 2 separated terraform workflows?

Update

By playing around with the order and dependencies of resources I found that generally the first request(s) to the Kubernetes API fail both with Kubernetes or Helm resources.

[mysticaltech]

@captnCC This regularly happens to me when I first try to kubectl into the cluster the first time. Glad you bumped into this issue so we can fix it. There is a param in k3s install config called tls-san. I believe that if we set it to the external IP of the node, your issue will disappear.

I just checked and it is already being used when use_control_plane_lb is set to true (see init.tf and control_planes.tf), so that should fix your issue out of the box. It basically creates an LB just to load balance calls to the control-plane, and during the implementation of that feature tls-san was correctly set, but not for the normal use case without that LBs.

The changes need to happen in both init.tf and control_planes.tf, where the param is set to an array of both the public and private IPs of the node in question. Without breaking the use_control_plane_lb logic. So if else..

[mysticaltech]

FYI have converted this to an issue to be fixed, see #325.