terraform-kvm-kubespray

Set up HA Kubernetes cluster using KVM, Terraform and Kubespray.

Requirements

• Git
• Cloud-init
• Ansible >= v2.6
• Terraform >= v0.13.x
• KVM - Kernel Virtual Machine
• Libvirt provider - Setup guide is provided in docs.
• Internet connection on machine that will run VMs and on VMs

Note: for Terraform v0.12.x see this branch.

Getting Started

If you run into any troubles during installation process, please check troubleshooting page first.

Libvirt provider

If you haven't yet, install libvirt provider.

virtualenv

Install virtualenv using the following command:

python -m pip install --user virtualenv

SSH keys

Generate SSH keys, which will be used to access created VMs:

ssh-keygen

Follow the instructions to create SSH keys:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/your_username/.ssh/id_rsa): [1]
Enter passphrase (empty for no passphrase): [2]
Enter same passphrase again: [2]
...

[1] You will be asked to enter file in which to save the key. Default is /home/your_username/.ssh/id_rsa.

[2] When asked to enter a password, press ENTER twice to skip setting a password. DO NOT enter it, otherwise Terraform will fail to initialize a cluster.

Finally, you have to enter a location of SSH private key in vm_ssh_private_key field in terraform.tfvars file.

Cluster setup

Clone project and move to main directory:

git clone https://github.com/MusicDin/terraform-kvm-kubespray.git

cd terraform-kvm-kubespray

Change variables in terraform.tfvars file to fit your needs. Variables are set to work out of the box. Only required variables that are not set are:

• vm_image_source URL or path on file system to OS image,
• vm_distro a Linux distribution of OS image.

IMPORTANT: Review variables before initializing a cluster, as current configuration will create 8 VMs which are quite resource heavy!

Note: Script also supports deployment of single node cluster.

Execute terraform script:

# Initializes terraform project
terraform init

# Shows what is about to be done
terraform plan

# Runs/creates project
terraform apply

Note: Installation process can take up to 20 minutes based on a current configuration.

Test cluster

All configuration files will be generated in config/ directory, and one of them will be admin.conf which is actually a kubeconfig file.

Test your cluster by displaying all cluster's nodes:

kubectl --kubeconfig=config/admin.conf get nodes

Cluster management

Add worker to the cluster

In terraform.tfvars file add MAC and IP address for a new VM to vm_worker_macs_ips.

Execute terraform script to add a worker:

terraform apply -var 'action=add_worker'

Remove worker from the cluster

In terraform.tfvars file remove MAC and IP address of VM that is going to be deleted from vm_worker_macs_ips.

Execute terraform script to remove a worker:

terraform apply -var 'action=remove_worker'

Upgrade cluster

In terraform.tfvars file modify:

• k8s_kubespray_version and
• k8s_version.

Note: Before upgrading make sure Kubespray supports provided Kubernetes version.

Execute terraform script to upgrade a cluster:

terraform apply -var 'action=upgrade'

IMPORTANT: Do not skip releases when upgrading--upgrade by one tag at a time. For more information read Kubespray upgrades.

Destroy cluster

To destroy the cluster, simply run:

terraform destroy

More documentation

• Setup libvirt provider
• Load balancing
• Troubleshooting
• Examples:
  • Load balancing to ingress controller
  • Single node cluster deployment
  • MetalLB configuration examples

Related projects

If you are interested in installing kubernetes cluster on vSphere instead of KVM check this project.

Having issues?

In case you have found a bug, or some unexpected behaviour please open an issue.

If you need anything else, you can contact me on GitHub.

License

Apache License 2.0