how-to-use-k8s-with-cri-containerd-and-kata.md

How to use Kata Containers and CRI (containerd plugin) with Kubernetes

• Requirements
• Install and configure containerd
• Install and configure Kubernetes
  • Install Kubernetes
  • Configure Kubelet to use containerd
  • Configure HTTP proxy - OPTIONAL
• Start Kubernetes
• Install a Pod Network
• Allow pods to run in the master node
• Create an untrusted pod using Kata Containers
• Delete created pod

This document describes how to set up a single-machine Kubernetes (k8s) cluster.

The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads.

For Kata Containers 1.5.0-rc2 and above, we will use containerd-shim-kata-v2 (short as shimv2 in this documentation) to launch Kata Containers. For the previous version of Kata Containers, the Pods are launched with kata-runtime.

Requirements

• Kubernetes, Kubelet, kubeadm
• containerd with cri plug-in
• Kata Containers

Note: For information about the supported versions of these components, see the Kata Containers versions.yaml file.

Install and configure containerd

First, follow the How to use Kata Containers and Containerd to install and configure containerd. Then, make sure the containerd works with the examples in it.

Install and configure Kubernetes

Install Kubernetes

• Follow the instructions for kubeadm installation.

• Check kubeadm is now available

  $ command -v kubeadm

Configure Kubelet to use containerd

In order to allow Kubelet to use containerd (using the CRI interface), configure the service to point to the containerd socket.

• Configure Kubernetes to use containerd

  $ sudo mkdir -p  /etc/systemd/system/kubelet.service.d/
  $ cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
  [Service]                                                 
  Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
  EOF

• Inform systemd about the new configuration

  $ sudo systemctl daemon-reload

Configure HTTP proxy - OPTIONAL

If you are behind a proxy, use the following script to configure your proxy for docker, Kubelet, and containerd:

$ services="
kubelet
containerd
docker
"

$ for service in ${services}; do

    service_dir="/etc/systemd/system/${service}.service.d/"
    sudo mkdir -p ${service_dir}

    cat << EOT | sudo tee "${service_dir}/proxy.conf"
[Service]
Environment="HTTP_PROXY=${http_proxy}"
Environment="HTTPS_PROXY=${https_proxy}"
Environment="NO_PROXY=${no_proxy}"
EOT
done

$ sudo systemctl daemon-reload

Start Kubernetes

• Make sure containerd is up and running

  $ sudo systemctl restart containerd
  $ sudo systemctl status containerd

• Prevent conflicts between docker iptables (packet filtering) rules and k8s pod communication

  If Docker is installed on the node, it is necessary to modify the rule below. See kubernetes/kubernetes#40182 for further details.

  $ sudo iptables -P FORWARD ACCEPT

• Start cluster using kubeadm

  $ sudo kubeadm init --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
  $ export KUBECONFIG=/etc/kubernetes/admin.conf
  $ sudo -E kubectl get nodes
  $ sudo -E kubectl get pods

Install a Pod Network

A pod network plugin is needed to allow pods to communicate with each other.

• Install the flannel plugin by following the Using kubeadm to Create a Cluster guide, starting from the Installing a pod network section.

• Create a pod network using flannel

  Note: There is no known way to determine programmatically the best version (commit) to use. See flannel-io/flannel#995.

  $ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

• Wait for the pod network to become available

  # number of seconds to wait for pod network to become available
  $ timeout_dns=420
  
  $ while [ "$timeout_dns" -gt 0 ]; do
      if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
          break
      fi
  
      sleep 1s
      ((timeout_dns--))
   done

• Check the pod network is running

  $ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )

Allow pods to run in the master node

By default, the cluster will not schedule pods in the master node. To enable master node scheduling:

$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-

Create an untrusted pod using Kata Containers

By default, all pods are created with the default runtime configured in CRI containerd plugin.

If a pod has the io.kubernetes.cri.untrusted-workload annotation set to "true", the CRI plugin runs the pod with the Kata Containers runtime.

• Create an untrusted pod configuration

  $ cat << EOT | tee nginx-untrusted.yaml
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-untrusted
    annotations:
      io.kubernetes.cri.untrusted-workload: "true"
  spec:
    containers:
    - name: nginx
      image: nginx
      
  EOT

• Create an untrusted pod

  $ sudo -E kubectl apply -f nginx-untrusted.yaml

• Check pod is running

  $ sudo -E kubectl get pods

• Check hypervisor is running

  $ ps aux | grep qemu

Delete created pod

$ sudo -E kubectl delete -f nginx-untrusted.yaml