Setting up our own Kubernetes/Containers based Private Cloud

Tutorials

• How to Install and Deploy Kubernetes on Ubuntu 16.04
• How To Create a Kubernetes Cluster Using Kubeadm on Ubuntu 16.04
• Bare-metal considerations : https://kubernetes.github.io/ingress-nginx/deploy/baremetal/

0. Preparation

• Change Hostname in Ubuntu 16.04 Without Restart
• Set up key-based authentication in SSH
• Enable sudo without password in Ubuntu/Debian
• Install Ansible - an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code
  • Configuration Management 101: Writing Ansible Playbooks
  • Getting Started With Kubernetes Operators (Ansible Based)

1. Install Containerd

Containerd replaces docker-ce as the container runtime in Kubernetes

• IBM Cloud Kubernetes Service Supports containerd
• Kubernetes Containerd Integration Goes GA

Steps

• Install containerd as CRI runtime
• Change default cgroup driver to systemd
  • Set plugins.cri.systemd_cgroup = true in /etc/containerd/config.toml
  • Manually configure the cgroup driver used by kubelet

2. Setup Kubernetes using kubeadm

• Install kubeadm, kubelet and kubectl
• Create a single control-plane cluster with kubeadm
• Kubernetes Internals
  • Container Networking From Scratch - Kristen Jacobs, KubeCon + CloudNativeCon Seattle 2018 - Slides here
  • Kubernetes Deconstructed: Understanding Kubernetes by Breaking It Down - Carson Anderson, KubeCon + CloudNativeCon SanDiego 2017 - Slides here - Detailed session here - Material here
  • Make Ingress-Nginx Work for you, and the Community - Fernando Diaz, KubeCon + CloudNativeCon SanDiego 2018

3. Enable Dashboard with Monitoring

• Deploy Kubernetes Dashboard https://github.com/kubernetes/dashboard
• Container Resource Monitoring
  • Prometheus - Overview, Components, Architecture
    • Intro + Deep Dive: Prometheus - Slides here
    • Monitoring Kubernetes Clusters with Prometheus - Related blog here
  • Prometheus Instrumentation
    • Monitor machines using Prometheus Node exporter
    • Monitor network devices using Prometheus SNMP exporter
    • Monitor JVM applications via JMX using Prometheus JMX Exporter
    • Monitoring Docker container metrics using cAdvisor
  • Install Prometheus on Kubernetes
    • An Introduction to Helm - Slides here
    • Kubernetes native deployment and management of Prometheus using Prometheus Operator
    • Helm chart to bootstrap prometheus-operator deployment on a Kubernetes cluster

4. Distributed File System

• Ceph - a distributed storage solution for block storage, object storage, and shared filesystems
  • Ceph Tech Talk - Intro to Ceph - Slides here
• Run Ceph in Kubernetes using Rook
  • Introduction to Rook: KubeCon + CloudNativeCon Seattle 2018 - Slides here
  • Deep Dive: Rook + Ceph: KubeCon + CloudNativeCon Seattle 2018 - Slides here
  • Rook-Ceph Deep Drive: KubeCon + CloudNativeCon San Diego 2019. From 22:46 to 43:00. Slides here.
• Ceph Storage Prerequisites
  • Use a file as a Linux block device
  • How to create virtual block device (loop device/filesystem) in Linux
  • Ceph Mimic | Using loop devices as OSD
  • Allow loop device as a LVM backend
  • Ceph Volume Prepare
  • Using the ceph-volume utility to deploy OSDs
• Setup of a Ceph cluster
  • rook/common.yaml
  • rook/operator.yaml
  • rook/cluster.yaml - Ceph Cluster CRD
• Setting up consumable storage - Block Devices, Shared Filesystem and Object Storage
  • Object Storage
  • Shared Filesystem
  • Block Storage
• Rook Ceph Common Issues
  • Rook toolbox for rook debugging and testing
  • OSD pods are not created on my devices
  • skipping device loop0: Failed to complete 'lsblk /dev/loop0': exit status 1
  • OSD Configuration Settings - storeType: filestore or bluestore
  • GParted - a free partition editor for graphically managing your disk partitions.

5. Useful Helm charts

Cloud Native Computing Foundation (CNCF) Survey 2019 Report

• https://hub.helm.sh/charts/bitnami/nginx
• https://hub.helm.sh/charts/nginx/nginx-ingress
• https://hub.helm.sh/charts/cloudposse/nfs-provisioner
• https://hub.helm.sh/charts/ibm-charts/ibm-rook-rbd-cluster
• https://hub.helm.sh/charts/stable/jenkins
• https://github.com/IBM/charts/tree/master/stable/ibm-istio
• https://gitlab.com/gitlab-org/charts/gitlab

Other Useful References

• Using Variables in Ansible such as {{ansible_distribution_release}}
• Use SSH Keys With PuTTY On Windows
• Ansible Modules Documentation
• Difference between systemctl and service commands
• How to check if port is in use on Linux or Unix
• Private Address Space reserved by Internet Assigned Numbers Authority (IANA)
• What is the difference between Docker, LXD, and LXC
• How to Install and Configure VNC on Ubuntu 16.04

Troubleshooting

• CoreDNS pods have CrashLoopBackOff or Error state
  • Troubleshooting Loops In Kubernetes Clusters
  • Ubuntu 16.04 Server Guide - Networking Configuration
  • resolvconf — a framework for managing multiple DNS configurations
  • dnsmasq - A lightweight DHCP and caching DNS server
  • nameserver 127.0.1.1 in resolv.conf won't go away!
  • Other related references
    • IP Addresses, Subnet Masks, and Default Gateways
    • NAT Configuration Primer
    • Kubernetes CoreDNS in CrashLoopBackOff
    • Disabling DNSMASQ via Network Manager on Ubuntu 16.04+
    • Disabling DNSMASQ as your local DNS server in Ubuntu
    • How do I set my DNS when resolv.conf is being overwritten?
• How to apt-delete-repository?
• Tear down / clean up Kubernetes cluster
• Use crictl to manually list, stop, remove pods
  • List pods $ sudo crictl -r /var/run/containerd/containerd.sock pods
  • Stop one or more running pods $ sudo crictl -r /var/run/containerd/containerd.sock stopp <pod-id>
  • Remove one or more pods $ sudo crictl -r /var/run/containerd/containerd.sock rmp <pod-id>
• sudo apt-get update failing - “could not open” list file due to “permission denied”