diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e73f49fcdf..a11435aef2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,7 @@ Read the following guide if you're interested in contributing to Ingress. [Make Ingress-Nginx Work for you, and the Community](https://youtu.be/GDm-7BlmPPg) from KubeCon Europe 2018 is a great video to get you started!! -Note that this guide refers to contributing to actual sources of the repository. If you interested in contributing through issue triaging, have a look at [this guide](./ISSUE_TRIAGE.md). +Note that this guide refers to contributing to actual sources of the repository. If you interested in contributing through issue triaging, have a look at [this guide](./ISSUE_TRIAGE.md). ## Contributor License Agreements @@ -19,7 +19,7 @@ Follow either of the two links above to access the appropriate CLA and instructi ## Finding Issues That Need Help -If you're new to the project and want to help, but don't know where to start, we have a semi-curated list of issues that should not need deep knowledge of the system. [Have a look and see if anything sounds interesting](https://github.com/kubernetes/ingress-nginx/issues?utf8=%E2%9C%93&q=is%3Aopen%20is%3Aissue%20label%3A%22help+wanted%22). +If you're new to the project and want to help, but don't know where to start, we have a semi-curated list of issues that should not need deep knowledge of the system. [Have a look and see if anything sounds interesting](https://github.com/kubernetes/ingress-nginx/issues?utf8=%E2%9C%93&q=is%3Aopen%20is%3Aissue%20label%3A%22help+wanted%22). Alternatively, search for the label [`triage-accepted`](https://github.com/kubernetes/ingress-nginx/issues?q=is%3Aopen+is%3Aissue+label%3Atriage%2Faccepted+) if you have some experience with ingress-nginx. Note, that it could make sense to grab issues with higher priority first. @@ -34,9 +34,9 @@ All changes must be code reviewed. Coding conventions and standards are explaine ### Merge Approval -Ingress Nginx collaborators may add "/lgtm" (Looks Good To Me) to indicate that a PR is acceptable. Any change requires at least one LGTM. No pull requests can be merged until at least one Ingress Nginx collaborator signs off with an LGTM. Adding the "/lgtm" comment result in the prow bot adding the `lgtm` label. Note that a pull request still needs an `approve` label from one of the owners. +Ingress Nginx collaborators may add "/lgtm" (Looks Good To Me) to indicate that a PR is acceptable. Any change requires at least one LGTM. No pull requests can be merged until at least one Ingress Nginx collaborator signs off with an LGTM. Adding the "/lgtm" comment result in the prow bot adding the `lgtm` label. Note that a pull request still needs an `approve` label from one of the owners. -Reviewers or members who want to become reviewers according to the [k8s membership ladder](https://github.com/kubernetes/community/blob/master/community-membership.md), could actively search for [pull requests that need a review](https://github.com/kubernetes/ingress-nginx/pulls?q=is%3Aopen+is%3Apr+label%3Atriage%2Faccepted). +Reviewers or members who want to become reviewers according to the [k8s membership ladder](https://github.com/kubernetes/community/blob/master/community-membership.md), could actively search for [pull requests that need a review](https://github.com/kubernetes/ingress-nginx/pulls?q=is%3Aopen+is%3Apr+label%3Atriage%2Faccepted). ## Support Channels @@ -47,3 +47,6 @@ Whether you are a user or contributor, official support channels include: - Post: [Kubernetes Forum](https://discuss.kubernetes.io) Before opening a new issue or submitting a new pull request, it's helpful to search the project - it's likely that another user has already reported the issue you're facing, or it's a known issue that we're already aware of. + +## New Contributor Tips +If you're a new contributor, you can follow the [New Contributor Tips guide](NEW_CONTRIBUTOR.md) diff --git a/NEW_CONTRIBUTOR.md b/NEW_CONTRIBUTOR.md new file mode 100644 index 0000000000..816d057786 --- /dev/null +++ b/NEW_CONTRIBUTOR.md @@ -0,0 +1,850 @@ +## New Contributor Tips + +Welcome to the Ingress Nginx new contributor tips. +This guide briefly outlines the necessary knowledge & tools, required to start working on Ingress-NGINX Issues. + +### Prerequisites +- Basic understanding of linux +- Familiarity with the command line on linux +- OSI Model(Links below) + +### Introduction +It all starts with the OSI model... +> The Open Systems Interconnection (OSI) model describes seven layers that computer systems use to communicate over a network. It was the first standard model for network communications, adopted by all major computer and telecommunication companies + +![Describes the 7 Layers of the OSI Model](https://i.imgur.com/qF0KjBq.png) + +#### Reading material for OSI Model +[OSI Model CertificationKits](https://www.certificationkits.com/cisco-certification/cisco-ccna-640-802-exam-certification-guide/cisco-ccna-the-osi-model/) + +### Approaching the problem + + +Not everybody knows everything. But the factors that help are a love/passion for this to begin. But to move forward, its the approach and not the knowledge that sustains prolonged joy, while working on issues. If the approach is simple and powered by good-wishes-for-community, then info & tools are forthcoming and easy. + +Here we take a bird's eye-view of the hops in the network plumbing, that a packet takes, from source to destination, when we run `curl`, from a laptop to a nginx webserver process, running in a container, inside a pod, inside a Kubernetes cluster, created using `kind` or `minikube` or any other cluster-management tool. + +### [Kind](https://kind.sigs.k8s.io/) cluster example on a Linux Host + +#### TL;DR +The destination of the packet from the curl command, is looked up, in the `routing table`. Based on the route, the the packet first travels to the virtual bridge `172.18.0.1` interface, created by docker, when we created the kind cluster on a laptop. Next the packet is forwarded to `172.18.0.2`(See below on how we got this IP address), within the kind cluster. The `kube-proxy` container creates iptables rules that make sure the packet goes to the correct pod ip in this case `10.244.0.5` + +Command: +``` +# docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +230e7246a32c kindest/node:v1.24.1 "/usr/local/bin/entr…" 2 weeks ago Up 54 seconds 127.0.0.1:38143->6443/tcp kind-control-plane + +# docker inspect kind-control-plane -f '{{ .NetworkSettings.Networks.kind.IPAddress }}' +172.18.0.2 + +``` + + + +If this part is confusing, you would first need to understand what a [bridge](https://tldp.org/HOWTO/BRIDGE-STP-HOWTO/what-is-a-bridge.html) is and what [docker network](https://docs.docker.com/network/) is. + + + +#### The journey of a curl packet. +Let's begin with creating a [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) Cluster on your laptop +``` +# kind create cluster +``` +This will create a cluster called `kind`, to view the clusters type +``` +# kind get clusters    +kind +``` +Kind ships with `kubectl`, so we can use that to communicate with our clusters. +``` +# kubectl get no -o wide    +NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +kind-control-plane Ready control-plane 5d23h v1.24.1 172.18.0.2 Ubuntu 21.10 5.18.12-arch1-1 containerd://1.6.4 +``` +Kind creates a cluster using docker container as nodes, it does this using [containerd](https://containerd.io/) within the docker container. +The concept of Docker in Docker is very important here. + +To start with simply create a nginx deployment using `kubectl`. +``` +# kubectl create deployment nginx --image nginx:alpine --port=80 +deployment.apps/nginx created +``` +Then we expose this as a NodePort Service. +``` +# kubectl expose deployment/nginx --type=NodePort +service/nginx-new exposed +``` +Command: Now we can see that the service has been exposed. +``` +# kubectl get svc -o wide +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR +nginx NodePort 10.96.176.241 80:32329/TCP 4d8h app=nginx +``` +Output Relevance: From the above output, we can see that our nginx pod is being exposed as the `NodePort` service type, and now we can curl the Node IP `172.18.0.2` with the exposed port `32329` + +Command: The pod has an IP as shown below +``` +# kubectl get po -o wide   +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +nginx-6c8b449b8f-pdvdk 1/1 Running 1 (32h ago) 4d8h 10.244.0.5 kind-control-plane +``` + +Command: We can use `curl` on the laptop to view the nginx container that is running on port `32329`. + +``` +# curl 172.18.0.2:32329 + + + + +Welcome to nginx! + + + +

Welcome to nginx!

+

If you see this page, the nginx web server is successfully installed and +working. Further configuration is required.

+ +

For online documentation and support please refer to +nginx.org.
+Commercial support is available at +nginx.com.

+ +

Thank you for using nginx.

+ + +``` +Now, we can check the ip interfaces as well subnets for our system is connected to: + +``` +$ ifconfig +ethbr0: flags=4163 mtu 1500 + inet 192.168.31.9 netmask 255.255.255.0 broadcast 192.168.31.255 + inet6 fe80::7530:9ae5:3e8d:e45a prefixlen 64 scopeid 0x20 + ether 2e:90:b3:e8:52:5b txqueuelen 1000 (Ethernet) + RX packets 31220566 bytes 44930589084 (41.8 GiB) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 18104006 bytes 1757183680 (1.6 GiB) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 + +br-2fffe5cd5d9e: flags=4163 mtu 1500 + inet 172.18.0.1 netmask 255.255.0.0 broadcast 172.18.255.255 + inet6 fc00:f853:ccd:e793::1 prefixlen 64 scopeid 0x0 + inet6 fe80::42:12ff:fed3:8fb0 prefixlen 64 scopeid 0x20 + inet6 fe80::1 prefixlen 64 scopeid 0x20 + ether 02:42:12:d3:8f:b0 txqueuelen 0 (Ethernet) + RX packets 3547 bytes 414792 (405.0 KiB) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 6267 bytes 8189931 (7.8 MiB) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 +docker0: flags=4099 mtu 1500 + inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 + inet6 fe80::42:a2ff:fe09:5edb prefixlen 64 scopeid 0x20 + ether 02:42:a2:09:5e:db txqueuelen 0 (Ethernet) + RX packets 14 bytes 2143 (2.0 KiB) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 40 bytes 6406 (6.2 KiB) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 +``` +From the above output we can see that, there are two bridges connected to our systems network interface,one is the docker default bridge`docker0` and the other created by kind +`br-2fffe5cd5d9e`. + +Since kind creates nodes as containers, this is easily accessible via `docker ps`. +``` +$ docker ps + +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +230e7246a32c kindest/node:v1.24.1 "/usr/local/bin/entr…" 6 days ago Up 33 hours 127.0.0.1:38143->6443/tcp kind-control-plane +``` +If we do a docker `exec` we can enter the container, we can also see the network interfaces within the container. +``` +# docker exec -it 230e7246a32c bash + +# root@kind-control-plane:/# ip a +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever + inet6 ::1/128 scope host + valid_lft forever preferred_lft forever +2: vethdb0d1da1@if2: mtu 1500 qdisc noqueue state UP group default + link/ether a2:a1:ce:08:d2:39 brd ff:ff:ff:ff:ff:ff link-netns cni-ddc25710-030a-cc05-c600-5a183fae01f7 + inet 10.244.0.1/32 scope global vethdb0d1da1 + valid_lft forever preferred_lft forever +3: veth4d76603f@if2: mtu 1500 qdisc noqueue state UP group default + link/ether 9a:9b:6b:3e:d1:53 brd ff:ff:ff:ff:ff:ff link-netns cni-f2270000-8fc8-6f89-e56b-4759ae10a084 + inet 10.244.0.1/32 scope global veth4d76603f + valid_lft forever preferred_lft forever +4: vethcc2586d6@if2: mtu 1500 qdisc noqueue state UP group default + link/ether 52:f9:20:63:62:a2 brd ff:ff:ff:ff:ff:ff link-netns cni-97e337cd-1322-c1fa-7523-789af94f397f + inet 10.244.0.1/32 scope global vethcc2586d6 + valid_lft forever preferred_lft forever +5: veth783189a9@if2: mtu 1500 qdisc noqueue state UP group default + link/ether ba:e1:55:1f:6f:12 brd ff:ff:ff:ff:ff:ff link-netns cni-90849001-668a-03d2-7d9e-192de79ccc59 + inet 10.244.0.1/32 scope global veth783189a9 + valid_lft forever preferred_lft forever +6: veth79c98c12@if2: mtu 1500 qdisc noqueue state UP group default + link/ether 22:05:55:c7:86:e9 brd ff:ff:ff:ff:ff:ff link-netns cni-734dfac9-9f70-ab33-265b-21569d90312a + inet 10.244.0.1/32 scope global veth79c98c12 + valid_lft forever preferred_lft forever +7: veth5b221c83@if2: mtu 1500 qdisc noqueue state UP group default + link/ether 92:3f:04:54:72:5a brd ff:ff:ff:ff:ff:ff link-netns cni-d8f6666b-1cfb-ef08-4bf8-237a7fc32da2 + inet 10.244.0.1/32 scope global veth5b221c83 + valid_lft forever preferred_lft forever +8: vethad630fb8@if2: mtu 1500 qdisc noqueue state UP group default + link/ether 32:78:ec:f6:01:ea brd ff:ff:ff:ff:ff:ff link-netns cni-6cb3c179-cb17-3b81-2051-27231c44a3c4 + inet 10.244.0.1/32 scope global vethad630fb8 + valid_lft forever preferred_lft forever +9: veth573a629b@if2: mtu 1500 qdisc noqueue state UP group default + link/ether e2:57:f8:c9:bc:94 brd ff:ff:ff:ff:ff:ff link-netns cni-d2dbb903-8310-57b4-7ba4-9f353dbc79dc + inet 10.244.0.1/32 scope global veth573a629b + valid_lft forever preferred_lft forever +10: eth0@if11: mtu 1500 qdisc noqueue state UP group default + link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0 + valid_lft forever preferred_lft forever + inet6 fc00:f853:ccd:e793::2/64 scope global nodad + valid_lft forever preferred_lft forever + inet6 fe80::42:acff:fe12:2/64 scope link + valid_lft forever preferred_lft forever +11: vethd7368e27@if2: mtu 1500 qdisc noqueue state UP group default + link/ether 8a:74:ec:f6:d6:c9 brd ff:ff:ff:ff:ff:ff link-netns cni-7c7eb9cd-bbb1-65b0-0480-b8f1265f2f36 + inet 10.244.0.1/32 scope global vethd7368e27 + valid_lft forever preferred_lft forever +12: veth7cadbf2b@if2: mtu 1500 qdisc noqueue state UP group default + link/ether 12:48:10:b7:b8:f5 brd ff:ff:ff:ff:ff:ff link-netns cni-b39e37b5-1bc8-626a-a553-a0be2f94a117 + inet 10.244.0.1/32 scope global veth7cadbf2b + valid_lft forever preferred_lft forever + +``` +When we run `curl 172.18.0.2:32329` on the laptop it first needs to figure out where `172.18.0.2`, to do this it refers to the host routing table. +``` +sudo netstat -rn    main  +Kernel IP routing table +Destination Gateway Genmask Flags MSS Window irtt Iface +0.0.0.0 192.168.31.1 0.0.0.0 UG 0 0 0 ethbr0 +172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 +172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-2fffe5cd5d9e +172.19.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-be5b544733a3 +192.168.31.0 0.0.0.0 255.255.255.0 U 0 0 0 ethbr0 +192.168.31.0 0.0.0.0 255.255.255.0 U 0 0 0 ethbr0 +192.168.39.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr2 +192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 +``` +Output Relevance: From the above output, you can see that the `iface`(Interface) for `172.18.0.0` is `br-2fffe5cd5d9e`, which means traffic that needs to go to `172.18.0.0` will go through `br-2fffe5cd5d9e` which is created by docker for the kind container (this is the node in case of kind cluster). + +Now we need to understand how the packet travels from the container interface to the pod with IP `10.244.0.5`. The component that handles this is called kube-proxy + +So what exactly is [kube-proxy](https://kubernetes.io/docs/concepts/overview/components/#kube-proxy): +> Kube-Proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept. +kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster + +So, as we can see that kube proxy handles the network rules required to aid the communication to the pods, we will look at the [iptables](https://linux.die.net/man/8/iptables) +> `iptables` is a command line interface used to set up and maintain tables for the Netfilter firewall for IPv4, included in the Linux kernel. The firewall matches packets with rules defined in these tables and then takes the specified action on a possible match. Tables is the name for a set of chains + +Command: +``` +# iptables -t nat -L PREROUTING -n +Chain PREROUTING (policy ACCEPT) +target prot opt source destination +KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */ +DOCKER_OUTPUT all -- 0.0.0.0/0 172.18.0.1 +CNI-HOSTPORT-DNAT all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL +``` + +``` +# iptables-save | grep PREROUTING +-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES +``` +Output Relevance: +> -A: append new iptable rule +> -j: jump to the target +> KUBE-SERVICES: target + +> The above output appends a new rule for PREROUTING which every network packet will go through first as they try to access any kubernetes service + + +What is `PREROUTING` in iptables? +>PREROUTING: This chain is used to make any routing related decisions before (PRE) sending any packets + +To dig in further we need to go to the target, `KUBE-SERVICES` for our nginx service. +``` +# iptables -t nat -L KUBE-SERVICES -n| grep nginx +KUBE-SVC-2CMXP7HKUVJN7L6M tcp -- 0.0.0.0/0 10.96.176.241 /* default/nginx cluster IP */ tcp dpt:80 +``` +Command: +``` +# iptables -t nat -L KUBE-SVC-2CMXP7HKUVJN7L6M -n +Chain KUBE-SVC-2CMXP7HKUVJN7L6M (2 references) +target prot opt source destination +KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.96.176.241 /* default/nginx cluster IP */ tcp dpt:80 +KUBE-SEP-4IEO3WJHPKXV3AOH all -- 0.0.0.0/0 0.0.0.0/0 /* default/nginx -> 10.244.0.5:80 */ + +# iptables -t nat -L KUBE-MARK-MASQ -n +Chain KUBE-MARK-MASQ (31 references) +target prot opt source destination +MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000 + +# iptables -t nat -L KUBE-SEP-4IEO3WJHPKXV3AOH -n +Chain KUBE-SEP-4IEO3WJHPKXV3AOH (1 references) +target prot opt source destination +KUBE-MARK-MASQ all -- 10.244.0.5 0.0.0.0/0 /* default/nginx */ +DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/nginx */ tcp to:10.244.0.5:80 +``` + + +``` +iptables-save | grep 10.96.176.241 + +-A KUBE-SERVICES -d 10.96.176.241/32 -p tcp -m comment --comment "default/nginx cluster IP" -m tcp --dport 80 -j KUBE-SVC-2CMXP7HKUVJN7L6M +-A KUBE-SVC-2CMXP7HKUVJN7L6M ! -s 10.244.0.0/16 -d 10.96.176.241/32 -p tcp -m comment --comment "default/nginx cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ +``` + +As you can see the rules added by `kube-proxy` helps the packet reach to the destination service. + +### Minikube KVM VM Example on Linux + +#### TL;DR +Now we look at the curl packet journey on minikube. The `routing table` is looked up to know the destination of the curl packet. The packet then first travels to the virtual bridge `192.168.39.1`, created by minikube kvm2 driver, when we created the minikube cluster, on a linux laptop. Then this packet is forwarded to `192.168.39.57`, within the minikube VM. We have docker containers running in the VM. Among them, the `kube-proxy` container creates iptables rules that make sure the packet goes to the correct pod ip, in this case `172.17.0.4`. + + +To begin with the minikube example, we first need to create a minikube cluster on a linux laptop. In this example I'll be using the `kvm2` driver option for `minikube start` command, as default. + +``` +minikube start +😄 minikube v1.26.0 on Arch "rolling" +🆕 Kubernetes 1.24.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.24.2 +✨ Using the kvm2 driver based on existing profile +👍 Starting control plane node minikube in cluster minikube +🏃 Updating the running kvm2 "minikube" VM ... +🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.12 ... + ▪ kubelet.housekeeping-interval=5m +🔎 Verifying Kubernetes components... + ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 + ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 + ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1 + ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 +🔎 Verifying ingress addon... +🌟 Enabled addons: ingress, storage-provisioner, default-storageclass +🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default +``` +**Note**: The KVM driver provides a lot of options on customizing the cluster, however that is currently beyond the scope of this guide. + +Next we will get the Node IP. +``` +$ kubectl get no -o wide   +NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +minikube Ready control-plane,master 25d v1.23.3 192.168.39.57 Buildroot 2021.02.4 4.19.202 docker://20.10.12 +``` +Minikube creates a Virtual Machine using the KVM2 driver(Other drivers such as Virtualbox do exist see `minikube start --help` for more information ), you should be able to see this with the following output(You may have to use sudo to get this output) + +``` +$ virsh --connect qemu:///system list + Id Name State +-------------------------- + 1 minikube running + + or + + $ sudo virsh list + Id Name State +-------------------------- + 1 minikube running + +``` + +Moving on, simply create a nginx deployment using `kubectl`. +``` +# kubectl create deployment nginx --image nginx:alpine --port=80 +deployment.apps/nginx created +``` +Then we expose this as a NodePort Service. +``` +# kubectl expose deployment/nginx --type=NodePort +service/nginx-new exposed +``` +Command: Now we can see that the service has been exposed. +``` +# kubectl get svc -o wide    main  +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR +kubernetes ClusterIP 10.96.0.1 443/TCP 25d +nginx-minikube NodePort 10.97.44.4 80:32007/TCP 45h app=nginx-minikube +``` +Output Relevance: From the above output, we can see that our nginx pod is being exposed as the `NodePort` service type, and now we can curl the Node IP `192.168.39.57` with the exposed port `32007` + +Command: The pod has an IP as shown below +``` +# kubectl get po -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +nginx-minikube-7546f79bd8-x88bt 1/1 Running 3 (43m ago) 45h 172.17.0.4 minikube + +``` + +Command: We can use `curl` on the laptop to view the nginx container that is running on port `32007`. +``` +curl 192.168.39.57:32007 + + + +Welcome to nginx! + + + +

Welcome to nginx!

+

If you see this page, the nginx web server is successfully installed and +working. Further configuration is required.

+ +

For online documentation and support please refer to +nginx.org.
+Commercial support is available at +nginx.com.

+ +

Thank you for using nginx.

+ + +``` + +So, how does this packet travel, lets dive in. +We can check the ip interfaces as well subnets for our system is connected to: +``` +$ ifconfig +virbr2: flags=4163 mtu 1500 + inet 192.168.39.1 netmask 255.255.255.0 broadcast 192.168.39.255 + ether 52:54:00:19:29:93 txqueuelen 1000 (Ethernet) + RX packets 5132 bytes 1777099 (1.6 MiB) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 6113 bytes 998530 (975.1 KiB) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 + +virbr0: flags=4163 mtu 1500 + inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 + ether 52:54:00:48:ee:35 txqueuelen 1000 (Ethernet) + RX packets 23648 bytes 1265196 (1.2 MiB) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 40751 bytes 60265308 (57.4 MiB) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 +``` +Output Relevance: From the above output you can see there are two Virtual Bridges created by minikube when we created the cluster on the network. Here, `virbr0` is the default NAT network bridge while `virbr2` is a isolated network bridge on which the pods run. + +Minikube creates a Virtual Machine, to enter the virtual machine we can simple do: +``` +# minikube ssh +``` + +The interfaces within the Virtual Machine are as follows. +``` +docker0 Link encap:Ethernet HWaddr 02:42:03:24:26:78 + inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 + UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 + RX packets:31478 errors:0 dropped:0 overruns:0 frame:0 + TX packets:36704 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:0 + RX bytes:3264056 (3.1 MiB) TX bytes:14061883 (13.4 MiB) + +eth0 Link encap:Ethernet HWaddr 52:54:00:C9:3A:73 + inet addr:192.168.39.57 Bcast:192.168.39.255 Mask:255.255.255.0 + UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 + RX packets:8245 errors:0 dropped:9 overruns:0 frame:0 + TX packets:3876 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:1000 + RX bytes:812006 (792.9 KiB) TX bytes:1044724 (1020.2 KiB) + +eth1 Link encap:Ethernet HWaddr 52:54:00:7B:37:79 + inet addr:192.168.122.35 Bcast:192.168.122.255 Mask:255.255.255.0 + UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 + RX packets:4459 errors:0 dropped:9 overruns:0 frame:0 + TX packets:201 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:1000 + RX bytes:298528 (291.5 KiB) TX bytes:25813 (25.2 KiB) + +lo Link encap:Local Loopback + inet addr:127.0.0.1 Mask:255.0.0.0 + UP LOOPBACK RUNNING MTU:65536 Metric:1 + RX packets:946772 errors:0 dropped:0 overruns:0 frame:0 + TX packets:946772 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:1000 + RX bytes:213465460 (203.5 MiB) TX bytes:213465460 (203.5 MiB) + +vetha4f1dc5 Link encap:Ethernet HWaddr 3E:1C:FE:C9:75:86 + UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 + RX packets:10 errors:0 dropped:0 overruns:0 frame:0 + TX packets:16 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:0 + RX bytes:1413 (1.3 KiB) TX bytes:955 (955.0 B) + +vethbf35613 Link encap:Ethernet HWaddr BA:31:7D:AE:2A:BF + UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 + RX packets:3526 errors:0 dropped:0 overruns:0 frame:0 + TX packets:3934 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:0 + RX bytes:342408 (334.3 KiB) TX bytes:380193 (371.2 KiB) + +vethe092a51 Link encap:Ethernet HWaddr 8A:37:D3:D9:D9:0E + UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 + RX packets:9603 errors:0 dropped:0 overruns:0 frame:0 + TX packets:11151 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:0 + RX bytes:1199235 (1.1 MiB) TX bytes:5449408 (5.1 MiB) +``` +Output Relevance: Here we have the Virtual Ethernet and we have docker bridges too since docker runs within the Virtual Machine. + +When we do a `curl` to `192.168.39.57:32007` on the laptop the packet first goes to the route table +``` +Destination Gateway Genmask Flags MSS Window irtt Iface +0.0.0.0 192.168.31.1 0.0.0.0 UG 0 0 0 ethbr0 +172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 +172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-2fffe5cd5d9e +172.19.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-be5b544733a3 +192.168.31.0 0.0.0.0 255.255.255.0 U 0 0 0 ethbr0 +192.168.31.0 0.0.0.0 255.255.255.0 U 0 0 0 ethbr0 +192.168.39.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr2 +192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 +``` +Output Relevance: As you can see multiple routes are defined here, of which our Virtual Machine Node IP(192.168.39.57) is also shown in the table, so the packet now knows where it has to go. + +With that clear we now know how the packet goes from the laptop to the virtual bridge and then enters the Virtual Machine. + +Inside the virtual machine, [kube-proxy](https://kubernetes.io/docs/concepts/overview/components/#kube-proxy) handles the routing using iptables. + +So what exactly is [kube-proxy](https://kubernetes.io/docs/concepts/overview/components/#kube-proxy)(For those who skipped the kind example): +> Kube-Proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept. +kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster + +So, as we can see that kube proxy handles the network rules required to aid the communication to the pods, we will look at the [iptables](https://linux.die.net/man/8/iptables) +> `iptables` is a command line interface used to set up and maintain tables for the Netfilter firewall for IPv4, included in the Linux kernel. The firewall matches packets with rules defined in these tables and then takes the specified action on a possible match. Tables is the name for a set of chains + +Command: + +``` +# minikube ssh   + _ _ + _ _ ( ) ( ) + ___ ___ (_) ___ (_)| |/') _ _ | |_ __ +/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\ +| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/ +(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____) + +$ sudo iptables -t nat -L PREROUTING -n +Chain PREROUTING (policy ACCEPT) +target prot opt source destination +KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */ +DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL + +$ iptables-save | grep PREROUTING +-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES + +``` + +Output Relevance: +> -A: append new iptable rule +> -j: jump to the target +> KUBE-SERVICES: target + +> The above output appends a new rule for PREROUTING which every network packet will go through first as they try to access any kubernetes service + + +What is `PREROUTING` in iptables? +>PREROUTING: This chain is used to make any routing related decisions before (PRE) sending any packets + +To dig in further we need to go to the target, `KUBE-SERVICES` for our nginx service. +``` +# iptables -t nat -L KUBE-SERVICES -n| grep nginx +KUBE-SVC-NRDCJV6H42SDXARP tcp -- 0.0.0.0/0 10.97.44.4 /* default/nginx-minikube cluster IP */ tcp dpt:80 +``` +Command: +``` +$ sudo iptables -t nat -L| grep KUBE-SVC-NRDCJV6H42SDXARP +KUBE-SVC-NRDCJV6H42SDXARP tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/nginx-minikube */ tcp dpt:32007 +KUBE-SVC-NRDCJV6H42SDXARP tcp -- 0.0.0.0/0 10.97.44.4 /* default/nginx-minikube cluster IP */ tcp dpt:80 + +$ sudo iptables -t nat -L KUBE-MARK-MASQ -n +Chain KUBE-MARK-MASQ (19 references) +target prot opt source destination +MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000 + +sudo iptables-save | grep 172.17.0.4 +-A KUBE-SEP-AHQQ7ZFXMEBNX76B -s 172.17.0.4/32 -m comment --comment "default/nginx-minikube" -j KUBE-MARK-MASQ +-A KUBE-SEP-AHQQ7ZFXMEBNX76B -p tcp -m comment --comment "default/nginx-minikube" -m tcp -j DNAT --to-destination 172.17.0.4:80 +``` +As you can see the rules added by kube-proxy helps the packet reach to the destination service. + + +### Connection termination +Connection termination is a type of event that occurs when there are load balancers present, the information for this is quite scarce, however I've found the following article, [IBM - Network Termination](https://www.ibm.com/docs/en/sva/9.0.4?topic=balancer-network-termination) that describes what it means by connection termination between clients(laptop) and server(load balancer) and the various other services. + +### Different types of connection errors. +The following article on [TCP/IP errors](https://www.ibm.com/docs/en/db2/11.1?topic=message-tcpip-errors) has a list of the important tcp timeout errors that we need to know. + + +| Common TCP/IP errors | Meaning | +| -------- | -------- | +| Resource temporarily unavailable.| Self-explanatory. | +| No space is left on a device or system table.|The disk partition is full| +|No route to the host is available.|The routing table doesn't know where to route the packet.| +|Connection was reset by the partner.|This usually means the packet was dropped as soon as it reached the server can be due to a firewall.| +|The connection was timed out.|This indicates the firewall blocking your connection or the connection took too long.| + +## OSI Model Layer 7 (Application Layer) + +[What is layer 7?](https://www.cloudflare.com/learning/ddos/what-is-layer-7/) +#### Summary +Layer 7 refers to the seventh and topmost layer of the Open Systems Interconnect (OSI) Model known as the application layer. This is the highest layer which supports end-user processes and applications. Layer 7 identifies the communicating parties and the quality of service between them, considers privacy and user authentication, as well as identifies any constraints on the data syntax. This layer is wholly application-specific. + + +## Setting up Ingress-Nginx Controller + +Since we are doing this on our local laptop, we are going to use the following tools: +- [Minikube using KVM driver](https://minikube.sigs.k8s.io/docs/start/) - The host is linux-based in our example +- [Metallb](https://metallb.universe.tf/) - Baremetal load-balancer. +- [KVM](https://www.linux-kvm.org/page/Main_Page) / [Oracle VirtualBox](https://www.virtualbox.org/wiki/Downloads) / [VMWare](https://www.vmware.com/in/products/workstation-pro.html) + + +### So let's begin with Metallb and Ingress-Nginx setup. + +For setting up metallb, we are going to follow the below steps: + + - To begin the installation, we will execute: +``` +minikube start +``` +- To install Metallb, one can install it using the [manifest](https://metallb.universe.tf/installation/#installation-by-manifest) or by using [helm](https://metallb.universe.tf/installation/#installation-with-helm), for now we will use the Manifest method: +``` +kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.4/config/manifests/metallb-native.yaml +``` + +- We need to now configure Metallb, we are using [Layer 2 configuration](https://metallb.universe.tf/configuration/#announce-the-service-ips), let's head over to the [Metallb Configuration](https://metallb.universe.tf/configuration/) website, here you will see how to setup metallb. +>Layer 2 mode does not require the IPs to be bound to the network interfaces of your worker nodes. It works by responding to ARP requests on your local network directly, to give the machine’s MAC address to clients. +In order to advertise the IP coming from an IPAddressPool, an L2Advertisement instance must be associated to the IPAddressPool. +- We have modified the IP address pool so that our loadbalancer knows which subnet to choose an IP from.Since we have only one minikube IP we need to modify the code given in the documentation. +Save this as `metallb-config.yaml`: +``` +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + name: first-pool + namespace: metallb-system +spec: + addresses: + # The configuration website show's you this + + #- 192.168.10.0/24 + #- 192.168.9.1-192.168.9.5 + #- fc00:f853:0ccd:e799::/124 + + # We are going to change this to `minikube ip` as such + - 192.168.39.57/32 +``` +Now deploy it using `kubectl` +``` +kubectl apply -f metallb-config.yaml +``` +- Now that metallb is setup, let's install [ingress-nginx](https://kubernetes.github.io/ingress-nginx/deploy/#quick-start) on the laptop. +Note: We are using the install by manifest option from the Installation manual +``` +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/cloud/deploy.yaml +``` +or one can also install it using the minikube addons: +``` +minikube addons enable ingress +``` + - Once your Ingress-Nginx controller is created you can run the following commands to see the output of the setup done. +``` +kubectl get pods -n ingress-nginx +NAME READY STATUS RESTARTS AGE +ingress-nginx-admission-create-65bld 0/1 Completed 0 14m +ingress-nginx-admission-patch-rwq4x 0/1 Completed 0 14m +ingress-nginx-controller-6dc865cd86-7c5zd 1/1 Running 0 14m +``` +The Ingress controller creates a Service with the type LoadBalancer and metallb provides the IP address. + +``` +kubectl -n ingress-nginx get svc + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +ingress-nginx-controller LoadBalancer 10.108.154.53 192.168.39.223 80:30367/TCP,443:31491/TCP 4d15h +ingress-nginx-controller-admission ClusterIP 10.98.54.3 443/TCP 4d15h +``` + +#### Creating an Ingress + +We will deploy a `httpd` service in a `httpd` namespace and create a ingress for it. + +First, let's create a namespace. +``` +kubectl create namespace httpd +``` + +Next we will create a deployment +``` +kubectl create deployment httpd -n httpd --image=httpd:alpine +``` + +Now, In order to create a service, let's expose this deployment +``` +kubectl expose deployment -n httpd httpd --port 80 +``` +Let's check the `pod` that is created + +``` +kubectl get po -n httpd +NAME READY STATUS RESTARTS AGE +httpd-fb7fcdc77-w287c 1/1 Running 0 64s +``` + +Let's list the services in the `httpd` namespace +``` +kubectl get svc -n httpd +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +httpd ClusterIP 10.104.111.0 80/TCP 13s +``` + +Once we have this we can now create a n ingress using the following +``` +kubectl -n httpd create ingress httpd --class nginx --rule httpd.dev.leonnunes.com/"*"=httpd:80 +``` +The above output, creates an ingress, for us with the rule to match the service if the host is `httpd.dev.leonnunes.com`. The class here is retrieved from the below command. + +To list the `ingressclasses` use +``` +kubectl get ingressclasses +NAME CONTROLLER PARAMETERS AGE +nginx k8s.io/ingress-nginx 6h49m +``` + +The following command shows the ingress created +``` +$ kubectl get ingress -A -o wide + +NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE +httpd httpd nginx httpd.dev.leonnunes.com 192.168.39.223 80 11d +``` + +To test if the rule works we can now do +``` +$ minikube ip +192.168.39.223 + +$ curl --resolve httpd.dev.leonnunes.com:80:192.168.39.223 httpd.dev.leonnunes.com +

It works!

+ +or + +curl -H "Host: httpd.dev.leonnunes.com" 192.168.39.223 +``` + +#### Example of Information found on layer 7 +We have setup `Ingress-Nginx`, using `nginx` as a class and `httpd` for this example. + +In order to display the info on Layer - 7, we have extracted the Layer 7 information from a simple `curl` request, and then using `tcpdump` command within the `httpd` pod we extracted the network packets and opened it using the `Wireshark` utility. + +Below given is the output that is important: +```bash +Frame 4: 391 bytes on wire (3128 bits), 391 bytes captured (3128 bits) +Linux cooked capture v2 +Internet Protocol Version 4, Src: 172.17.0.4, Dst: 172.17.0.3 +Transmission Control Protocol, Src Port: 49074, Dst Port: 80, Seq: 1, Ack: 1, Len: 319 +Hypertext Transfer Protocol + GET / HTTP/1.1\r\n + Host: httpd.dev.leonnunes.com\r\n + X-Request-ID: 6e1a790412a0d1615dc0231358dc9c8b\r\n + X-Real-IP: 172.17.0.1\r\n + X-Forwarded-For: 172.17.0.1\r\n + X-Forwarded-Host: httpd.dev.leonnunes.com\r\n + X-Forwarded-Port: 80\r\n + X-Forwarded-Proto: http\r\n + X-Forwarded-Scheme: http\r\n + X-Scheme: http\r\n + User-Agent: curl/7.84.0\r\n + Accept: */*\r\n + \r\n + [Full request URI: http://httpd.dev.leonnunes.com/] + [HTTP request 1/1] + [Response in frame: 6] + +``` +The above output shows the information that the `httpd` pod recieves. The `curl` command sends the host header, `Host: httpd.dev.leonnunes.com`, to the nginx controller, that then matches the rule and sends the information to the right controller + +The following output shows what is sent via the laptop. +``` +curl --resolve httpd.dev.leonnunes.com:80:192.168.39.57 -H "Host: httpd.dev.leonnunes.com" 192.168.39.57 -vL +* Added httpd.dev.leonnunes.com:80:192.168.39.57 to DNS cache +* Trying 192.168.39.57:80... +* Connected to 192.168.39.57 (192.168.39.57) port 80 (#0) +> GET / HTTP/1.1 +> Host: httpd.dev.leonnunes.com +> User-Agent: curl/7.84.0 +> Accept: */* +> +* Mark bundle as not supporting multiuse +< HTTP/1.1 200 OK +< Date: Mon, 22 Aug 2022 16:05:27 GMT +< Content-Type: text/html +< Content-Length: 45 +< Connection: keep-alive +< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT +< ETag: "2d-432a5e4a73a80" +< Accept-Ranges: bytes +< +

It works!

+* Connection #0 to host 192.168.39.57 left intact +``` +As you can see from the above output there are several headers added to the curl output after it reaches the `httpd` pod, these headers are added by the Ingress Nginx Controller. + + +### References +#### Basics of Networking + - https://www.cisco.com/en/US/docs/security/vpn5000/manager/reference/guide/appA.html + - http://web.stanford.edu/class/cs101/ + - https://www.geeksforgeeks.org/basics-computer-networking/ + - Subnetting + - https://www.computernetworkingnotes.com/ccna-study-guide/subnetting-tutorial-subnetting-explained-with-examples.html + +#### Video Links + - https://www.youtube.com/playlist?list=PLhfrWIlLOoKPc2RecyiM_A9nf3fUU3e6g + - https://www.youtube.com/watch?v=S7MNX_UD7vY&list=PLIhvC56v63IJVXv0GJcl9vO5Z6znCVb1P + +### Topics to read about + - Docker in Docker + - [Docker/Containers](https://www.oreilly.com/library/view/docker-deep-dive/9781800565135/) + - Containers + +### Basics of Kubernetes +#### Reading Material +- https://nubenetes.com/kubernetes-tutorials/ +- https://kubernetes.io/docs/concepts/ +#### Video Material +- [Techworld with Nana 101](https://www.youtube.com/playlist?list=PLy7NrYWoggjziYQIDorlXjTvvwweTYoNC) +- [Jeff Geerling Kubernetes 101](https://www.youtube.com/watch?v=IcslsH7OoYo&list=PL2_OBreMn7FoYmfx27iSwocotjiikS5BD) + +#### Hands-On Kubernetes +- https://kube.academy/ +- https://www.civo.com/academy + +### Networking in Kubernetes +- [Kubernetes Networking 101](https://youtu.be/CYnwBIpvSlM?t=284) +- [CNCF Kubernetes 101](https://www.youtube.com/watch?v=cUGXu2tiZMc) + +### Tools/Commands to help with troubleshooting. +- [mtr](https://www.redhat.com/sysadmin/linux-mtr-command) - Tracing the packet from the source to destination +- [tcpdump](https://linuxconfig.org/how-to-use-tcpdump-command-on-linux) - Monitor packets +- [wireshark](https://www.lifewire.com/wireshark-tutorial-4143298) - Read/Sniff packets +- [nslookup](https://phoenixnap.com/kb/nslookup-command) - Lookup Nameservers +- [netstat](https://www.lifewire.com/netstat-command-2618098) - List network details +- [curl](https://linuxhandbook.com/curl-command-examples/) - Curl a website from the command line +- [ifconfig](https://www.tecmint.com/ifconfig-command-examples/)/[ip](https://www.geeksforgeeks.org/ip-command-in-linux-with-examples/) - Show ip address configuration +- [dig](https://www.geeksforgeeks.org/dig-command-in-linux-with-examples/) - Query Nameservers +- [ipcalc](https://www.linux.com/topic/networking/how-calculate-network-addresses-ipcalc/) - Calculate IP addresses +- Advanced Tools for troubleshooting + - [Netshoot](https://github.com/nicolaka/netshoot) - Troubleshoot Networks +- Cluster Creation tools + - [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) + - [minikube](https://minikube.sigs.k8s.io/docs/start/)