Add kubernetes deployment example

adaptant-labs · Oct 29, 2020 · a723547 · a723547
1 parent 4472a28
commit a723547
Show file tree

Hide file tree

Showing 2 changed files with 135 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -20,7 +20,8 @@ feature.node.kubernetes.io/pci-0880_1ac1.present
 beta.devicetree.org/fsl-imx8mq-phanbell
 ```
 
-depending upon whether we are using the device plugin, NFD-based feature discovery, or DT-based resource discovery.
+depending upon whether we are using the device plugin, NFD-based feature discovery, or [DT-based][k8s-dt-node-labeller]
+resource discovery.
 
 Deployments wishing to target an EdgeTPU will then have to consider these different scenarios as part of their
 `nodeSelectorTerms` when determining node affinity for pod scheduling:
@@ -68,6 +69,8 @@ and are at risk of quickly becoming out of date.
 The auto labeller works around this by acting as a single source of truth for generalized labelling, while only
 requiring dependent labels to be updated in one place.
 
+[k8s-dt-node-labeller]: https://github.com/adaptant-labs/k8s-dt-node-labeller
+
 ## Label Definitions
 
 The auto labeller makes use of a series of flat files that contain dependent labels. These are provided by default in
@@ -93,6 +96,82 @@ If planning to use an external volume for label definitions, these can be passed
 under the `/labels` mount point. Note that the contents of the `labels/` directory will already be exposed as a
 persisted volume when the container is first run.
 
+## Usage
+
+General usage is as follows:
+
+```
+$ k8s-auto-labeller --help
+Node Auto Labeller for Kubernetes
+Usage: ./k8s-auto-labeller [flags]
+  -kubeconfig string
+	Paths to a kubeconfig. Only required if out-of-cluster.
+  -label-dir string
+	Label directory to monitor (default "labels")
+  -master --kubeconfig
+	(Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.
+```
+
+Using a USB-attached EdgeTPU as an example:
+
+```
+$ k8s-auto-labeller
+{"level":"info","ts":1604007914.093528,"logger":"k8s-auto-labeller","msg":"Adding watcher","dir":"/labels"}
+{"level":"info","ts":1604007914.0936112,"logger":"k8s-auto-labeller","msg":"Adding watcher","dir":"/labels/sodalite.eu"}
+{"level":"info","ts":1604007914.5515614,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
+{"level":"info","ts":1604007914.5517762,"logger":"k8s-auto-labeller.watcher","msg":"Monitoring filesystem for events..."}
+{"level":"info","ts":1604007914.5517755,"logger":"k8s-auto-labeller.entrypoint","msg":"starting manager"}
+{"level":"info","ts":1604007914.5519438,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
+{"level":"info","ts":1604007914.551965,"logger":"controller","msg":"Starting EventSource","controller":"k8s-auto-labeller","source":"kind source: /, Kind="}
+{"level":"info","ts":1604007914.6523216,"logger":"controller","msg":"Starting Controller","controller":"k8s-auto-labeller"}
+{"level":"info","ts":1604007914.65236,"logger":"controller","msg":"Starting workers","controller":"k8s-auto-labeller","worker count":1}
+{"level":"info","ts":1604007914.6525555,"logger":"k8s-auto-labeller.reconciler","msg":"Reconciling node","request":"/sgx-celsius-w550power","node":"sgx-celsius-w550power"}
+{"level":"info","ts":1604007914.6526318,"logger":"k8s-auto-labeller.reconciler","msg":"Setting label","request":"/sgx-celsius-w550power","label":"sodalite.eu/edgetpu"}
+...
+```
+
+The label state can be confirmed by querying the node labels directly:
+
+```
+$ kubectl get nodes sgx-celsius-w550power --show-labels
+NAME                    STATUS   ROLES    AGE    VERSION        LABELS
+sgx-celsius-w550power   Ready    <none>   148d   v1.18.3+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,feature.node.kubernetes.io/usb-fe_1a6e_089a.present=true,k3s.io/hostname=sgx-celsius-w550power,k3s.io/internal-ip=192.168.188.92,kubernetes.io/arch=amd64,kubernetes.io/hostname=sgx-celsius-w550power,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s,sodalite.eu/edgetpu=true
+```
+
+when the device is removed, the label is automatically cleared:
+
+```
+{"level":"info","ts":1604008598.1130025,"logger":"k8s-auto-labeller.reconciler","msg":"Reconciling node","request":"/sgx-celsius-w550power","node":"sgx-celsius-w550power"}
+{"level":"info","ts":1604008598.1130307,"logger":"k8s-auto-labeller.reconciler","msg":"Clearing label","request":"/sgx-celsius-w550power","label":"sodalite.eu/edgetpu"}
+...
+
+$ kubectl get nodes sgx-celsius-w550power --show-labels
+NAME                    STATUS   ROLES    AGE    VERSION        LABELS
+sgx-celsius-w550power   Ready    <none>   148d   v1.18.3+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,k3s.io/hostname=sgx-celsius-w550power,k3s.io/internal-ip=192.168.188.92,kubernetes.io/arch=amd64,kubernetes.io/hostname=sgx-celsius-w550power,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s
+```
+
+### Running as a Kubernetes Deployment
+
+An example Deployment configuration for automated cluster-wide node labelling is provided in
+`k8s-auto-labeller-deployment.yaml`, which can be, as the name implies, directly applied to the running cluster:
+
+```
+$ kubectl apply -f https://raw.githubusercontent.com/adaptant-labs/k8s-auto-labeller/k8s-auto-labeller-deployment.yaml
+```
+
+This will create a single Deployment in the cluster under the `kube-system` namespace. It will further create a special
+`auto-labeller` service account, cluster role, and binding with the permission to list, watch, and update nodes.
+
+Multi-arch containers are provided, allowing for direct deployment into clusters with both `amd64` and `arm64` nodes.
+
+### Automated Descheduling of Pods
+
+When used in conjunction with the [k8s-node-label-monitor], it is possible to automatically deschedule pods that have been
+scheduled with label dependencies that are no longer satisfied. This can be useful for evicting pods that have a hard
+hardware dependency where it no longer makes sense to keep them running when the hardware is no longer available.
+
+[k8s-node-label-monitor]: https://github.com/adaptant-labs/k8s-node-label-monitor
+
 ## Features and bugs
 
 Please file feature requests and bugs in the [issue tracker][tracker].

diff --git a/k8s-auto-labeller-deployment.yaml b/k8s-auto-labeller-deployment.yaml
@@ -0,0 +1,55 @@
+kind: ServiceAccount
+apiVersion: v1
+metadata:
+  name: auto-labeller
+  namespace: kube-system
+  labels:
+    addonmanager.kubernetes.io/mode: Reconcile
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: system:auto-labeller
+  labels:
+    addonmanager.kubernetes.io/mode: Reconcile
+rules:
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["list", "watch", "update", "get"]
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: system:auto-labeller
+  labels:
+    addonmanager.kubernetes.io/mode: Reconcile
+subjects:
+  - kind: ServiceAccount
+    name: auto-labeller
+    namespace: kube-system
+roleRef:
+  kind: ClusterRole
+  name: system:auto-labeller
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: auto-labeller
+  name: auto-labeller
+  namespace: kube-system
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: auto-labeller
+  template:
+    metadata:
+      labels:
+        app: auto-labeller
+    spec:
+      serviceAccountName: auto-labeller
+      containers:
+        - image: adaptant/k8s-auto-labeller:latest
+          name: auto-labeller