Add user docs for pod priority and preemption (#5328)

* Add user docs for pod priority and preemption

* Update pod-priority-preemption.md

* More updates

_data/concepts.yml

@@ -61,6 +61,7 @@ toc:
   - docs/concepts/configuration/taint-and-toleration.md
   - docs/concepts/configuration/secret.md
   - docs/concepts/configuration/organize-cluster-access-kubeconfig.md
+  - docs/concepts/configuration/pod-priority-preemption.md
 
 - title: Services, Load Balancing, and Networking
   section:

docs/concepts/configuration/pod-priority-preemption.md

@@ -0,0 +1,244 @@
+---
+approvers:
+- davidopp
+- wojtek-t
+title: Pod Priority and Preemption
+---
+
+{% capture overview %}
+
+{% include feature-state-alpha.md %}
+
+[Pods](/docs/user-guide/pods) in Kubernetes 1.8 and later can have priority. Priority
+indicates the importance of a Pod relative to other Pods. When a Pod cannot be scheduled,
+the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the
+pending Pod possible. In a future Kubernetes release, priority will also affect
+out-of-resource eviction ordering on the Node.
+
+**Note:** Preemption does not respect PodDisruptionBudget; see 
+[the limitations section](#poddisruptionbudget-is-not-supported) for more details.
+{: .note}
+
+{% endcapture %}
+
+{% capture body %}
+
+## How to use priority and preemption
+To use priority and preemption in Kubernetes 1.8, follow these steps:
+
+1. Enable the feature.
+
+1. Add one or more PriorityClasses.
+
+1. Create Pods with `PriorityClassName` set to one of the added PriorityClasses.
+Of course you do not need to create the Pods directly; normally you would add 
+`PriorityClassName` to the Pod template of a collection object like a Deployment.
+
+The following sections provide more information about these steps.
+
+## Enabling priority and preemption
+
+Pod priority and preemption is disabled by default in Kubernetes 1.8.
+To enable the feature, set this command-line flag for the API server 
+and the scheduler:
+
+```
+--feature-gates=PodPriority=true
+```
+
+Also set this flag for API server:
+
+
+```
+--runtime-config=scheduling.k8s.io/v1alpha1=true
+```
+
+After the feature is enabled, you can create [PriorityClasses](#priorityclass)
+and create Pods with [`PriorityClassName`](#pod-priority) set.
+
+If you try the feature and then decide to disable it, you must remove the PodPriority
+command-line flag or set it to false, and then restart the API server and
+scheduler. After the feature is disabled, the existing Pods keep their priority
+fields, but preemption is disabled, and priority fields are ignored, and you
+cannot set PriorityClassName in new Pods.
+
+## PriorityClass
+
+A PriorityClass is a non-namespaced object that defines a mapping from a priority
+class name to the integer value of the priority. The name is specified in the `name`
+field of the PriorityClass object's metadata. The value is specified in the required
+`value` field. The higher the value, the higher the priority. 
+
+A PriorityClass object can have any 32-bit integer value smaller than or equal to
+1 billion. Larger numbers are reserved for critical system Pods that should not
+normally be preempted or evicted. A cluster admin should create one PriorityClass
+object for each such mapping that they want.
+
+PriorityClass also has two optional fields: `globalDefault` and `description`.
+The `globalDefault` field indicates that the value of this PriorityClass should
+be used for Pods without a `PriorityClassName`. Only one PriorityClass with
+`globalDefault`  set to true can exist in the system. If there is no PriorityClass
+with `globalDefault` set, the priority of Pods with no `PriorityClassName` is zero.
+
+The `description` field is an arbitrary string. It is meant to tell users of
+the cluster when they should use this PriorityClass.
+
+**Note 1**: If you upgrade your existing cluster and enable this feature, the priority
+of your existing Pods will be considered to be zero.
+{: .note}
+
+**Note 2**: Addition of a PriorityClass with `globalDefault` set to true does not
+change the priorities of existing Pods. The value of such a PriorityClass is used only
+for Pods created after the PriorityClass is added.
+{: .note}
+
+**Note 3**: If you delete a PriorityClass, existing Pods that use the name of the
+deleted priority class remain unchanged, but you are not able to create more Pods
+that use the name of the deleted PriorityClass.
+{: .note}
+
+### Example PriorityClass
+
+```yaml
+apiVersion: v1
+kind: PriorityClass
+metadata:
+  name: high-priority
+value: 1000000
+globalDefault: false
+description: "This priority class should be used for XYZ service pods only."
+```
+
+## Pod priority
+
+After you have one or more PriorityClasses, you can create Pods that specify one
+of those PriorityClass names in their specifications. The priority admission
+controller uses the `priorityClassName` field and populates the integer value
+of the priority. If the priority class is not found, the Pod is rejected.
+
+The following YAML is an example of a Pod configuration that uses the PriorityClass
+created in the preceding example. The priority admission controller checks the
+specification and resolves the priority of the Pod to 1000000.
+
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx
+  labels:
+    env: test
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    imagePullPolicy: IfNotPresent
+  priorityClassName: high-priority
+```
+
+## Preemption
+
+When Pods are created, they go to a queue and wait to be scheduled. The scheduler
+picks a Pod from the queue and tries to schedule it on a Node. If no Node is found
+that satisfies all the specified requirements of the Pod, preemption logic is triggered 
+for the pending Pod. Let's call the pending pod P. Preemption logic tries to find a Node
+where removal of one or more Pods with lower priority than P would enable P to be scheduled
+on that Node. If such a Node is found, one or more lower priority Pods get
+deleted from the Node. After the Pods are gone, P can be scheduled on the Node. 
+
+### Limitations of preemption (alpha version)
+
+#### Starvation of preempting Pod
+
+When Pods are preempted, the victims get their
+[graceful termination period](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods).
+They have that much time to finish their work and exit. If they don't, they are
+killed. This graceful termination period creates a time gap between the point
+that the scheduler preempts Pods and the time when the pending Pod (P) can be
+scheduled on the Node (N). In the meantime, the scheduler keeps scheduling other
+pending Pods. As victims exit or get terminated, the scheduler tries to schedule
+Pods in the pending queue, and one or more of them may be considered and
+scheduled to N before the scheduler considers scheduling P on N. In such a case,
+it is likely that when all the victims exit, Pod P won't fit on Node N anymore.
+So, scheduler will have to preempt other Pods on Node N or another Node so that
+P can be scheduled. This scenario might be repeated again for the second and
+subsequent rounds of preemption, and P might not get scheduled for a while.
+This scenario can cause problems in various clusters, but is particularly
+problematic in clusters with a high Pod creation rate.
+
+We will address this problem in the beta version of Pod preemption. The solution
+we plan to implement is
+[provided here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/pod-preemption.md#preemption-mechanics).
+
+#### PodDisruptionBudget is not supported
+
+A [Pod Disruption Budget (PDB)](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
+allows application owners to limit the number Pods of a replicated application that
+are down simultaneously from voluntary disruptions. However, the alpha version of
+preemption does not respect PDB when choosing preemption victims.
+We plan to add PDB support in beta, but even in beta, respecting PDB will be best
+effort. The Scheduler will try to find victims whose PDB won't be violated by preemption,
+but if no such victims are found, preemption will still happen, and lower priority Pods
+will be removed despite their PDBs  being violated.
+
+#### Inter-Pod affinity on lower-priority Pods
+
+In version 1.8, a Node is considered for preemption only when
+the answer to this question is yes: "If all the Pods with lower priority than
+the pending Pod are removed from the Node, can the pending pod be scheduled on
+the Node?"
+
+**Note:** Preemption does not necessarily remove all lower-priority Pods. If the 
+pending pod can be scheduled by removing fewer than all lower-priority Pods, then
+only a portion of the lower-priority Pods are removed. Even so, the answer to the
+preceding question must be yes. If the answer is no, the Node is not considered
+for preemption.
+{: .note}
+
+If a pending Pod has inter-pod affinity to one or more of the lower-priority Pods
+on the Node, the inter-Pod affinity rule cannot be satisfied in the absence of those
+lower-priority Pods. In this case, the scheduler does not preempt any Pods on the
+Node. Instead, it looks for another Node. The scheduler might find a suitable Node
+or it might not. There is no guarantee that the pending Pod can be scheduled.
+
+We might address this issue in future versions, but we don't have a clear plan yet.
+We will not consider it a blocker for Beta or GA. Part
+of the reason is that finding the set of lower-priority Pods that satisfy all
+inter-Pod affinity rules is computationally expensive, and adds substantial 
+complexity to the preemption logic. Besides, even if preemption keeps the lower-priority
+Pods to satisfy inter-Pod affinity, the lower priority Pods might be preempted
+later by other Pods, which removes the benefits of having the complex logic of 
+respecting inter-Pod affinity.
+
+Our recommended solution for this problem is to create inter-Pod affinity only towards
+equal or higher priority pods.
+
+#### Cross node preemption
+
+Suppose a Node N is being considered for preemption so that a pending Pod P
+can be scheduled on N. P might become feasible on N only if a Pod on another
+Node is preempted. Here's an example:
+
+* Pod P is being considered for Node N.
+* Pod Q is running on another Node in the same zone as Node N.
+* Pod P has anit-affinity with Pod Q.
+* There are no other cases of anti-affinity between Pod P and other Pods in the zone.
+* In order to schedule Pod P on Node N, Pod Q should be preempted, but scheduler
+does not perform cross-node preemption. So, Pod P will be deemed unschedulable
+on Node N.
+
+If Pod Q were removed from its Node, the anti-affinity violation would be gone,
+and Pod P could possibly be scheduled on Node N.
+
+We may consider adding cross Node preemption in future versions if we find an
+algorithm with reasonable performance. We cannot promise anything at this point, 
+and cross Node preemption will not be considered a blocker for Beta or GA.
+
+{% endcapture %}
+
+{% capture whatsnext %}
+* Learn more about [this](...).
+* See this [related task](...).
+{% endcapture %}
+
+{% include templates/concept.md %}