[MRG] Add pod anti-affinity rule to build pods

[betatim]

With this rule build pods will prefer to be scheduled on separate nodes
if possible.

Let's see if this is enough to get build pods to spread out over the cluster instead of all (seemingly) ending up on the same node (ref: jupyterhub/mybinder.org-deploy#920)

[consideRatio]

NOTE: If we have 20 build pods (how many do we typically have?), i think "soft anti pod affinity" will only make them spread initially when some nodes have no build pods at all, but when all of the nodes have one build pod it wont matter if some have five or some have 20.

[consideRatio]

Ah btw also, previously these rules had bad performance associated with them, but no longer since k8s 1.12 if i recall correctly.

They made it like 100 times more performant.

[betatim]

but when all of the nodes have one build pod it wont matter if some have five or some have 20.

that is my understanding as well. Not sure yet what we should do once all nodes have a build pod. Probably we should start consuming a resource (a la jupyterhub/mybinder.org-deploy#955) with each build pod and have a scheduler that tries to keep consumption of the resource balanced.

Ah btw also, previously these rules had bad performance associated with them

Do you know some details on how bad the "bad" was? Was it something that only starts showing up if you cluster has tens of nodes or thousands of pods or already at the (small) scale of a BinderHub cluster (ten nodes, hundreds of pods)?

[betatim]

Merging this because you left an approving review and I don't think this will be controversial.

[consideRatio]

Probably we should start consuming a resource (a la jupyterhub/mybinder.org-deploy#955) with each build pod and have a scheduler that tries to keep consumption of the resource balanced.

Yeah something like that sounds good, but I think right now Kubernetes isn't mature enough to support us introducing something like this in the helm charts. Soonish I think the k8s scheduler, perhaps also in 1.13, is configurable with external configmaps etc rather than deploying a custom binary etc that is preconfigured etc.

[consideRatio]

+1 ;D I was just about to as well

[consideRatio]

Do you know some details on how bad the "bad" was? Was it something that only starts showing up if you cluster has tens of nodes or thousands of pods or already at the (small) scale of a BinderHub cluster (ten nodes, hundreds of pods)?

I think it was actually an issue for 100+ nodes or so, so it wasn't an issue before nor now