feat: add tolerations handling to kubernetes driver

[szeber]

This PR is to add support for Kubernetes tolerations. Should support all current tolerations options and adding multiple tolerations. This should cover the use cases listed in #377, but it doesn't add the full manifests support.

Adding full support to manifests would be a much more flexible option, but this should solve the main pain points listed in that ticket.

[tonistiigi]

@szeber missing DCO

[AkihiroSuda]

Can we have effect examples?

[szeber]

Added an example there. Would it make sense to add a link here to the relevant kube docs? https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

[szeber]

Thanks @AkihiroSuda @tonistiigi, updated, could you please recheck?

[AkihiroSuda]

CI failing

#13 98.37 driver/kubernetes/factory.go:5: File is not `goimports`-ed (goimports)
#13 98.37 	corev1 "k8s.io/api/core/v1"
#13 98.37 driver/kubernetes/factory.go:[125](https://github.com/docker/buildx/runs/5842873028?check_suite_focus=true#step:4:125):11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (golint)
#13 98.37 			} else {
#13 98.37 			       ^
#13 98.37 driver/kubernetes/factory.go:152:15: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (golint)
#13 98.37 							} else {
#13 98.37 							       ^
#13 ERROR: process "/bin/sh -c golangci-lint run" did not complete successfully: exit code: 1
------
 > [stage-0 5/6] RUN --mount=target=/go/src/github.com/docker/buildx --mount=target=/root/.cache,type=cache   golangci-lint run:
#13 98.37 driver/kubernetes/factory.go:5: File is not `goimports`-ed (goimports)
#13 98.37 	corev1 "k8s.io/api/core/v1"
#13 98.37 driver/kubernetes/factory.go:125:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (golint)
#13 98.37 			} else {
#13 98.37 			       ^
#13 98.37 driver/kubernetes/factory.go:152:15: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (golint)
#13 98.37 							} else {
#13 98.37 							       ^
------

[szeber]

@AkihiroSuda hopefully this fixes the lint

[szeber]

@AkihiroSuda hopefully the last try, sorry :)

[ktaborski]

@szeber thanks for creating this change. I tried to use it in my env, but for some reason it's not working:

$ echo FROM scratch > Dockerfile.dummy
$ buildx version
github.com/docker/buildx a60150c a60150cbc65687c7cc921d519a2f35758202edb6
$ buildx create --name test_1 --driver kubernetes --driver-opt image=moby/buildkit:master,namespace=jenkins,requests.cpu=500m,requests.memory=300Mi,tolerations="key=spot,operator=exists;key=dedicated,value=jenkins" --use
test_1
$ buildx build . -f Dockerfile.dummy
[+] Building 0.0s (0/0)                                                                                                                                                                                                                      
Error: no valid drivers found: invalid driver option operator for driver kubernetes
<help message>

$ buildx create --name test_2 --driver kubernetes --driver-opt image=moby/buildkit:master,namespace=jenkins,requests.cpu=500m,requests.memory=300Mi --use
test_2
$  buildx build . -f Dockerfile.dummy
WARNING: No output specified for kubernetes driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 5.2s (3/3) FINISHED                                                                                                                                                                                                             
 => [internal] booting buildkit                                                                                                                                                                                                         4.3s
 => => waiting for 1 pods to be ready                                                                                                                                                                                                   4.1s
 => [internal] load build definition from Dockerfile.dummy                                                                                                                                                                              0.2s
 => => transferring dockerfile: 56B                                                                                                                                                                                                     0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.2s
 => => transferring context: 2B
 
 <It works>

$ buildx create --name test_3 --driver kubernetes --driver-opt tolerations="key=spot,operator=exists;key=dedicated,value=jenkins" --use
test_3
$ buildx build . -f Dockerfile.dummy
[+] Building 0.0s (0/0)                                                                                                                                                                                                                      
Error: no valid drivers found: invalid driver option operator for driver kubernetes
<help message>

Can you suggest me, what I am doing wrong?

[ktaborski]

@szeber thanks for creating this change. I tried to use it in my env, but for some reason it's not working:

$ echo FROM scratch > Dockerfile.dummy
$ buildx version
github.com/docker/buildx a60150c a60150cbc65687c7cc921d519a2f35758202edb6
$ buildx create --name test_1 --driver kubernetes --driver-opt image=moby/buildkit:master,namespace=jenkins,requests.cpu=500m,requests.memory=300Mi,tolerations="key=spot,operator=exists;key=dedicated,value=jenkins" --use
test_1
$ buildx build . -f Dockerfile.dummy
[+] Building 0.0s (0/0)                                                                                                                                                                                                                      
Error: no valid drivers found: invalid driver option operator for driver kubernetes
<help message>

$ buildx create --name test_2 --driver kubernetes --driver-opt image=moby/buildkit:master,namespace=jenkins,requests.cpu=500m,requests.memory=300Mi --use
test_2
$  buildx build . -f Dockerfile.dummy
WARNING: No output specified for kubernetes driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 5.2s (3/3) FINISHED                                                                                                                                                                                                             
 => [internal] booting buildkit                                                                                                                                                                                                         4.3s
 => => waiting for 1 pods to be ready                                                                                                                                                                                                   4.1s
 => [internal] load build definition from Dockerfile.dummy                                                                                                                                                                              0.2s
 => => transferring dockerfile: 56B                                                                                                                                                                                                     0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.2s
 => => transferring context: 2B
 
 <It works>

$ buildx create --name test_3 --driver kubernetes --driver-opt tolerations="key=spot,operator=exists;key=dedicated,value=jenkins" --use
test_3
$ buildx build . -f Dockerfile.dummy
[+] Building 0.0s (0/0)                                                                                                                                                                                                                      
Error: no valid drivers found: invalid driver option operator for driver kubernetes
<help message>

Can you suggest me, what I am doing wrong?

[szeber]

@ktaborski I don't think you did anything wrong, there is a bug in how the options are getting parsed.

@AkihiroSuda by the looks of it both this and nodeselector is broken when , is present in the argument list... I'm looking for options to solve this

[szeber]

So for example --nodeselector="foo=bar,foo2=bar2" is parsed incorrectly. It gets parsed to

{
"nodeselector": "foo=bar",
"foo2": "bar"
}

[tonistiigi]

You need to either quote it correctly "foo=1,2",bar=3 with proper escaping for the quotes so your shell doesn't eat them.

[szeber]

@tonistiigi Yes, that is one part, but the CSV parser doesn't like it quoted that way. what seems to work for node selectors is --driver-opts \"nodeselector=foo=bar,foo2=bar2\"