Kubernetes CI Policy: critical jobs must be Guaranteed Pod QOS #18530

BenTheElder · 2020-07-29T19:40:46Z

Part of #18551

This is a policy action Item out of Policies to improve Kubernetes CI discussed at SIG-Testing yesterday.

Why it’s necessary:
- We believe jobs are getting starved of the resources they require, because the scheduler is not able to effectively place them on a node with sufficient resources
- We believe that jobs that declare resource requests have a better chance at being placed on a node with sufficient resources for them (at scheduling time)
- We believe that jobs that declare resource limits are going to succeed because they have accurately declared resources, not because they are getting lucky and bursting above their requested resources
“Critical” here is release-blocking and merge-blocking for kubernetes/kubernetes
- trying to avoid needing to tackle the long tail of O(hundreds) of job configurations that do not specify resource requests/limits today, and focus on the release/merge blocking jobs first
- beyond Kubernetes release blocking / presubmit blocking tests figuring out priority fairly would be tricky. We've already started moving these specifically to CNCF owned GCP projects, so the tentative "priority" plan here is to just have dedicated cluster(s) for these workloads, versus general CI. If not for that, I think we should definitely consider it on those tasks at least.
Guaranteed QOS means
- Non-zero CPU & memory resource requests to avoid starvation (+1)
- Matching CPU & memory resource limits to ensure successes aren’t just due to burst (+1)
- For more info, see: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/

Checklist of release-blocking jobs: (h/t @tpepper)

force k8s-infra-prow-build jobs to be Guaranteed Pod QOS #18471 - everything else (initial proof of concept)
Make critical jobs Guaranteed Pod QOS: ci-kubernetes-build #18577 - ci-kubernetes-build
Make critical jobs Guaranteed Pod QOS: ci-kubernetes-build-fast #18578 - ci-kubernetes-build-fast
Make critical jobs Guaranteed Pod QOS: ci-kubernetes-e2e-device-plugin-gpu #18579 - ci-kubernetes-e2e-gce-device-plugin-gpu
Make critical jobs Guaranteed Pod QOS: ci-kubernetes-e2e-gci-gce-ingress #18580 - ci-kubernetes-e2e-gci-gce-ingress
Make critical jobs Guaranteed Pod QOS: periodic-kubernetes-bazel-build #18581 - periodic-kubernetes-bazel-build
Make critical jobs Guaranteed Pod QOS: periodic-kubernetes-bazel-test #18582 - periodic-kubernetes-bazel-test

Checklist of merge-blocking jobs (suggestions are based on metrics explorer, check against resource requests too!)

For release-blocking jobs:

replace the dashboard list with actual checklist of job names, along with checkboxes for which ones have been done
Add Pod QOS Guaranteed tests for critical jobs #18556 - implement test that enforces this policy at presubmit time - log instead of error
all individual release-blocking job issues listed above have been fixed (resources declared, jobs still pass after some soak time, jobs aren't declaring overly egregious resources)
Enforce release-blocking jobs must be Pod QOS Guaranteed #18751 - flip policy enforcement test from logging to erroring
declare that we have finished tuning release-blocking jobs

For merge-blocking jobs:

decide that we are comfortable enough with the move to go1.15 to introduce the instability in CI signal that setting/tuning resource limits may introduce -
replace the dashboard list with actual checklist of job names, along with checkboxes for which ones have been done
Add Pod QOS Guaranteed tests for critical jobs #18556 - implement test that enforces this policy at presubmit time - log instead of error
all individual release-blocking job issues listed above have been fixed (resources declared, jobs still pass after some soak time, jobs aren't declaring overly egregious resources)
Enforce critical merge-blocking jobs pod qos guaranteed #18834 - flip policy enforcement test from logging to erroring
declare that we have finished

(Punted "decide how we're going to measure success" to #18785)

How to make a guess:

if the job is running in k8s-infra-prow-build, go look at its historical resource usage (see "how to see resources" below)
if the job has resources declared already, try using that as the limit
if there's a suggestion, try using that
see if there is a similar look release-blocking job (name: ci-...) and compare limits
merge-blocking are going to take more cpu/memory than release-blocking since they need to build/compile
jobs that want as much cpu as possible on a 8cpu node can't ask for 8 cpu's: not all is allocatable, and some overhead must be budgeted for 100m init containers that are patched onto the pod by prow (I see 7300m-7500m used in some places, would start on the low end)
NOTE: unfortunately we have no resource utilization history available to the community for jobs running in k8s-prow-build (@spiffxp added guesses based on ~same metric-explorer approach for k8s-prow-build)

How to see resources (note: this only works for jobs that are running in k8s-infra-prow-build)

Open a pull-request against this file to be added to [email protected]: https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L680-L693
Use metrics explorer to look at a specific resource for jobs matching a regex:
- e.g. "Memory usage" for "ci-kubernetes-verify.*"
Metrics of interest:
- Memory usage
- Memory limit utilization
- CPU usage time
- CPU limit utilization
Be sure to vary time window, peaks are smoothed out over longer time windows
Consider switching to the query editor and writing MQL to come up with a more useful query? (ref: https://cloud.google.com/monitoring/mql/query-editor)
To see resource usage in aggregate:
- prow-build - resource usage in aggregate for the whole cluster
- prowjobs-experiment - cpu/mem usage and request utilization by prowjob

Once the above has been completed, we can move on to the next step: migrating everything to a dedicated cluster.

tpepper · 2020-07-29T19:50:28Z

/cc

BenTheElder · 2020-07-29T19:53:17Z

Additionally: This should be enforced by test-infra presubmit, to prevent regressions.

MushuEE · 2020-07-29T20:17:18Z

If I can find time I will try to generate a list of jobs that need updates and their paths, throw it up in a sheet with check boxes so that people can claim them to avoid duplicating efforts.

hasheddan · 2020-07-30T13:47:38Z

Though not yet on community owned infra, build-master and build-master-fast both experienced timeouts this morning, likely due to resource constraints.

Example from build-master-fast: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-build/1288690796073062402

Example from build-master: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-build-fast/1288815869782134786

BenTheElder · 2020-07-30T16:43:35Z

I added some notes around how to identify which jobs we're talking about here to the OP.

helenfeng737 · 2020-07-30T18:00:30Z

/cc

helenfeng737 · 2020-07-30T18:45:04Z

@BenTheElder I'm a bit confused here. For this: #18159 you said that we don't really want presubmit in testgrid. Could you explain more? Thanks

BenTheElder · 2020-07-30T18:51:43Z

@ZhiFeng1993 that's not related to this issue. Currently because they are in testgrid, that's a quick way to find a lot of the jobs.
This issue is about how the jobs are configured, not about testgrid. See e.g. #18471 mentioned in the original post.

Let's try to keep discussion here on-topic, lots of people are interested in this issue, it's going to take a lot of work, and github does not handle lots of comments well. 😅

@tpepper you can subscribe to a github issue by clicking "subscribe" on the righthand side of the web UI 🙃

tpepper · 2020-07-30T22:04:12Z

sig-release-master-blocking are:

config/jobs/kubernetes-sigs/kind/kind-release-blocking.yaml

"kind-master-parallel" aka ci-kubernetes-kind-e2e-parallel
- has limits
- has requests
"kind-ipv6-master-parallel" aka ci-kubernetes-kind-ipv6-e2e-parallel
- has limits
- has requests

config/jobs/kubernetes/sig-cli/sig-cli-config.yaml

"skew-cluster-latest-kubectl-stable1-gce" aka ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew
- has limits
- has requests

config/jobs/kubernetes/sig-cloud-provider/gcp/gce-conformance.yaml

"Conformance - GCE - master" aka ci-kubernetes-gce-conformance-latest
- has limits
- has requests

config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gce.yaml

config/jobs/kubernetes/sig-cloud-provider/gcp/gpu/gpu-gce.yaml

"gce-device-plugin-gpu-master" aka ci-kubernetes-e2e-gce-device-plugin-gpu
- needs limits (adding in Make critical jobs Guaranteed Pod QOS for ci-kubernetes-e2e-gce-devic… #18621)
- needs requests (adding in Make critical jobs Guaranteed Pod QOS for ci-kubernetes-e2e-gce-devic… #18621)

config/jobs/kubernetes/sig-network/sig-network-misc.yaml

[] "gci-gce-ingress" aka ci-kubernetes-e2e-gci-gce-ingress
- [] needs limits (adding in critical jobs Guaranteed Pod QOS: ci-kubernetes-e2e-gci-gce-ingress #18627)
- [] needs requests (adding in critical jobs Guaranteed Pod QOS: ci-kubernetes-e2e-gci-gce-ingress #18627)

config/jobs/kubernetes/sig-node/node-kubelet.yaml

"node-kubelet-master" aka ci-kubernetes-node-kubelet
- has limits
- has requests

config/jobs/kubernetes/sig-release/kubernetes-builds.yaml

"build-master" aka ci-kubernetes-build
- has requests
- needs limits (adding in # (adding in Make critical jobs Guaranteed Pod QOS for ci-kubernetes-build-* #18620)
"build-master-fast" aka ci-kubernetes-build-fast
- has requests
- has limits (added in Make critical jobs Guaranteed Pod QOS for ci-kubernetes-build-fast #18619)

config/jobs/kubernetes/sig-scalability/sig-scalability-release-blocking-jobs.yaml

"gce-cos-master-scalability-100" aka ci-kubernetes-e2e-gci-gce-scalability
- has requests
- has limits

config/jobs/kubernetes/sig-testing/bazel-build-test.yaml

"bazel-build-master" aka periodic-kubernetes-bazel-build-master
- needs requests (adding in critical jobs Guaranteed Pod QOS: periodic-kubernetes-bazel-build #18629)
- needs limits (adding in critical jobs Guaranteed Pod QOS: periodic-kubernetes-bazel-build #18629)
"bazel-test-master" aka periodic-kubernetes-bazel-test-master
- needs requests (adding in critical jobs Guaranteed Pod QOS: periodic-kubernetes-bazel-test #18630)
- needs limits (adding in critical jobs Guaranteed Pod QOS: periodic-kubernetes-bazel-test #18630)

config/jobs/kubernetes/sig-testing/conformance-e2e.yaml

"conformance-ga-only" aka k8s-infra-prow-build
- has requests
- has limits

config/jobs/kubernetes/sig-testing/integration.yaml

"integration-master" aka ci-kubernetes-integration-master
- has requests
- has limits

config/jobs/kubernetes/sig-testing/verify.yaml

"verify-master" aka ci-kubernetes-verify-master
- has requests
- has limits

tpepper · 2020-07-30T22:31:59Z

tpepper · 2020-07-30T22:38:33Z

tpepper · 2020-07-30T22:43:50Z

tpepper · 2020-07-30T22:48:27Z

spiffxp · 2020-07-30T23:46:09Z

Thanks for the lists tim, I tried to consolidate in description

LappleApple · 2020-07-31T13:53:36Z

/cc

spiffxp · 2020-07-31T17:30:26Z

#18556 enforces the policy in test form, but only logs instead of errors

Set resources limits and requests as suggested. Ref: kubernetes#18591 Part of: kubernetes#18530 Signed-off-by: Arnaud Meukam <[email protected]>

spiffxp · 2020-08-08T01:30:25Z

Alright, we've got limits set on all of the release-blocking jobs! I'm going to flip this test to fail instead of log

spiffxp@spiffxp-macbookpro:test-infra (master %)$ go test -v -count=1 ./config/tests/jobs
# ...
=== RUN   TestKubernetesReleaseBlockingJobsShouldHavePodQOSGuaranteed
--- PASS: TestKubernetesReleaseBlockingJobsShouldHavePodQOSGuaranteed (0.02s)
# ...

spiffxp · 2020-08-08T01:32:02Z

The merge-blocking situation is still pretty incomplete. I suspect we've at least missed a few release-branch jobs

spiffxp@spiffxp-macbookpro:test-infra (master %)$ go test -v -count=1 ./config/tests/jobs
# ...
=== RUN   TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-build ([]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-build ([]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-build ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-build ([]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-build ([]): container '' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-build ([]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.19]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.19]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.19]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.19]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.19]): container '' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.19]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.18]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.18]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.18]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.18]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.18]): container '' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.18]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.16]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.16]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.16]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.16]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.16]): container '' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.16]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([]): container '' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.17]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.17]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.17]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.17]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.17]): container '' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-bazel-test ([release-1.17]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-dependencies ([]): container 'main' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-dependencies ([]): container 'main' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-dependencies ([]): container 'main' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-dependencies ([]): container 'main' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-dependencies ([]): container 'main' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-dependencies ([]): container 'main' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-gce-network-proxy-grpc ([]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-gce-network-proxy-grpc ([]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-gce-network-proxy-grpc ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-gce-network-proxy-grpc ([]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-gce-network-proxy-grpc ([]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.18]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.18]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.18]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.18]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.19]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.19]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.19]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.19]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind-ipv6 ([]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind-ipv6 ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind-ipv6 ([]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind-ipv6 ([]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-files-remake ([release-1.19]): container 'main' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-files-remake ([release-1.19]): container 'main' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-files-remake ([release-1.19]): container 'main' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-files-remake ([release-1.19]): container 'main' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-files-remake ([release-1.19]): container 'main' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-files-remake ([release-1.19]): container 'main' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.17]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.17]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.17]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.17]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.17]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.19]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.19]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.19]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.19]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.19]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.16]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.16]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.16]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.16]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.16]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.18]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.18]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.18]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.18]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-kubemark-e2e-gce-big ([release-1.18]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.17]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.17]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.17]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.17]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.17]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.19]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.19]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.19]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.19]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.19]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.16]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.16]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.16]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.16]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.16]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.18]): container '' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.18]): container '' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.18]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.18]): container '' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.18]): container '' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.19]): container 'main' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.19]): container 'main' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.19]): container 'main' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.19]): container 'main' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.19]): container 'main' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.19]): container 'main' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.17]): container 'main' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.17]): container 'main' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.17]): container 'main' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.17]): container 'main' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.17]): container 'main' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.17]): container 'main' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.18]): container 'main' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.18]): container 'main' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.18]): container 'main' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.18]): container 'main' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.18]): container 'main' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.18]): container 'main' resources.limits[memory] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.16]): container 'main' should have resources.limits[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.16]): container 'main' should have resources.requests[cpu] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.16]): container 'main' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.16]): container 'main' should have resources.limits[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.16]): container 'main' should have resources.requests[memory] specified
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-typecheck ([release-1.16]): container 'main' resources.limits[memory] should be non-zero
--- PASS: TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed (0.00s)
# ...

RobertKielty · 2020-08-10T10:18:25Z

Looking at this test this morning (filtered on CPU not being zero so as to count the number of job file edits required to finish this out)

go test -v -run TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed  | grep  "resources.limits\[cpu\] should be non-zero"
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.18]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind ([release-1.19]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-e2e-kind-ipv6 ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.17]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.19]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.16]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([]): container '' resources.limits[cpu] should be non-zero
    TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed: jobs_test.go:1035: pull-kubernetes-node-e2e ([release-1.18]): container '' resources.limits[cpu] should be non-zero

Will ping @spiffxp later about me doing this work.

spiffxp · 2020-08-10T16:59:52Z

@RobertKielty you'll need to address comments on #18668 and then that should take care of the node jobs

#18691 is in flight for the kind jobs

spiffxp · 2020-08-11T23:50:22Z

I mentioned earlier today during SIG Testing meeting, but I suspect any of the issues that have been held open for soak time, making sure things are still running ok, etc. Can now probably be closed. @RobertKielty mentioned he was going to take a look at some. I will take a pass at some point but it may not be until Thursday at the rate I'm going

spiffxp · 2020-08-12T22:55:52Z

Anecdotally, while attempting to push some last minute PR's through the door for patch releases, it sure seems like merge-blocking presubmits are still flaking pretty badly.

BenTheElder · 2020-08-12T23:41:50Z

@spiffxp can you include your anecdata?
there is still some https://prow.k8s.io/?repo=kubernetes%2Fkubernetes&type=presubmit&state=error, we still ought to do #18226

spiffxp · 2020-08-13T00:38:01Z

It's not as straightforward as jobs hitting "error" state, though there are those (I'm just as willing to chalk that up to "now that we'e asking for resources, we're discovering they're not available instead of finding out the hard way")

I'll see if I can find a better way to measure/express this. But it's the fact that humans have sat on PR's hitting "/test" or "/retest" continually. Here’s a quick scan of PRs that have merged recently in release-1.16, release-1.17, release-1.18 and master. Is this worse or better than before? I’m not sure. Is this sort of thing worth scripting and generating a report/metric? Maybe

kubernetes/kubernetes#93927
Release-1.16
3 /test comments in the past 24h (integration, verify, gce-100)

kubernetes/kubernetes#93813
Release-1.16
One bot /retest comment in the past 24h

kubernetes/kubernetes#93924
Release-1.17
2 /test comments in past 24h (integration)

kubernetes/kubernetes#93696
Release-1.17
1 /test comment in past 24h (kubemark)

kubernetes/kubernetes#93812
Release-1.17
3 /retest comments in past 24h

kubernetes/kubernetes#93754
Release-1.18
1 /test comment in past 24h (e2e-gce)

kubernetes/kubernetes#93695
Release-1.18
1 /retest comment in past 24h
1 /test comment in past 24h (kind)

kubernetes/kubernetes#93811
Release-1.18
1 /retest comment in past 24h

kubernetes/kubernetes#93929
Master
2 /retest comments in past 24h

kubernetes/kubernetes#93829
Master
0 /retest or /test comments, woo!

kubernetes/kubernetes#93857
Master
3 /retest comments in past 24h

kubernetes/kubernetes#93907
Master
1 /retest comment in past 24h

kubernetes/kubernetes#93521
Master
1 /retest comment in past 24h
1 /test comment in past 24h (kind ipv6)

kubernetes/kubernetes#93895
Master
0 /retest or /test comments in past 24h, woo!

kubernetes/kubernetes#93893
Master
3 /retest comments in past 24h

kubernetes/kubernetes#93831
Master
3 /retest comments in past 24h

BenTheElder · 2020-08-13T03:57:13Z

Test flake fixes aren't always back ported to older releases, and I had some concerns about some recent CPU limits being set lower on older branches ...
Flakes on the master branch would be my greatest concern.

The first one I sampled had kubernetes/kubernetes#93929 (comment)
that's a failure to download things in bazel WORKSPACE, which doesn't really retry sufficiently. It's unrelated to the work here and not new.

spiffxp · 2020-08-14T00:31:39Z

Alright, we've got limits set on all of the merge-blocking jobs! I'm going to flip this test to fail instead of log

$ go test -v -count=1 ./config/tests/jobs/jobs_test.go
=== RUN   TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed
--- PASS: TestKubernetesMergeBlockingJobsShouldHavePodQOSGuaranteed (0.00s)

What remains is:

verifying a few of the merge-blocking jobs are still healthy with their limits in place
jobs aren't declaring overly egregious resources (I have spot checked this a little bit, and am inclined to defer doing this properly until after moving merge-blocking jobs to k8s-infra, so non-googlers can help with this)
declaring victory

spiffxp · 2020-08-25T17:22:01Z

CSV report generated by https://github.com/kubernetes/test-infra/tree/master/experiment/prowjob-report then imported into google sheets

https://docs.google.com/spreadsheets/d/1dgSxfa0jdaYk76S6DqJeazl2TKVAvZcCg-7Yx3U0j-I/edit#gid=1043959272

spiffxp · 2020-08-25T22:31:46Z

/close
Discussed during SIG Testing meeting today, we're calling this done!

k8s-ci-robot · 2020-08-25T22:32:00Z

@spiffxp: Closing this issue.

In response to this:

/close
Discussed during SIG Testing meeting today, we're calling this done!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BenTheElder added the kind/bug Categorizes issue or PR as related to a bug. label Jul 29, 2020

This was referenced Jul 29, 2020

Tune memory down for ci-kubernetes-verify jobs #18502

Merged

Tune ci-kubernetes-verify-* memory down to 16Gi #18508

Merged

force k8s-infra-prow-build jobs to be Guaranteed Pod QOS #18471

Merged

BenTheElder added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Jul 30, 2020

BenTheElder mentioned this issue Jul 30, 2020

Increase memory limit for 5k correctness test #18496

Merged

hasheddan mentioned this issue Jul 30, 2020

[SIG Release, SIG Testing] "Extract" phase is failing for multiple jobs using the ci/latest-fast version marker kubernetes/kubernetes#93561

Closed

BenTheElder mentioned this issue Jul 30, 2020

ci-kubernetes-conformance-kind-ga-only is experiencing pod scheduling timeout #18507

Closed

hasheddan mentioned this issue Jul 30, 2020

Pass prowjob.json to metadata lens #18537

Merged

BenTheElder mentioned this issue Jul 30, 2020

Move remake verify job to periodic #18524

Merged

spiffxp mentioned this issue Jul 30, 2020

Kubernetes CI Policy (Umbrella issue) #18551

Open

8 tasks

spiffxp changed the title ~~Critical jobs that run as Pods must be scheduled as Guaranteed QOS Pods (ie best-effort pods)~~ Kubernetes CI Policy: critical jobs must be Guaranteed Pod QOS Jul 30, 2020

This was referenced Aug 5, 2020

Implements Guarantees QoS for pull-kubernetes-conformance-kind-ga-only-parallel #18671

Merged

Implements QoS Guarantee for pull-kubernetes-files-remake #18676

Merged

This was referenced Aug 5, 2020

Make critical jobs Guaranteed Pod QOS: pull-kubernetes-e2e-gce-network-proxy-grpc #18677

Closed

Make critical jobs Guaranteed Pod QOS: pull-kubernetes-e2e-gce-network-proxy-http-connect #18678

Closed

ameukam mentioned this issue Aug 5, 2020

Configure Guaranteed Pod QoS for pull-kubernetes-e2e-kind #18691

Merged

pontiyaraja mentioned this issue Aug 7, 2020

Make pull-kubernetes-kubemark-e2e-gce-big job guaranteed Pod QOS #18698

Merged

spiffxp mentioned this issue Aug 8, 2020

Enforce release-blocking jobs must be Pod QOS Guaranteed #18751

Merged

spiffxp mentioned this issue Aug 14, 2020

Enforce critical merge-blocking jobs pod qos guaranteed #18834

Merged

jprzychodzen mentioned this issue Aug 14, 2020

Backport [scalability][presubmits][k/k] Update resources #18841

Merged

MushuEE mentioned this issue Aug 20, 2020

Small fix to multicontainer job: change bash to sh #18914

Merged

k8s-ci-robot closed this as completed Aug 25, 2020

BenTheElder added this to the v1.19 milestone Aug 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes CI Policy: critical jobs must be Guaranteed Pod QOS #18530

Kubernetes CI Policy: critical jobs must be Guaranteed Pod QOS #18530

BenTheElder commented Jul 29, 2020 •

edited by spiffxp

Loading

tpepper commented Jul 29, 2020

BenTheElder commented Jul 29, 2020

MushuEE commented Jul 29, 2020

hasheddan commented Jul 30, 2020 •

edited

Loading

BenTheElder commented Jul 30, 2020

helenfeng737 commented Jul 30, 2020

helenfeng737 commented Jul 30, 2020

BenTheElder commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

spiffxp commented Jul 30, 2020

LappleApple commented Jul 31, 2020

spiffxp commented Jul 31, 2020

spiffxp commented Aug 8, 2020 •

edited

Loading

spiffxp commented Aug 8, 2020

RobertKielty commented Aug 10, 2020

spiffxp commented Aug 10, 2020

spiffxp commented Aug 11, 2020

spiffxp commented Aug 12, 2020

BenTheElder commented Aug 12, 2020

spiffxp commented Aug 13, 2020

BenTheElder commented Aug 13, 2020

spiffxp commented Aug 14, 2020

spiffxp commented Aug 25, 2020

spiffxp commented Aug 25, 2020

k8s-ci-robot commented Aug 25, 2020

Kubernetes CI Policy: critical jobs must be Guaranteed Pod QOS #18530

Kubernetes CI Policy: critical jobs must be Guaranteed Pod QOS #18530

Comments

BenTheElder commented Jul 29, 2020 • edited by spiffxp Loading

tpepper commented Jul 29, 2020

BenTheElder commented Jul 29, 2020

MushuEE commented Jul 29, 2020

hasheddan commented Jul 30, 2020 • edited Loading

BenTheElder commented Jul 30, 2020

helenfeng737 commented Jul 30, 2020

helenfeng737 commented Jul 30, 2020

BenTheElder commented Jul 30, 2020 • edited Loading

tpepper commented Jul 30, 2020 • edited Loading

tpepper commented Jul 30, 2020 • edited Loading

tpepper commented Jul 30, 2020 • edited Loading

tpepper commented Jul 30, 2020 • edited Loading

tpepper commented Jul 30, 2020 • edited Loading

spiffxp commented Jul 30, 2020

LappleApple commented Jul 31, 2020

spiffxp commented Jul 31, 2020

spiffxp commented Aug 8, 2020 • edited Loading

spiffxp commented Aug 8, 2020

RobertKielty commented Aug 10, 2020

spiffxp commented Aug 10, 2020

spiffxp commented Aug 11, 2020

spiffxp commented Aug 12, 2020

BenTheElder commented Aug 12, 2020

spiffxp commented Aug 13, 2020

BenTheElder commented Aug 13, 2020

spiffxp commented Aug 14, 2020

spiffxp commented Aug 25, 2020

spiffxp commented Aug 25, 2020

k8s-ci-robot commented Aug 25, 2020

BenTheElder commented Jul 29, 2020 •

edited by spiffxp

Loading

hasheddan commented Jul 30, 2020 •

edited

Loading

BenTheElder commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

tpepper commented Jul 30, 2020 •

edited

Loading

spiffxp commented Aug 8, 2020 •

edited

Loading