Make critical jobs Guaranteed Pod QOS: periodic-kubernetes-bazel-test

[spiffxp]

What should be cleaned up or changed:

This is part of #18530

The following jobs should be Guaranteed Pod QOS, meaning they should have CPU and memory resource limits, and matching resource requests:

• periodic-kubernetes-bazel-test-1-16
• periodic-kubernetes-bazel-test-1-17
• periodic-kubernetes-bazel-test-1-18
• periodic-kubernetes-bazel-test-1-19
• periodic-kubernetes-bazel-test-master

These jobs run on (google.com only) k8s-prow-build, so @spiffxp has provided the following guess:

• suggest 4 cpu, slightly above 36Gi mem (38?)

General steps to follow:

• update the job definitions in /config/jobs to have matching CPU and memory limits and requests
  • e.g. (ci-kubernetes-e2e-gci-gce has a resources: field with matching entries)
• open a pull request, include a link to this issue and cc @kubernetes/ci-signal in the description
• keep an eye on the jobs for the next few days, and if they start failing more than usual, open followup pull requests to raise resources
  • can look at recent prow.k8s.io runs (e.g. https://prow.k8s.io/?job=ci-kubernetes-e2e-gci-gce)
  • can look at job history (e.g. https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce)
  • can look at testgrid dashboards (e.g. https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-default)
• after the job has remained stable for a few days, declare victory

/sig testing
/sig release
/area jobs
/area release-eng

[spiffxp]

/assign @tpepper

[spiffxp]

/remove-help

[tpepper]

/reopen
for "keep an eye on the jobs for the next few days"

[k8s-ci-robot]

@tpepper: Reopened this issue.

In response to this:

/reopen
for "keep an eye on the jobs for the next few days"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

• https://testgrid.k8s.io/sig-release-master-blocking#bazel-test-master&width=5&graph-metrics=test-duration-minutes
  • this jumped up in duration and has gotten slightly flakier since switching away from RBE
  • since we've got a separate effort to deflake the unit tests in general I'm ok with this for now
• https://testgrid.k8s.io/sig-release-1.19-blocking#bazel-test-1.19&width=5&graph-metrics=test-duration-minutes
  • currently still running RBE and looking ok
• https://testgrid.k8s.io/sig-release-1.18-blocking#bazel-test-1.18&width=5&graph-metrics=test-duration-minutes
  • slight trend upward in duration but ok
• https://testgrid.k8s.io/sig-release-1.17-blocking#bazel-test-1.17&width=5&graph-metrics=test-duration-minutes
  • fine
• https://testgrid.k8s.io/sig-release-1.16-blocking#bazel-test-1.16&width=5&graph-metrics=test-duration-minutes
  • runs slower more often but within tolerable limits

[spiffxp]

/close
calling this good enough

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
calling this good enough

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.