[Node] Add node-e2e tester #103

amwat · 2021-03-05T09:18:02Z

Also add a noop deployer to go with it.

Started with REMOTE=true only since most testing uses that.

fixes: #45

ref:
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/e2e-node-tests.md#delete-instance-after-tests-run
ref: https://github.com/kubernetes/kubernetes/blob/96be00df69390ed41b8ec22facc43bcbb9c88aae/build/root/Makefile#L206-L271
ref: https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-e2e-node.sh
xref: kubernetes/enhancements#2464

Currently fully relies on make test-e2e-node which itself does a build https://github.com/kubernetes/kubernetes/blob/cea1d4e20b4a7886d8ff65f34c6d4f95efcb4742/test/e2e_node/builder/build.go,
might be possible to breakout the build part separately into a dedicated node deployer in the future.

Also sadly, local runs don't stream the test logs everything is dumped when the tests finish

tested locally with

kubetest2 noop -v2 --test=node -- --repo-root=<repo-root> --gcp-zone=<zone> --gcp-project=<project> -v2 --focus-regex='should.not.show.its.pid.in.the.non-hostpid.containers.\[NodeFeature:HostAccess\]' --test-args='--kubelet-flags="--cgroups-per-qos=true --cgroup-root=/"'

started with porting over functionality needed by pull-kubernetes-node-e2e

/cc @BenTheElder @spiffxp @dims

amwat · 2021-03-08T20:04:32Z

FYI @SergeyKanzhelev

amwat · 2021-03-12T22:24:24Z

ping

amwat · 2021-03-15T18:57:06Z

/cc @ehashman

bobbypage · 2021-03-26T00:30:20Z

/cc

k8s-ci-robot · 2021-03-26T00:30:22Z

@bobbypage: GitHub didn't allow me to request PR reviews from the following users: bobbypage.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ehashman · 2021-03-26T00:32:12Z

What is this?

BenTheElder · 2021-03-26T00:46:29Z

kubetest2 replaces https://github.com/kubernetes/test-infra/tree/master/kubetest (deprecated).
this PR introduces support for invoking the node_e2e test suite

BenTheElder · 2021-03-26T00:47:26Z

kubetest2-noop/deployer/deployer.go

+)
+
+// Name is the name of the deployer
+const Name = "noop"


this should probably be a seperate PR, so folks from SIG node can focus on the e2e_node specific parts.

bobbypage · 2021-03-26T00:47:46Z

pkg/testers/node/node.go

+	gceProjectResourceType = "gce-project"
+)
+
+type Tester struct {


how do we pass in --image-config-file? https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/runner/remote/run_remote.go#L55

Many tests pass in custom image file, e.g. https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/kubernetes/test-infra%24+file:%5Econfig/jobs+--image-config-file%3D&patternType=literal

So this PR isn't the full migration, I anticipate we'll port over specific functionality as and when needed. Infact I'd prefer that so we don't just blindly port over all the flags.

I chose the subset of things used by pull-kubernetes-node-e2e as a start

Got it, makes sense, would be good clarify that in the PR description that goal here is just pull-kubernetes-node-e2e for now.

Ack, updated PR description

bobbypage · 2021-03-26T00:55:54Z

pkg/testers/node/node.go

+	argsFromFlags := []string{
+		"SKIP=" + t.SkipRegex,
+		"FOCUS=" + t.FocusRegex,
+		"RUNTIME=" + t.Runtime,


I'm not sure if just exposing RUNTIME is enough. For example, take this job: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-node/crio.yaml#L52

- '--node-test-args=--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --non-masquerade-cidr=0.0.0.0/0" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'

I'm not clear how that translates into equivalent args here...

Note that power users can still set the corrsponding ENVs before invoking kubetest2 (until kubetest2 supports them as flags (if at all))

Note that power users can still set the corrsponding ENVs before invoking kubetest2

How does that work? Doesn't the make cmd execution need to inherit the bash env variables, i.e. something like cmd.Env = os.Environ()?

Additionally, I think the challenge is it's not clear what exact envs you need set because the current jobs using kubetest(1) node e2e runner took in more or less same args as run_remote.go. However since this is based on test-e2e-node.sh, for all those jobs, you'll need to translate those args to the equivalent envs from test-e2e-node.sh which isn't super straightforward.

https://golang.org/pkg/os/exec/#Cmd

cmd will inherit all exported envs from parent process by default.

I think the challenge is it's not clear what exact envs you need set

Ack, but I think that is orthogonal to the kubetest2 tester invocation. i.e. they come more from https://github.com/kubernetes/kubernetes/blob/96be00df69390ed41b8ec22facc43bcbb9c88aae/build/root/Makefile#L206-L271

Ideally, we will port over all relevant functionality as flags over time.

I think the main question at hand is whether or not sig-node thinks make test-e2e-node is a better common entrypoint for node testing as opposed what we have currently https://github.com/kubernetes/test-infra/blob/4be647a393b6b491a2c0f00145d8bb9f8416b082/kubetest/e2e.go#L556-L611

ack got it.

I think the main question at hand is whether or not sig-node thinks make test-e2e-node is a better common entrypoint for node testing as opposed what we have currently

Yup, this is the main question.

As I mentioned the earlier, the problem I see is if the goal is for us is to migrate all e2e node jobs to kubetest2 eventually this will basically require rewriting all of the job configs since the current kubetest(1) interface doesn't rely on make test-e2e-node. On other side standardizing on make test-e2e-node is maybe a good move eventually :) but worth to discuss.

Folks in the sig-node-ci group would be candidates to ask about this.

cc a couple folks to get their thoughts on usage of make test-e2e-node vs existing kubetest node e2e interface.

/cc @SergeyKanzhelev @ehashman @fromanirh @knabben @harche

+1 for standardizing on make test-e2e-node

k8s-ci-robot · 2021-03-31T04:47:31Z

@bobbypage: GitHub didn't allow me to request PR reviews from the following users: fromanirh, knabben.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

ack got it.

I think the main question at hand is whether or not sig-node thinks make test-e2e-node is a better common entrypoint for node testing as opposed what we have currently

Yup, this is the main question.

As I mentioned the earlier, the problem I see is if the goal is for us is to migrate all e2e node jobs to kubetest2 eventually this will basically require rewriting all of the job configs since the current kubetest1 interface doesn't rely on make test-e2e-node. On other side standardizing on make test-e2e-node maybe a good move eventually :) but worth to discuss.

Folks in the sig-node-ci group would be candidates to ask about this.

cc a couple folks to get their thoughts on usage of make test-e2e-node vs existing kubetest node e2e interface.

/cc @SergeyKanzhelev @ehashman @fromanirh @knabben

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BenTheElder · 2021-06-18T03:23:03Z

FYI kubetest is definitely deprecated, so we would still appreciate eyes on the migration, given node e2e is a bit different from the swaths of cluster e2e jobs SIG Testing tends to be most familiar with.

spiffxp · 2021-07-23T20:01:11Z

/assign @ehashman @SergeyKanzhelev @derekwaynecarr
Can y'all find someone from SIG Node who is willing to take a look at this? The goal is to make running node-e2e less painful, and a thing that node owns

dims · 2021-08-03T22:50:10Z

/cc will take a look in the next couple of days!

dims · 2021-08-06T18:48:45Z

Ran this:

kubetest2 noop -v2 --test=node -- --repo-root=/home/dims/go/src/k8s.io/kubernetes --gcp-zone=$MY_ZONE --gcp-project=$MY_PROJECT -v2 --focus-regex='should.not.show.its.pid.in.the.non-hostpid.containers.\[NodeFeature:HostAccess\]' --test-args='--kubelet-flags="--cgroups-per-qos=true --cgroup-root=/"'

got this:

Ran 1 of 354 Specs in 142.957 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 353 Skipped


Ginkgo ran 1 suite in 2m24.56251924s
Test Suite Passed

Success Finished Test Suite on Host test-cos-beta-89-16108-403-11

There are definitely some rough edges around credentials, ssh keys, etc that we will eventually need to work through. Let's merge and iterate!

dims · 2021-08-06T18:49:14Z

/approve
/lgtm

thanks @amwat

k8s-ci-robot · 2021-08-06T18:49:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amwat, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [amwat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2021-08-06T18:50:50Z

@amwat: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
gke-up-down-singlecluster	`99f690a`	link	`/test gke-up-down-singlecluster`
gke-up-down-multicluster	`99f690a`	link	`/test gke-up-down-multicluster`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dims · 2021-08-06T19:30:04Z

/lgtm

k8s-ci-robot requested review from BenTheElder, dims and spiffxp March 5, 2021 09:18

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 5, 2021

k8s-ci-robot requested a review from ehashman March 15, 2021 18:57

BenTheElder reviewed Mar 26, 2021

View reviewed changes

bobbypage reviewed Mar 26, 2021

View reviewed changes

k8s-ci-robot requested a review from SergeyKanzhelev March 31, 2021 04:47

k8s-ci-robot assigned derekwaynecarr, ehashman and SergeyKanzhelev Jul 23, 2021

k8s-ci-robot assigned dims Aug 6, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2021

amwat added 3 commits August 6, 2021 12:09

Add noop deployer

842913c

add node tester

d1530ff

add boskos heartbeat to node tester

bf408b7

amwat force-pushed the node branch from 99f690a to bf408b7 Compare August 6, 2021 19:28

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2021

k8s-ci-robot merged commit d9db4a4 into kubernetes-sigs:master Aug 6, 2021

amwat deleted the node branch August 6, 2021 19:33

amwat mentioned this pull request Aug 6, 2021

Release #17

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Node] Add node-e2e tester #103

[Node] Add node-e2e tester #103

amwat commented Mar 5, 2021 •

edited

Loading

amwat commented Mar 8, 2021

amwat commented Mar 12, 2021

amwat commented Mar 15, 2021

bobbypage commented Mar 26, 2021

k8s-ci-robot commented Mar 26, 2021

ehashman commented Mar 26, 2021

BenTheElder commented Mar 26, 2021

BenTheElder Mar 26, 2021

amwat Mar 26, 2021

bobbypage Mar 26, 2021

amwat Mar 26, 2021

bobbypage Mar 26, 2021

amwat Mar 29, 2021

bobbypage Mar 26, 2021 •

edited

Loading

amwat Mar 26, 2021

bobbypage Mar 26, 2021 •

edited

Loading

amwat Mar 29, 2021

bobbypage Mar 31, 2021 •

edited

Loading

harche Apr 5, 2021

k8s-ci-robot commented Mar 31, 2021

BenTheElder commented Jun 18, 2021

spiffxp commented Jul 23, 2021

dims commented Aug 3, 2021

dims commented Aug 6, 2021

dims commented Aug 6, 2021

k8s-ci-robot commented Aug 6, 2021

k8s-ci-robot commented Aug 6, 2021 •

edited

Loading

dims commented Aug 6, 2021

[Node] Add node-e2e tester #103

[Node] Add node-e2e tester #103

Conversation

amwat commented Mar 5, 2021 • edited Loading

amwat commented Mar 8, 2021

amwat commented Mar 12, 2021

amwat commented Mar 15, 2021

bobbypage commented Mar 26, 2021

k8s-ci-robot commented Mar 26, 2021

ehashman commented Mar 26, 2021

BenTheElder commented Mar 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbypage Mar 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbypage Mar 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbypage Mar 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 31, 2021

BenTheElder commented Jun 18, 2021

spiffxp commented Jul 23, 2021

dims commented Aug 3, 2021

dims commented Aug 6, 2021

dims commented Aug 6, 2021

k8s-ci-robot commented Aug 6, 2021

k8s-ci-robot commented Aug 6, 2021 • edited Loading

dims commented Aug 6, 2021

amwat commented Mar 5, 2021 •

edited

Loading

bobbypage Mar 26, 2021 •

edited

Loading

bobbypage Mar 26, 2021 •

edited

Loading

bobbypage Mar 31, 2021 •

edited

Loading

k8s-ci-robot commented Aug 6, 2021 •

edited

Loading