Implement CLI and E2E testing using the KUDO test harness. #656

jbarrick-mesosphere · 2019-07-26T00:05:38Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This adds a kubectl setting to the KUDO test harness TestSuite and TestStep configurations to support running kubectl commands as a part of tests.

Which issue(s) this PR fixes:

Fixes #368 (support CLI commands)
Fixes #369 (support install kudo frameworks)
Fixes #520 (panic when executing test harness)

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Test harness now supports running kubectl commands as a part of tests.

jbarrick-mesosphere · 2019-07-26T00:07:40Z

pkg/test/utils/kubernetes.go

+}
+
+// Kubeconfig converts a rest.Config into a YAML kubeconfig and writes it to w
+func Kubeconfig(cfg *rest.Config, w io.Writer) error {


I wasn't able to find a method that could convert a rest.Config to a kubeconfig already. There's a number of cases that we need to be able to support:

Merged kubeconfig

In cluster config

Mocked control plane (just requires setting the Server setting)

Kind config

In some cases we have the path and could copy it.. but other times the path might not exist or be ambiguous and we would want to generate a kubeconfig.

But maybe there's a better way to do this.. Thoughts?

jbarrick-mesosphere · 2019-07-26T00:37:43Z

@alenkacz that create-operator-in-operator test started failing on here pretty frequently and I noticed that when the test passes it's always close to the timeout (28 seconds or so). I think the tests I've added here have pushed it over the edge to where it times out more often than not.. But I think I'm on to something. When I limit the test parallelism to 2 tests at once instead of 4, everything is a lot faster.

This leads me to wonder if either:

The test harness hammers the API server too hard with its polling.
KUDO's reconciliation is slow and having multiple Instances at once can delay reconciliation.

When running the tests, I noticed the PlanExecution controller trying to fetch lots of instances even once they were deleted, I wonder if it's doing a lot of potentially unnecessary work:

2019/07/25 17:32:53 InstanceController: Received Reconcile request for instance "oio-instance-inner-instance"
2019/07/25 17:32:53 InstanceController: Received Reconcile request for instance "oio-instance"
2019/07/25 17:32:53 PlanExecutionController: Error getting Instance operator-test-instance in kudo-test-fitting-goat: Instance.kudo.dev "operator-test-instance" not found
2019/07/25 17:32:54 PlanExecutionController: Error getting Instance icto-upgrade in kudo-test-teaching-bull: Instance.kudo.dev "icto-upgrade" not found
2019/07/25 17:32:55 PlanExecutionController: Error getting Instance icto-upgrade in kudo-test-teaching-bull: Instance.kudo.dev "icto-upgrade" not found
2019/07/25 17:32:56 PlanExecutionController: Error getting Instance toy1 in kudo-test-solid-lizard: Instance.kudo.dev "toy1" not found
2019/07/25 17:32:57 PlanExecutionController: Error getting Instance operator-test-instance in kudo-test-fitting-goat: Instance.kudo.dev "operator-test-instance" not found
2019/07/25 17:32:58 PlanExecutionController: Error getting Instance icto-no-trigger in kudo-test-vast-bunny: Instance.kudo.dev "icto-no-trigger" not found
2019/07/25 17:32:59 PlanExecutionController: Error getting Instance icto-fallback-to-deploy in kudo-test-diverse-joey: Instance.kudo.dev "icto-fallback-to-deploy" not found
2019/07/25 17:33:00 PlanExecutionController: Error getting Instance icto-no-trigger in kudo-test-vast-bunny: Instance.kudo.dev "icto-no-trigger" not found
2019/07/25 17:33:01 PlanExecutionController: Error getting Instance icto-upgrade-fallback-to-deploy in kudo-test-safe-bullfrog: Instance.kudo.dev "icto-upgrade-fallback-to-deploy" not found
2019/07/25 17:33:02 PlanExecutionController: Error getting Instance racy-instance in kudo-test-supreme-dassie: Instance.kudo.dev "racy-instance" not found
2019/07/25 17:33:03 PlanExecutionController: Error getting Instance immutable-client-ip-instance in kudo-test-healthy-grub: Instance.kudo.dev "immutable-client-ip-instance" not found
2019/07/25 17:33:04 PlanExecutionController: Error getting Instance icto-custom-trigger in kudo-test-wealthy-buffalo: Instance.kudo.dev "icto-custom-trigger" not found
2019/07/25 17:33:05 PlanExecutionController: Error getting Instance icto-custom-trigger in kudo-test-wealthy-buffalo: Instance.kudo.dev "icto-custom-trigger" not found
2019/07/25 17:33:06 PlanExecutionController: Error getting Instance icto-upgrade-fallback-to-deploy in kudo-test-safe-bullfrog: Instance.kudo.dev "icto-upgrade-fallback-to-deploy" not found
2019/07/25 17:33:07 PlanExecutionController: Received update event for an instance named: oio-instance-inner-instance

jbarrick-mesosphere · 2019-07-26T01:09:53Z

Yup, that did it. That last commit that makes Reconcile() in the PlanExecutionController not return an error cuts test execution time to 20 seconds:

--- PASS: kudo (19.92s)

That's down from the 76 seconds successful runs took before:

--- PASS: kudo (76.05s)

I'm going to drop that commit from here and put it into a separate PR.

jbarrick-mesosphere · 2019-07-26T01:15:59Z

Okay, submitted that fix as #657 :)

pkg/test/step.go

alenkacz

I have couple of questions but overall it looks good :)

alenkacz · 2019-07-29T07:17:00Z

Makefile

@@ -117,6 +117,11 @@ generate:
 generate-clean:
 	rm -rf hack/code-gen

+.PHONY: cli-test
+# Build CLI for tests
+cli-test:


why do we need a special build for tests? why is make cli not good enough?

make cli also runs all of the code generation and such so it takes quite a while to run (30 seconds or so?). As well as all of the linting (personally, I like to lint before I commit but that's just me). I thought it made sense to have a separate build that is quick for testing. What do you think?

I actually agree with you now that I think about it, everytime I am building CLI locally I hate when it stops on linting error. WDYT then about having just one build and one that is fast? If not we should at least comment why we need two.

So it actually occurs to me that we don't use make cli in CI or releasing at all - if it is used, it's used only by developers while developing.

Maybe we just drop the slow one then?

I'm gonna take your suggestion:

make cli make cli-fast

And then we can discuss what to do about the other targets separately.

pkg/test/case.go

pkg/test/harness.go

pkg/test/step.go

pkg/test/utils/kubernetes.go

test/integration/cli-install/00-install.yaml

Fixes kudobuilder#368 (support CLI commands) Fixes kudobuilder#369 (support install kudo frameworks) Fixes kudobuilder#520 (panic when executing test harness) This adds a `kubectl` setting to the KUDO test harness TestSuite and TestStep configurations to support running kubectl commands as a part of tests.

jbarrick-mesosphere · 2019-07-29T19:23:21Z

Okay, I think I addressed everything!

alenkacz

I have only one comment in the makefile, otherwise nice work 👏

kudo-ci · 2019-07-30T08:02:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alenkacz, jbarrick-mesosphere

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alenkacz,jbarrick-mesosphere]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

alenkacz · 2019-07-30T08:03:11Z

I've set the do not merge label to make sure you have time to see my makefile comment even though it's not a huge issue :)

jbarrick-mesosphere · 2019-07-30T17:38:02Z

/hold cancel

jbarrick-mesosphere added kind/feature component/testing labels Jul 26, 2019

jbarrick-mesosphere added this to the v0.4.0 milestone Jul 26, 2019

jbarrick-mesosphere requested review from kensipe, gkleiman, alenkacz, gerred and fabianbaier July 26, 2019 00:05

jbarrick-mesosphere requested a review from runyontr as a code owner July 26, 2019 00:05

jbarrick-mesosphere commented Jul 26, 2019

View reviewed changes

jbarrick-mesosphere force-pushed the cli-testing branch from a4953d6 to 445d3d1 Compare July 26, 2019 00:12

jbarrick-mesosphere force-pushed the cli-testing branch from fc6730b to fbf1ac0 Compare July 26, 2019 01:02

jbarrick-mesosphere force-pushed the cli-testing branch from fbf1ac0 to 445d3d1 Compare July 26, 2019 01:15

oliviabarrick mentioned this pull request Jul 26, 2019

Fix leftover bug after renaming of labels #658

Merged

alenkacz reviewed Jul 26, 2019

View reviewed changes

pkg/test/step.go Outdated Show resolved Hide resolved

jbarrick-mesosphere modified the milestones: v0.4.0, 0.5.0 Jul 26, 2019

alenkacz reviewed Jul 29, 2019

View reviewed changes

jbarrick-mesosphere force-pushed the cli-testing branch from 445d3d1 to e58ba10 Compare July 29, 2019 17:30

jbarrick-mesosphere added 4 commits July 29, 2019 10:46

Log files that are ignored by test harness.

e6a1bb4

factor DoKubectl -> testutils.RunKubectlCommands

7a43d13

Clean up kubectl argument parsing.

46c1b69

Fix broken logging and log stderr instead of writing it to an error.

8f0a9ad

alenkacz approved these changes Jul 30, 2019

View reviewed changes

kudo-ci added the approved label Jul 30, 2019

alenkacz added the do-not-merge/hold label Jul 30, 2019

make cli-test -> make cli-fast

c7d9011

kudo-ci added the size/XL label Jul 30, 2019

kudo-ci removed the do-not-merge/hold label Jul 30, 2019

make cli-test -> make cli-fast

c55f64a

jbarrick-mesosphere merged commit 3f45ece into kudobuilder:master Jul 31, 2019

jbarrick-mesosphere deleted the cli-testing branch July 31, 2019 00:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement CLI and E2E testing using the KUDO test harness. #656

Implement CLI and E2E testing using the KUDO test harness. #656

jbarrick-mesosphere commented Jul 26, 2019

jbarrick-mesosphere Jul 26, 2019

jbarrick-mesosphere commented Jul 26, 2019 •

edited

Loading

jbarrick-mesosphere commented Jul 26, 2019

jbarrick-mesosphere commented Jul 26, 2019

alenkacz left a comment

alenkacz Jul 29, 2019

jbarrick-mesosphere Jul 29, 2019

alenkacz Jul 30, 2019

jbarrick-mesosphere Jul 30, 2019

jbarrick-mesosphere Jul 30, 2019

jbarrick-mesosphere commented Jul 29, 2019

alenkacz left a comment

kudo-ci commented Jul 30, 2019

alenkacz commented Jul 30, 2019

jbarrick-mesosphere commented Jul 30, 2019

Implement CLI and E2E testing using the KUDO test harness. #656

Implement CLI and E2E testing using the KUDO test harness. #656

Conversation

jbarrick-mesosphere commented Jul 26, 2019

jbarrick-mesosphere Jul 26, 2019

Choose a reason for hiding this comment

jbarrick-mesosphere commented Jul 26, 2019 • edited Loading

jbarrick-mesosphere commented Jul 26, 2019

jbarrick-mesosphere commented Jul 26, 2019

alenkacz left a comment

Choose a reason for hiding this comment

alenkacz Jul 29, 2019

Choose a reason for hiding this comment

jbarrick-mesosphere Jul 29, 2019

Choose a reason for hiding this comment

alenkacz Jul 30, 2019

Choose a reason for hiding this comment

jbarrick-mesosphere Jul 30, 2019

Choose a reason for hiding this comment

jbarrick-mesosphere Jul 30, 2019

Choose a reason for hiding this comment

jbarrick-mesosphere commented Jul 29, 2019

alenkacz left a comment

Choose a reason for hiding this comment

kudo-ci commented Jul 30, 2019

alenkacz commented Jul 30, 2019

jbarrick-mesosphere commented Jul 30, 2019

jbarrick-mesosphere commented Jul 26, 2019 •

edited

Loading