Add sources metadata conformance test #3319

devguyio · 2020-06-13T21:09:53Z

Proposed Changes

Add initial source conformance test for source label
Refactor ChannelTestRunner to a generic ComponentsTestRunner

Release Note

 - Add Eventing sources conformance test for validating required labels on Eventing source CRDs

knative-prow-robot · 2020-06-13T21:09:59Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

mattmoor

Produced via:
gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -type f -name '*.go' -print)

mattmoor

Produced via:
gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -type f -name '*.go' -print)

mattmoor

Produced via:
gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -type f -name '*.go' -print)

devguyio · 2020-06-14T08:35:12Z

/test all

matzew · 2020-06-15T08:00:42Z

test/conformance/helpers/broker_control_plane_test_helper.go

@@ -39,7 +39,7 @@ import (
 func BrokerV1Beta1ControlPlaneTestHelperWithChannelTestRunner(
 	t *testing.T,
 	brokerClass string,
-	channelTestRunner lib.ChannelTestRunner,
+	channelTestRunner lib.ComponentsTestRunner,


wanna change this great (IMO) rename here as well:

https://github.com/knative/eventing-contrib/blob/master/test/conformance/main_test.go#L30

yep, will do

matzew · 2020-06-15T08:02:02Z

test/conformance/helpers/channel_crd_metadata_test_helper.go

@@ -25,11 +25,16 @@ import (
 	"knative.dev/eventing/test/lib"
 )

+var channelLabels = map[string]string{


matzew · 2020-06-15T08:04:35Z

test/conformance/helpers/metadata.go

+	"knative.dev/eventing/test/lib"
+)
+
+func objectHasRequiredLabels(client *lib.Client, object metav1.TypeMeta, key string, value string) (bool, error) {


objectHasRequiredLabel ? singular?

matzew · 2020-06-15T08:06:32Z

test/conformance/helpers/source_crd_metadata_test_helper.go

+		// Each source MUST have the following:
+		//   label of duck.knative.dev/source: "true"
+		t.Run("Source CRD has required label", func(t *testing.T) {
+			yes, err := objectHasRequiredLabels(client, source, "duck.knative.dev/source", "true")


perhaps might move the duck.knative.dev/source to a map as well ?

mattmoor

Produced via:
gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -type f -name '*.go' -print)
goimports -w $(find -name '*.go' | grep -v vendor | grep -v third_party | grep -v wire_gen.go)

test/conformance/helpers/channel_crd_metadata_test_helper.go

antoineco

Moar conformance, noice!

antoineco · 2020-06-15T15:35:05Z

test/e2e_flags.go

+	SourceUsage  = "The names of the source type metas, separated by comma. " +
+		"Example: \"sources.knative.dev/v1alpha1:ApiServerSource," +
+		"sources.knative.dev/v1alpha1:PingSource\"."
+)
 // EventingFlags holds the command line flags specific to knative/eventing.
 //var EventingFlags *EventingEnvironmentFlags


This line is a leftover and should be removed.

test/e2e_flags.go

antoineco · 2020-06-15T15:41:09Z

test/conformance/helpers/channel_crd_metadata_test_helper.go

+	"messaging.knative.dev/subscribable": "true",
+	"duck.knative.dev/addressable":       "true",


Don't we have constants for those labels?

none that I know of, lemme know if you know them

Haven't found any so I created:

Add constants for supported Eventing labels and annotations #3321

Add constants for supported duck labels and annotations pkg#1407

subscribable is here:
pkg/apis/messaging/register.go
https://github.com/knative/eventing/blob/master/pkg/apis/messaging/register.go#L25

test/conformance/helpers/metadata.go

antoineco · 2020-06-15T16:07:06Z

test/e2e_flags.go

 // InitializeEventingFlags registers flags used by e2e tests, calling flag.Parse() here would fail in
 // go1.13+, see https://github.com/knative/test-infra/issues/1329 for details
 func InitializeEventingFlags() {
-	flag.Var(&EventingFlags.Channels, "channels", "The names of the channel type metas, separated by comma. Example: \"messaging.knative.dev/v1alpha1:InMemoryChannel,messaging.cloud.google.com/v1alpha1:Channel,messaging.knative.dev/v1alpha1:KafkaChannel\".")
-	flag.StringVar(&EventingFlags.BrokerClass, "brokerclass", "MTChannelBasedBroker", "Which brokerclass to test, requires the proper Broker implementation to have been installed, and only one value. brokerclass must be (for now) 'MTChannelBasedBroker'.")
+
+	flag.Var(&EventingFlags.Channels, "channels", ChannelUsage)
+	flag.StringVar(&EventingFlags.BrokerClass, "brokerclass", "MTChannelBasedBroker", BrokerUsage)
+	flag.Var(&EventingFlags.Sources, "sources", SourceUsage)
 	flag.Parse()


calling flag.Parse() here would fail

flag.Parse()

More seriously, the comment is probably outdated.

I've no context of this whole comment so I'd like to keep it out of this PR scope. I'll follow up on slack on it though since I've came across it. Good catch 👍

Created #3322

antoineco · 2020-06-15T16:10:01Z

test/flags/channels.go

+	return fmt.Sprint(*channels)
+}
+
+// Set converts the input string to Channels.


"Appends a new Channel corresponding to the input string" seems to describe the actual code better.

antoineco · 2020-06-15T16:10:20Z

test/flags/sources.go

+	return fmt.Sprint(*sources)
+}
+
+// Set converts the input string to Sources.


"Appends a new Source corresponding to the input string" seems to describe the actual code better.

I'll refactor this in a separate PR and actually rename the method to Append. Good catch 👍 . I'm not fully familiar with the code base so I'm not sure who's using who and I wanna limit the blast radius of this PR.

Can we at least have an accurate comment in this PR?

antoineco · 2020-06-15T16:10:45Z

test/flags/sources.go

+			Kind:       split[1],
+		}
+		if !isValidSource(tm.Kind) {
+			log.Fatalf("The given source name %q is invalid, tests cannot be run.\n", source)


Mixed case within the function "Source/source"

antoineco · 2020-06-15T16:11:12Z

test/flags/channels.go

+			Kind:       split[1],
+		}
+		if !isValidChannel(tm.Kind) {
+			log.Fatalf("The given channel name %q is invalid, tests cannot be run.\n", channel)


Mixed case within the function "Channel/channel"

👍 Generally I think the error messages in the test folder can use some love

antoineco · 2020-06-15T17:02:52Z

/lgtm

antoineco · 2020-06-15T21:16:16Z

test/lib/config.go

@@ -37,7 +37,7 @@ var MessagingChannelTypeMeta = metav1.TypeMeta{
 	Kind:       resources.ChannelKind,
 }

-// ChannelFeatureMap saves the channel-features mapping.
+// ComponentFeatureMap saves the channel-features mapping.


@Abd4llA this one remains

knative-prow-robot · 2020-06-15T21:16:43Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Abd4llA
To complete the pull request process, please assign chizhg
You can assign the PR to them by writing /assign @chizhg in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

devguyio · 2020-06-16T08:00:45Z

Holding until refactoring is done in eventing-contrib
/hold

knative-prow-robot · 2020-06-16T12:46:25Z

@Abd4llA: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

antoineco · 2020-06-16T12:47:40Z

/lgtm cancel

matzew · 2020-06-16T15:31:44Z

How about we split this work into two PRs ?

one that does the refactoring and the testlib import-alias
second that actually adds the first sources test

While PR 1 was merged, the same changes could already be applied to eventing-contrib, and than indepently send just the work for the sources

devguyio · 2020-06-22T11:05:09Z

Split into two PRs:

devguyio added 3 commits June 13, 2020 23:05

Introduce sources e2e flag

e9133e3

Refactor ChannelTestRunner to ComponentTestRunner

6c310ee

Add source labels conformance test

15cafa9

knative-prow-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 13, 2020

googlebot added the cla: yes Indicates the PR's author has signed the CLA. label Jun 13, 2020

knative-prow-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. area/test-and-release Test infrastructure, tests or release labels Jun 13, 2020

knative-prow-robot requested review from chaodaiG and steuhs June 13, 2020 21:10

mattmoor reviewed Jun 13, 2020

View reviewed changes

Refactor out object labels verification code in conformance tests

1cd5601

mattmoor reviewed Jun 14, 2020

View reviewed changes

Fix missing header in conformance helper test file

6271e51

mattmoor reviewed Jun 14, 2020

View reviewed changes

matzew reviewed Jun 15, 2020

View reviewed changes

Move object labels check to a shared package and sort imports

7a4c038

mattmoor reviewed Jun 15, 2020

View reviewed changes

test/conformance/helpers/channel_crd_metadata_test_helper.go Outdated Show resolved Hide resolved

devguyio marked this pull request as ready for review June 15, 2020 13:49

knative-prow-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 15, 2020

Remove extra empty line in channel crd metadata conf test

040361e

antoineco approved these changes Jun 15, 2020

View reviewed changes

antoineco mentioned this pull request Jun 15, 2020

Fix comment of test.InitializeEventingFlags func #3322

Closed

Some code cleanup in test after review comments

c3b9953

devguyio added 2 commits June 15, 2020 18:51

Add more descriptive comments to test flags Set methods

c09a0d3

Clean up conformance tests helper metadata

0c4f116

knative-prow-robot assigned antoineco Jun 15, 2020

knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 15, 2020

antoineco suggested changes Jun 15, 2020

View reviewed changes

knative-prow-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 16, 2020

knative-prow-robot removed the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2020

devguyio closed this Jun 22, 2020

		"messaging.knative.dev/subscribable": "true",
		"duck.knative.dev/addressable": "true",

Add sources metadata conformance test #3319

Add sources metadata conformance test #3319

Conversation

devguyio commented Jun 13, 2020 • edited Loading

Proposed Changes

knative-prow-robot commented Jun 13, 2020

mattmoor left a comment

Choose a reason for hiding this comment

mattmoor left a comment

Choose a reason for hiding this comment

mattmoor left a comment

Choose a reason for hiding this comment

devguyio commented Jun 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoor left a comment

Choose a reason for hiding this comment

antoineco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devguyio Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devguyio Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

antoineco commented Jun 15, 2020

Choose a reason for hiding this comment

knative-prow-robot commented Jun 15, 2020

devguyio commented Jun 16, 2020

knative-prow-robot commented Jun 16, 2020

antoineco commented Jun 16, 2020

matzew commented Jun 16, 2020

devguyio commented Jun 22, 2020

devguyio commented Jun 13, 2020 •

edited

Loading

devguyio Jun 15, 2020 •

edited

Loading

devguyio Jun 15, 2020 •

edited

Loading