Add upgrade test against previous server version to prevent regressions #6896

JeromeJu · 2023-07-04T17:39:39Z

Changes

This commit adds an upgrade test cases to test against previous server
version by adding a light-weight test checking the existing functionalities of
simple resources so as to prevent regressions introduced in the current
release. It creates a pipelineRun with a basic TaskRun and a basic
Pipeline with a Simple PipelineTask and examines the runs created are successful.

This helps prevent the regression where the client has introduced regressions towards the last release i.e. Regression: PipelineTaskRunSpec.Metadata incompatible with older webhooks

fixes: #5782

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

[n/a] Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
Has Tests included if any functionality added or changed
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
[n/a] Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
[n/a] Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

NONE

JeromeJu · 2023-07-04T17:41:24Z

/kind misc

vdemeester

Quick note, slightly related to this but not entirely: the e2e-tests-upgrade.sh runs based of latest releases (so 0.49 today for example) but not from latest LTS upgrade, which I think, would be interesting/important as well 👼🏼

vdemeester · 2023-07-05T05:47:55Z

test/upgrade_test.go

@@ -0,0 +1,180 @@
+//go:build e2e


As is it will be executed on all go_test_e2e run which.. might or might not be something we want ?

For now I think it might actually be nice to have it with the e2e tests. Since upgrade tests aren't run as part of CI, having this test in e2e tests would ensure breaking changes to these tests aren't checked in. It might be a better solution just to run upgrade tests in CI though.

lbernick

@JeromeJu can you please add more info to your commit message/PR description about the types of regressions these changes are intended to catch?

lbernick · 2023-07-05T12:58:27Z

test/upgrade_test.go

@@ -0,0 +1,180 @@
+//go:build e2e


For now I think it might actually be nice to have it with the e2e tests. Since upgrade tests aren't run as part of CI, having this test in e2e tests would ensure breaking changes to these tests aren't checked in. It might be a better solution just to run upgrade tests in CI though.

test/upgrade_test.go

test/e2e-tests-upgrade.sh

test/upgrade_test.go

lbernick · 2023-07-07T14:17:21Z

test/upgrade_test.go

+	}
+
+	expectedTaskRun := getExpectedTaskRunUpgrade(t, taskRunName, namespace, pipelineRunName, task1Name)
+	if d := cmp.Diff(expectedTaskRun, r, append([]cmp.Option{filterV1TaskRunSA, filterV1StepState, filterOwnerReference, filterV1TaskRunStatus, filterV1SidecarState, filterVolumeMountsName, filterTaskRunStatusProvenance}, filterV1TaskRunFields...)...); d != "" {


what does filterV1TaskRunStatus ignore? same with filterV1PipelineRunStatus. it sounds like it ignores the whole status; is it actually specific fields?

These are the filters in test dir for ignoring StartTime and CompletionTime at

pipeline/test/conversion_test.go

Line 41 in c97a7e6

filterV1beta1PipelineRunStatus = cmpopts.IgnoreFields(v1beta1.PipelineRunStatusFields{}, "StartTime", "CompletionTime")

.

I changed it to filterTaskRunStatusFields into this file.

test/upgrade_test.go

test/e2e-tests-upgrade.sh

test/upgrade_test.go

test/e2e-tests-upgrade.sh

This commit adds an upgrade test cases to test against previous server version by adding a light-weight test checking the existing functionalities of simple resources so as to prevent regressions introduced in the current release. It creates a pipelineRun with a basic TaskRun and a basic Pipeline with a Simple PipelineTask and examines the runs created are successful. This helps prevent the regression where the client has introduced regressions towards the last release i.e. [Regression: PipelineTaskRunSpec.Metadata incompatible with older webhooks](tektoncd#4913)

lbernick · 2023-07-07T20:57:00Z

test/upgrade_test.go

+	}
+
+	expectedPipelineRun := parse.MustParseV1PipelineRun(t, fmt.Sprintf(expectedSimplePipelineRunYaml, pipelineRunName, namespace, pipelineName, taskRunName))
+	if d := cmp.Diff(expectedPipelineRun, pr, append([]cmp.Option{filterV1PipelineRunStatus, filterV1PipelineRunSA, filterPipelineRunStatusFields}, filterV1PipelineRunFields...)...); d != "" {


Suggested change

if d := cmp.Diff(expectedPipelineRun, pr, append([]cmp.Option{filterV1PipelineRunStatus, filterV1PipelineRunSA, filterPipelineRunStatusFields}, filterV1PipelineRunFields...)...); d != "" {

if d := cmp.Diff(expectedPipelineRun, pr, filterV1PipelineRunStatus, filterV1PipelineRunSA, filterPipelineRunStatusFields, filterV1PipelineRunFields...); d != "" {

I am not sure if this could work according to https://pkg.go.dev/github.com/google/go-cmp/cmp#Diff seems we need to take in a slice instead of expanding it 🤔

If filterV1PipelineRunFields is an array, then I think it can't work as suggested @lbernick (because of Go 😅 )

tekton-robot · 2023-07-07T20:58:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lbernick

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lbernick]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

test/upgrade_test.go

Yongxuanzhang

/lgtm

tekton-robot added the release-note-none Denotes a PR that doesnt merit a release note. label Jul 4, 2023

tekton-robot requested review from dibyom and vdemeester July 4, 2023 17:39

tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 4, 2023

tekton-robot added the kind/misc Categorizes issue or PR as a miscellaneuous one. label Jul 4, 2023

vdemeester reviewed Jul 5, 2023

View reviewed changes

lbernick reviewed Jul 5, 2023

View reviewed changes

JeromeJu mentioned this pull request Jul 5, 2023

Refactor install_pipeline_crd SYSTEM_NAMESPACE env setup #6901

Merged

7 tasks

JeromeJu force-pushed the upgrade-regression branch 3 times, most recently from 384459a to 8d32c04 Compare July 7, 2023 13:54

lbernick reviewed Jul 7, 2023

View reviewed changes

test/e2e-tests-upgrade.sh Outdated Show resolved Hide resolved

JeromeJu force-pushed the upgrade-regression branch from 8a1d2ef to 2c0bdd3 Compare July 7, 2023 18:58

lbernick approved these changes Jul 7, 2023

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 7, 2023

Yongxuanzhang reviewed Jul 11, 2023

View reviewed changes

test/upgrade_test.go Show resolved Hide resolved

Yongxuanzhang reviewed Jul 17, 2023

View reviewed changes

tekton-robot assigned Yongxuanzhang Jul 17, 2023

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 17, 2023

tekton-robot merged commit 3364c72 into tektoncd:main Jul 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add upgrade test against previous server version to prevent regressions #6896

Add upgrade test against previous server version to prevent regressions #6896

JeromeJu commented Jul 4, 2023 •

edited

Loading

JeromeJu commented Jul 4, 2023

vdemeester left a comment

vdemeester Jul 5, 2023

lbernick Jul 5, 2023

lbernick left a comment

lbernick Jul 5, 2023

lbernick Jul 7, 2023

JeromeJu Jul 7, 2023

lbernick Jul 7, 2023

JeromeJu Jul 10, 2023

vdemeester Jul 10, 2023

tekton-robot commented Jul 7, 2023

Yongxuanzhang left a comment

	if d := cmp.Diff(expectedPipelineRun, pr, append([]cmp.Option{filterV1PipelineRunStatus, filterV1PipelineRunSA, filterPipelineRunStatusFields}, filterV1PipelineRunFields...)...); d != "" {
	if d := cmp.Diff(expectedPipelineRun, pr, filterV1PipelineRunStatus, filterV1PipelineRunSA, filterPipelineRunStatusFields, filterV1PipelineRunFields...); d != "" {

Add upgrade test against previous server version to prevent regressions #6896

Add upgrade test against previous server version to prevent regressions #6896

Conversation

JeromeJu commented Jul 4, 2023 • edited Loading

Changes

Submitter Checklist

Release Notes

JeromeJu commented Jul 4, 2023

vdemeester left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbernick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Jul 7, 2023

Yongxuanzhang left a comment

Choose a reason for hiding this comment

JeromeJu commented Jul 4, 2023 •

edited

Loading