Continue to set latest ready revision if error encountered #6592

taragu · 2020-01-21T21:51:41Z

/lint

Proposed Changes

Currently in the config reconciler, if the LRR does not exist, we don't continue the rest of the logic to set the LRR. This PR fixes it by logging the error and continuing.

Release Note

NONE

knative-prow-robot

@taragu: 0 warnings.

In response to this:

/lint

Fixes #6588

Proposed Changes

Currently in the config reconciler, if the LRR does not exist, we don't continue the rest of the logic to set the LRR. This PR fixes it by logging the error and continuing.

Release Note
NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vagababov · 2020-01-21T22:12:57Z

pkg/reconciler/configuration/configuration.go

+			start := lrr.Generation
+			var generations []string
+			for i := start; i <= int64(config.Generation); i++ {
+				generations = append(generations, strconv.FormatInt(i, 10))


Suggested change

generations = append(generations, strconv.FormatInt(i, 10))

generations = append(generations, strconv.Itoa(i))

Actually i here is int64 so FormatInt(..) might be better because then we don't have to cast int64 to int

taragu · 2020-01-22T13:56:30Z

/retest

taragu · 2020-01-23T18:54:14Z

@vagababov would you take a look at this again when you get a chance?

vagababov

I'll leave to Dan to review in depth.
/assign @dgerd

dgerd · 2020-01-24T19:30:05Z

pkg/reconciler/configuration/configuration.go

 	lister := c.revisionLister.Revisions(config.Namespace)
 	configSelector := labels.SelectorFromSet(map[string]string{
 		serving.ConfigurationLabelKey: config.Name,
 	})
 	if config.Status.LatestReadyRevisionName != "" {
 		lrr, err := lister.Get(config.Status.LatestReadyRevisionName)
+		// Record the error and continue because we still want to set the LRR to the correct revision


nit: sentences end in periods.

Suggested change

// Record the error and continue because we still want to set the LRR to the correct revision

// Record the error and continue because we still want to set the LRR to the correct revision.

dgerd · 2020-01-24T20:23:27Z

pkg/reconciler/configuration/configuration.go

+				generations,
+			)
+			if err != nil {
+				return nil, err


You also have an opportunity to exit out of this function here without returning a list of eligible Revisions.

IIUC this whole section is just limiting the search space down to Revisions between the previous Ready and the latest Created. Why not fallback to the broader search in this case as well?

taragu · 2020-02-20T17:12:36Z

Verified that this also fixes #6876

mattmoor · 2020-02-21T04:42:21Z

pkg/reconciler/configuration/configuration.go

-		for i := start; i <= int64(config.Generation); i++ {
-			generations = append(generations, strconv.FormatInt(i, 10))
-		}
+			logger.Errorf("Error getting latest ready revision %q: %v", config.Status.LatestReadyRevisionName, err)


Can we add a detailed comment here?

Something like:

Suggested change

logger.Errorf("Error getting latest ready revision %q: %v", config.Status.LatestReadyRevisionName, err)

// If the user deletes the LatestReadyRevision then this may return an error due to the

// dangling reference. Proceed to calculate the next-latest ready revision so that the

// caller can synthesize a new Revision at the current generation to replace the one deleted.

logger.Errorf("Error getting latest ready revision %q: %v", config.Status.LatestReadyRevisionName, err)

Currently in the config reconciler, if the LRR does not exist, we don't continue the rest of the logic to set the LRR. This PR fixes it by logging the error and continuing.

knative-metrics-robot · 2020-02-21T14:26:22Z

The following is the coverage report on the affected files.
Say /test pull-knative-serving-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/configuration/configuration.go	91.6%	93.5%	1.9

taragu · 2020-02-21T16:07:01Z

/retest

dprotaso · 2020-02-21T19:01:26Z

Looking through the lineage of the changes around getting/setting the latest ready revision I'm curious if things would be simpler if we were to relax the need for the LatestReadyRevision to always roll forward. This seems to be a conclusion of the discussions here

As an example, here's a timeline of a configuration's revisions:

time	revision	event
1	Revision-01	created
2	Revision-02	created
3	Revision-03	created
4	Revision-01	ready
5	Revision-02	ready
6	Revision-02	not ready

@mattmoor @dgerd Should the LastReadyRevision rollback to Revision-01?

dprotaso · 2020-02-21T17:55:39Z

pkg/reconciler/configuration/configuration_test.go

@@ -468,6 +468,35 @@ func TestReconcile(t *testing.T) {
 			Eventf(corev1.EventTypeNormal, "LatestReadyUpdate", "LatestReadyRevisionName updated to %q", "revnotready-00002"),
 		},
 		Key: "foo/revnotready",
+	}, {
+		Name: "current LRR doesn't exist, LCR is ready",


no acronyms please

dprotaso · 2020-02-21T19:08:14Z

Also based on the comment being added it seems like we're fine with rolling back the latest ready revision when the revision is deleted

Proceed to calculate the next-latest ready revision so that the caller can synthesize a new Revision at the current generation to replace the one deleted

dprotaso · 2020-02-26T19:09:27Z

A suggestion from @isdal is to prevent revision deletions under certain conditions (ie. they're the latest ready) but I think you'll still encounter this issue if your informer cache is stale and not up to date.

isdal · 2020-02-26T21:53:32Z

@whaught should take credit :-) He has some background around pros/cons here which I think apply. Preventing deletion is the least surprising option to customers in my mind, however it has a few downsides too.

whaught · 2020-02-26T21:58:56Z

The least surprising thing might go a little further. In the ValidatingWebHook, on delete you might look for a parented Route and throw an error if the Revision is still referenced (and actively serving or being migrated to).

This prevents users from causing an unintended interruption if they try to delete such a revision, only to have it be mysteriously taken down and then recreated.

dprotaso · 2020-02-27T15:31:20Z

Moving the discussion to an issue in order to unblock this PR (who's immediate purpose is to address the flakes)

/lgtm
/approve

knative-prow-robot · 2020-02-27T15:36:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dprotaso, taragu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/reconciler/OWNERS~~ [dprotaso]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dprotaso · 2020-02-27T19:23:53Z

/test pull-knative-serving-autotls-tests

knative-test-reporter-robot · 2020-02-27T19:54:22Z

The following jobs failed:

Test name	Triggers	Retries
pull-knative-serving-unit-tests		0/3

Failed non-flaky tests preventing automatic retry of pull-knative-serving-unit-tests:

pkg/reconciler/ingress.TestGlobalResyncOnUpdateGatewayConfigMap

taragu · 2020-02-27T20:04:00Z

/retest

googlebot added the cla: yes Indicates the PR's author has signed the CLA. label Jan 21, 2020

knative-prow-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 21, 2020

knative-prow-robot reviewed Jan 21, 2020

View reviewed changes

knative-prow-robot added the area/API API objects and controllers label Jan 21, 2020

knative-prow-robot requested review from tcnghia and vagababov January 21, 2020 21:52

vagababov reviewed Jan 21, 2020

View reviewed changes

taragu force-pushed the delete-latest-rev-6588 branch from 701b03c to 47f9764 Compare January 22, 2020 13:34

knative-prow-robot assigned dgerd Jan 23, 2020

vagababov reviewed Jan 23, 2020

View reviewed changes

dgerd reviewed Jan 24, 2020

View reviewed changes

taragu force-pushed the delete-latest-rev-6588 branch from 9e906b4 to 8fafe73 Compare February 20, 2020 16:59

mattmoor reviewed Feb 20, 2020

View reviewed changes

taragu force-pushed the delete-latest-rev-6588 branch from 0604b0a to cb53641 Compare February 20, 2020 17:07

taragu mentioned this pull request Feb 20, 2020

Wrong revision is picked up for traffic target marked as "latest" #6876

Closed

mattmoor reviewed Feb 21, 2020

View reviewed changes

Tara Gu added 4 commits February 21, 2020 09:20

Continue to set latest ready revision if error encountered

fb8d9ec

Currently in the config reconciler, if the LRR does not exist, we don't continue the rest of the logic to set the LRR. This PR fixes it by logging the error and continuing.

Review

72e1cb6

Fix rebase

317b902

Review

40791d8

taragu force-pushed the delete-latest-rev-6588 branch from 26bf704 to 40791d8 Compare February 21, 2020 14:21

dprotaso reviewed Feb 21, 2020

View reviewed changes

dprotaso mentioned this pull request Feb 27, 2020

Clarify LatestReadyRevision semantics #7077

Closed

knative-prow-robot assigned dprotaso Feb 27, 2020

knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 27, 2020

knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 27, 2020

knative-prow-robot merged commit 627315b into knative:master Feb 27, 2020

sayanh mentioned this pull request May 4, 2020

Fix EventMesh upgrade test on GKE kyma-project/kyma#8280

Closed

taragu deleted the delete-latest-rev-6588 branch July 29, 2020 11:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Continue to set latest ready revision if error encountered #6592

Continue to set latest ready revision if error encountered #6592

taragu commented Jan 21, 2020 •

edited

Loading

knative-prow-robot left a comment

vagababov Jan 21, 2020

taragu Jan 22, 2020

taragu Jan 22, 2020

taragu commented Jan 22, 2020

taragu commented Jan 23, 2020

vagababov left a comment

dgerd Jan 24, 2020

dgerd Jan 24, 2020

taragu commented Feb 20, 2020

mattmoor Feb 21, 2020

knative-metrics-robot commented Feb 21, 2020

taragu commented Feb 21, 2020

dprotaso commented Feb 21, 2020 •

edited

Loading

dprotaso Feb 21, 2020

dprotaso commented Feb 21, 2020

dprotaso commented Feb 26, 2020

isdal commented Feb 26, 2020

whaught commented Feb 26, 2020

dprotaso commented Feb 27, 2020

knative-prow-robot commented Feb 27, 2020

dprotaso commented Feb 27, 2020

knative-test-reporter-robot commented Feb 27, 2020

taragu commented Feb 27, 2020

	generations = append(generations, strconv.FormatInt(i, 10))
	generations = append(generations, strconv.Itoa(i))

	// Record the error and continue because we still want to set the LRR to the correct revision
	// Record the error and continue because we still want to set the LRR to the correct revision.

-			logger.Errorf("Error getting latest ready revision %q: %v", config.Status.LatestReadyRevisionName, err)
+			// If the user deletes the LatestReadyRevision then this may return an error due to the
+			// dangling reference.  Proceed to calculate the next-latest ready revision so that the
+			// caller can synthesize a new Revision at the current generation to replace the one deleted.
+			logger.Errorf("Error getting latest ready revision %q: %v", config.Status.LatestReadyRevisionName, err)

Continue to set latest ready revision if error encountered #6592

Continue to set latest ready revision if error encountered #6592

Conversation

taragu commented Jan 21, 2020 • edited Loading

Proposed Changes

knative-prow-robot left a comment

Choose a reason for hiding this comment

Proposed Changes

vagababov Jan 21, 2020

Choose a reason for hiding this comment

taragu Jan 22, 2020

Choose a reason for hiding this comment

taragu Jan 22, 2020

Choose a reason for hiding this comment

taragu commented Jan 22, 2020

taragu commented Jan 23, 2020

vagababov left a comment

Choose a reason for hiding this comment

dgerd Jan 24, 2020

Choose a reason for hiding this comment

dgerd Jan 24, 2020

Choose a reason for hiding this comment

taragu commented Feb 20, 2020

mattmoor Feb 21, 2020

Choose a reason for hiding this comment

knative-metrics-robot commented Feb 21, 2020

taragu commented Feb 21, 2020

dprotaso commented Feb 21, 2020 • edited Loading

dprotaso Feb 21, 2020

Choose a reason for hiding this comment

dprotaso commented Feb 21, 2020

dprotaso commented Feb 26, 2020

isdal commented Feb 26, 2020

whaught commented Feb 26, 2020

dprotaso commented Feb 27, 2020

knative-prow-robot commented Feb 27, 2020

dprotaso commented Feb 27, 2020

knative-test-reporter-robot commented Feb 27, 2020

taragu commented Feb 27, 2020

taragu commented Jan 21, 2020 •

edited

Loading

dprotaso commented Feb 21, 2020 •

edited

Loading