fix: fallback to patch on scale conflict

Signed-off-by: Zach Aller <[email protected]> fix: switch to retry logic Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> retry experiments Signed-off-by: Zach Aller <[email protected]> remove TODO Signed-off-by: Zach Aller <[email protected]> remove accidental add Signed-off-by: Zach Aller <[email protected]> remove accidental add Signed-off-by: Zach Aller <[email protected]> add retry to setting revision Signed-off-by: Zach Aller <[email protected]> chore(deps): bump slsa-framework/slsa-github-generator from 1.10.0 to 2.0.0 (argoproj#3537) chore(deps): bump slsa-framework/slsa-github-generator Bumps [slsa-framework/slsa-github-generator](https://github.com/slsa-framework/slsa-github-generator) from 1.10.0 to 2.0.0. - [Release notes](https://github.com/slsa-framework/slsa-github-generator/releases) - [Changelog](https://github.com/slsa-framework/slsa-github-generator/blob/main/CHANGELOG.md) - [Commits](slsa-framework/slsa-github-generator@v1.10.0...v2.0.0) --- updated-dependencies: - dependency-name: slsa-framework/slsa-github-generator dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump sigstore/cosign-installer from 3.4.0 to 3.5.0 (argoproj#3522) Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 3.4.0 to 3.5.0. - [Release notes](https://github.com/sigstore/cosign-installer/releases) - [Commits](sigstore/cosign-installer@e1523de...59acb62) --- updated-dependencies: - dependency-name: sigstore/cosign-installer dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump golangci/golangci-lint-action from 4 to 5 (argoproj#3540) Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 4 to 5. - [Release notes](https://github.com/golangci/golangci-lint-action/releases) - [Commits](golangci/golangci-lint-action@v4...v5) --- updated-dependencies: - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> docs: provide recommendation for strategies (argoproj#3531) * docs: provide recommendation for strategies Signed-off-by: Kostis (Codefresh) <[email protected]> * docs: traffic manager clarifications Signed-off-by: Kostis (Codefresh) <[email protected]> * docs: explain canary with/out traffic manager Signed-off-by: Kostis (Codefresh) <[email protected]> * docs: add 3 columns on the comparison table Signed-off-by: Kostis (Codefresh) <[email protected]> --------- Signed-off-by: Kostis (Codefresh) <[email protected]> feat(dashboard): change the color of the current rollout step (argoproj#3526) I feel that having the current (running) step in a orange color is misleading, as orange usually means warning. This commit changes the color to the `$argo-running-color`. Signed-off-by: Alejandro López Sánchez <[email protected]> chore(deps): bump github.com/aws/aws-sdk-go-v2/service/cloudwatch from 1.37.0 to 1.38.0 (argoproj#3525) chore(deps): bump github.com/aws/aws-sdk-go-v2/service/cloudwatch Bumps [github.com/aws/aws-sdk-go-v2/service/cloudwatch](https://github.com/aws/aws-sdk-go-v2) from 1.37.0 to 1.38.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/service/s3/v1.38.0/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.37.0...service/s3/v1.38.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/cloudwatch dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> perform all of set revision actions on retry Signed-off-by: Zach Aller <[email protected]> fix variable Signed-off-by: Zach Aller <[email protected]> add retry counts to log Signed-off-by: Zach Aller <[email protected]> add retry counts to logs Signed-off-by: Zach Aller <[email protected]> clean logs, always dump controller e2e logs Signed-off-by: Zach Aller <[email protected]> lower timeout Signed-off-by: Zach Aller <[email protected]> bump timeout on e2e Signed-off-by: Zach Aller <[email protected]> retry on rollout conflict Signed-off-by: Zach Aller <[email protected]> don't reque on rs changes Signed-off-by: Zach Aller <[email protected]> reque rs Signed-off-by: Zach Aller <[email protected]> bump qps for e2e Signed-off-by: Zach Aller <[email protected]> fix gen-crd Signed-off-by: Zach Aller <[email protected]> switch to patch Signed-off-by: Zach Aller <[email protected]> switch to patch Signed-off-by: Zach Aller <[email protected]> add log Signed-off-by: Zach Aller <[email protected]> move log lines Signed-off-by: Zach Aller <[email protected]> Trigger Build Signed-off-by: Zach Aller <[email protected]> fix one e2e test Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> add test Signed-off-by: Zach Aller <[email protected]> chore(deps): bump actions/setup-go from 5.0.0 to 5.0.1 (argoproj#3552) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5.0.0 to 5.0.1. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v5.0.0...v5.0.1) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump codecov/codecov-action from 4.3.0 to 4.3.1 (argoproj#3550) Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.3.0 to 4.3.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v4.3.0...v4.3.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump google.golang.org/protobuf from 1.33.0 to 1.34.0 (argoproj#3548) Bumps google.golang.org/protobuf from 1.33.0 to 1.34.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> refactor Signed-off-by: Zach Aller <[email protected]> add test for updating rs revision Signed-off-by: Zach Aller <[email protected]> add retry for ephemeral metadata Signed-off-by: Zach Aller <[email protected]> clear some fields Signed-off-by: Zach Aller <[email protected]> add logs Signed-off-by: Zach Aller <[email protected]> refactor into function Signed-off-by: Zach Aller <[email protected]> change log Signed-off-by: Zach Aller <[email protected]> switch rollout update to patch fallback Signed-off-by: Zach Aller <[email protected]> siwtch ephemeral metadata sync to shared function Signed-off-by: Zach Aller <[email protected]> siwtch merge type Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> don't update status Signed-off-by: Zach Aller <[email protected]> switch rollout update to not use patch Signed-off-by: Zach Aller <[email protected]> change log Signed-off-by: Zach Aller <[email protected]> switch to small patch Signed-off-by: Zach Aller <[email protected]> some cleanup Signed-off-by: Zach Aller <[email protected]> remove not found rollout removal Signed-off-by: Zach Aller <[email protected]> working setup Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> fix test Signed-off-by: Zach Aller <[email protected]> small cleanup Signed-off-by: Zach Aller <[email protected]>
zachaller · May 7, 2024 · 7943d8d · 7943d8d
1 parent 125fc3d
commit 7943d8d
Show file tree

Hide file tree

Showing 13 changed files with 237 additions and 43 deletions.
diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml
@@ -95,4 +95,4 @@ jobs:
         with:
           name: e2e-controller-k8s-${{ matrix.kubernetes-minor-version }}.log
           path: /tmp/e2e-controller.log
-        if: ${{ failure() }}
+        if: ${{ always() }}
diff --git a/Makefile b/Makefile
@@ -22,7 +22,7 @@ DEV_IMAGE ?= false
 E2E_INSTANCE_ID ?= argo-rollouts-e2e
 E2E_TEST_OPTIONS ?=
 E2E_PARALLEL ?= 1
-E2E_WAIT_TIMEOUT ?= 120
+E2E_WAIT_TIMEOUT ?= 90
 GOPATH ?= $(shell go env GOPATH)
 
 # Global toolchain configuration
@@ -239,7 +239,7 @@ start-e2e: ## start e2e test environment
 
 .PHONY: test-e2e
 test-e2e: install-devtools-local
-	${DIST_DIR}/gotestsum --rerun-fails-report=rerunreport.txt --junitfile=junit.xml --format=testname --packages="./test/e2e" --rerun-fails=5 -- -timeout 60m -count 1 --tags e2e -p ${E2E_PARALLEL} -parallel ${E2E_PARALLEL} -v --short ./test/e2e ${E2E_TEST_OPTIONS}
+	${DIST_DIR}/gotestsum --rerun-fails-report=rerunreport.txt --junitfile=junit.xml --format=testname --packages="./test/e2e" --rerun-fails=5 -- -timeout 90m -count 1 --tags e2e -p ${E2E_PARALLEL} -parallel ${E2E_PARALLEL} -v --short ./test/e2e ${E2E_TEST_OPTIONS}
 
 .PHONY: test-unit
  test-unit: install-devtools-local ## run unit tests

diff --git a/experiments/replicaset.go b/experiments/replicaset.go
@@ -6,6 +6,8 @@ import (
 	"fmt"
 	"time"
 
+	"github.com/argoproj/argo-rollouts/utils/diff"
+
 	log "github.com/sirupsen/logrus"
 	appsv1 "k8s.io/api/apps/v1"
 	"k8s.io/apimachinery/pkg/api/errors"
@@ -287,16 +289,39 @@ func (ec *experimentContext) scaleReplicaSet(rs *appsv1.ReplicaSet, newScale int
 	sizeNeedsUpdate := oldScale != newScale
 	scaled := false
 	var err error
+	var updatedRS *appsv1.ReplicaSet
 	if sizeNeedsUpdate {
 		rsCopy := rs.DeepCopy()
 		*(rsCopy.Spec.Replicas) = newScale
-		rs, err = ec.kubeclientset.AppsV1().ReplicaSets(rsCopy.Namespace).Update(ctx, rsCopy, metav1.UpdateOptions{})
+
+		updatedRS, err = ec.kubeclientset.AppsV1().ReplicaSets(rsCopy.Namespace).Update(ctx, rsCopy, metav1.UpdateOptions{})
+		if err != nil {
+			if errors.IsConflict(err) {
+				ec.log.Infof("Conflict when updating replicaset %s, falling back to patch", rs.Name)
+
+				patchRS := appsv1.ReplicaSet{}
+				patchRS.Spec.Replicas = rsCopy.Spec.Replicas
+
+				patch, changed, err := diff.CreateTwoWayMergePatch(appsv1.ReplicaSet{}, patchRS, appsv1.ReplicaSet{})
+				if err != nil {
+					return scaled, nil, err
+				}
+
+				if changed {
+					ec.log.Infof("Patching expirment replicaset with patch: %s", string(patch))
+					updatedRS, err = ec.kubeclientset.AppsV1().ReplicaSets(rs.Namespace).Patch(ctx, rs.Name, patchtypes.StrategicMergePatchType, patch, metav1.PatchOptions{})
+					if err != nil {
+						return scaled, nil, err
+					}
+				}
+			}
+		}
 		if err == nil && sizeNeedsUpdate {
 			scaled = true
 			ec.recorder.Eventf(ec.ex, record.EventOptions{EventReason: conditions.ScalingReplicaSetReason}, "Scaled %s ReplicaSet %s from %d to %d", scalingOperation, rs.Name, oldScale, newScale)
 		}
 	}
-	return scaled, rs, err
+	return scaled, updatedRS, err
 }
 
 func newReplicaSetAnnotations(experimentName, templateName string) map[string]string {

diff --git a/hack/gen-crd-spec/main.go b/hack/gen-crd-spec/main.go
@@ -96,11 +96,12 @@ func NewCustomResourceDefinition() []*extensionsobj.CustomResourceDefinition {
 	// clean up stuff left by controller-gen
 	deleteFile("config/webhook/manifests.yaml")
 	deleteFile("config/webhook")
-	deleteFile("config/argoproj.io_analysisruns.yaml")
-	deleteFile("config/argoproj.io_analysistemplates.yaml")
-	deleteFile("config/argoproj.io_clusteranalysistemplates.yaml")
-	deleteFile("config/argoproj.io_experiments.yaml")
-	deleteFile("config/argoproj.io_rollouts.yaml")
+	deleteFile("config/crd/argoproj.io_analysisruns.yaml")
+	deleteFile("config/crd/argoproj.io_analysistemplates.yaml")
+	deleteFile("config/crd/argoproj.io_clusteranalysistemplates.yaml")
+	deleteFile("config/crd/argoproj.io_experiments.yaml")
+	deleteFile("config/crd/argoproj.io_rollouts.yaml")
+	deleteFile("config/crd")
 	deleteFile("config")
 
 	crds := []*extensionsobj.CustomResourceDefinition{}

diff --git a/rollout/canary.go b/rollout/canary.go
@@ -112,7 +112,7 @@ func (c *rolloutContext) reconcileCanaryStableReplicaSet() (bool, error) {
 	}
 	scaled, _, err := c.scaleReplicaSetAndRecordEvent(c.stableRS, desiredStableRSReplicaCount)
 	if err != nil {
-		return scaled, fmt.Errorf("failed to scaleReplicaSetAndRecordEvent in reconcileCanaryStableReplicaSet:L %w", err)
+		return scaled, fmt.Errorf("failed to scaleReplicaSetAndRecordEvent in reconcileCanaryStableReplicaSet: %w", err)
 	}
 	return scaled, err
 }

diff --git a/rollout/canary_test.go b/rollout/canary_test.go
@@ -8,6 +8,11 @@ import (
 	"testing"
 	"time"
 
+	"k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	k8stesting "k8s.io/client-go/testing"
+
 	"github.com/stretchr/testify/assert"
 	appsv1 "k8s.io/api/apps/v1"
 	v1 "k8s.io/api/apps/v1"
@@ -2141,3 +2146,90 @@ func TestCanaryReplicaAndSpecChangedTogether(t *testing.T) {
 	// check the canary one is updated
 	assert.NotEqual(t, originReplicas, int(*updated.Spec.Replicas))
 }
+
+func TestSyncRolloutWithConflictInScaleReplicaSet(t *testing.T) {
+	f := newFixture(t)
+	defer f.Close()
+
+	steps := []v1alpha1.CanaryStep{
+		{
+			SetWeight: int32Ptr(10),
+		}, {
+			Pause: &v1alpha1.RolloutPause{
+				Duration: v1alpha1.DurationFromInt(10),
+			},
+		},
+	}
+	r1 := newCanaryRollout("foo", 10, nil, steps, int32Ptr(1), intstr.FromInt(1), intstr.FromInt(0))
+	r2 := bumpVersion(r1)
+
+	rs1 := newReplicaSetWithStatus(r1, 9, 9)
+	rs2 := newReplicaSetWithStatus(r2, 1, 1)
+	f.kubeobjects = append(f.kubeobjects, rs1, rs2)
+	f.replicaSetLister = append(f.replicaSetLister, rs1, rs2)
+
+	f.rolloutLister = append(f.rolloutLister, r2)
+	f.objects = append(f.objects, r2)
+
+	f.expectPatchRolloutAction(r2)
+	f.expectUpdateReplicaSetAction(rs2) // attempt to scale replicaset but conflict
+	f.expectPatchReplicaSetAction(rs2)  // instead of update patch replicaset
+
+	key := fmt.Sprintf("%s/%s", r2.Namespace, r2.Name)
+	c, i, k8sI := f.newController(func() time.Duration { return 30 * time.Minute })
+
+	f.kubeclient.PrependReactor("update", "replicasets", func(action k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+		return true, &appsv1.ReplicaSet{}, errors.NewConflict(schema.GroupResource{
+			Group:    "Apps",
+			Resource: "ReplicaSet",
+		}, "", fmt.Errorf("test error"))
+	})
+
+	f.runController(key, true, false, c, i, k8sI)
+}
+
+func TestSyncRolloutWithConflictInSyncReplicaSetRevision(t *testing.T) {
+	f := newFixture(t)
+	defer f.Close()
+
+	steps := []v1alpha1.CanaryStep{
+		{
+			SetWeight: int32Ptr(10),
+		}, {
+			Pause: &v1alpha1.RolloutPause{
+				Duration: v1alpha1.DurationFromInt(10),
+			},
+		},
+	}
+	r1 := newCanaryRollout("foo", 3, nil, steps, int32Ptr(1), intstr.FromInt(1), intstr.FromInt(0))
+	r2 := bumpVersion(r1)
+
+	rs1 := newReplicaSetWithStatus(r1, 3, 3)
+	rs2 := newReplicaSetWithStatus(r2, 3, 3)
+	rs2.Annotations["rollout.argoproj.io/revision"] = "1"
+
+	f.kubeobjects = append(f.kubeobjects, rs1, rs2)
+	f.replicaSetLister = append(f.replicaSetLister, rs1, rs2)
+
+	f.rolloutLister = append(f.rolloutLister, r2)
+	f.objects = append(f.objects, r2)
+
+	key := fmt.Sprintf("%s/%s", r1.Namespace, r1.Name)
+	c, i, k8sI := f.newController(func() time.Duration { return 30 * time.Minute })
+
+	f.kubeclient.PrependReactor("update", "replicasets", func(action k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+		return true, &appsv1.ReplicaSet{}, errors.NewConflict(schema.GroupResource{
+			Group:    "Apps",
+			Resource: "ReplicaSet",
+		}, "", fmt.Errorf("test error"))
+	})
+
+	f.expectPatchRolloutAction(r2)
+	f.expectUpdateReplicaSetAction(rs1) // attempt to update replicaset revision but conflict
+	f.expectPatchReplicaSetAction(rs1)  // instead of update patch replicaset
+
+	f.expectUpdateReplicaSetAction(rs2) // attempt to scale replicaset but conflict
+	f.expectPatchReplicaSetAction(rs2)  // instead of update patch replicaset
+
+	f.runController(key, true, false, c, i, k8sI)
+}
diff --git a/rollout/controller.go b/rollout/controller.go
@@ -9,18 +9,21 @@ import (
 	"sync"
 	"time"
 
+	"github.com/argoproj/argo-rollouts/utils/diff"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 
 	"github.com/argoproj/argo-rollouts/pkg/apis/rollouts"
 	smiclientset "github.com/servicemeshinterface/smi-sdk-go/pkg/gen/client/split/clientset/versioned"
 	log "github.com/sirupsen/logrus"
 	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/errors"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
+	patchtypes "k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/validation/field"
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/client-go/dynamic"
@@ -937,3 +940,78 @@ func remarshalRollout(r *v1alpha1.Rollout) *v1alpha1.Rollout {
 	}
 	return &remarshalled
 }
+
+// updateReplicaSetWithPatch updates the replicaset using patch and on
+func (c *rolloutContext) updateReplicaSetFallbackToPatch(ctx context.Context, rs *appsv1.ReplicaSet) (*appsv1.ReplicaSet, error) {
+	rsCopy := rs.DeepCopy()
+	updatedRS, err := c.kubeclientset.AppsV1().ReplicaSets(rs.Namespace).Update(ctx, rs, metav1.UpdateOptions{})
+	if err != nil {
+		if errors.IsConflict(err) {
+			c.log.Infof("Conflict when updating replicaset %s, falling back to patch", rs.Name)
+
+			patchRS := appsv1.ReplicaSet{}
+			patchRS.Spec.Replicas = rsCopy.Spec.Replicas
+			patchRS.Annotations = rsCopy.Annotations
+			patchRS.Labels = rsCopy.Labels
+			patchRS.Spec.Template.Labels = rsCopy.Spec.Template.Labels
+			patchRS.Spec.Template.Annotations = rsCopy.Spec.Template.Annotations
+			patchRS.Spec.Selector = rsCopy.Spec.Selector
+
+			patch, changed, err := diff.CreateTwoWayMergePatch(appsv1.ReplicaSet{}, patchRS, appsv1.ReplicaSet{})
+			if err != nil {
+				return nil, err
+			}
+
+			if changed {
+				c.log.Infof("Patching replicaset with patch: %s", string(patch))
+				updatedRS, err = c.kubeclientset.AppsV1().ReplicaSets(rs.Namespace).Patch(ctx, rs.Name, patchtypes.StrategicMergePatchType, patch, metav1.PatchOptions{})
+				if err != nil {
+					return nil, err
+				}
+			}
+
+			err = c.replicaSetInformer.GetIndexer().Update(updatedRS)
+			if err != nil {
+				err = fmt.Errorf("error updating replicaset informer in scaleReplicaSet: %w", err)
+				return nil, err
+			}
+
+			return updatedRS, err
+		}
+	}
+
+	return updatedRS, err
+}
+
+// updateRolloutWithRetry updates the rollout with a retry if there is a conflict from an update operation, it runs the modifyRollout function to update a fresh rollout from the cluster.
+//func (c *rolloutContext) updateRolloutWithRetry(ctx context.Context, ro *v1alpha1.Rollout, modifyRollout func(ro *v1alpha1.Rollout) *v1alpha1.Rollout) (*v1alpha1.Rollout, error) {
+//	updatedRollout, err := c.argoprojclientset.ArgoprojV1alpha1().Rollouts(c.rollout.Namespace).Update(context.TODO(), c.rollout, metav1.UpdateOptions{})
+//	if err != nil {
+//		if errors.IsConflict(err) {
+//			retryCount := 0
+//			errRetry := retry.RetryOnConflict(retry.DefaultBackoff, func() error {
+//				retryCount++
+//				c.log.Infof("conflict when updating rollout %s, retrying the update operation with new rollout from cluster, attempt: %d", c.rollout.Name, retryCount)
+//				roGet, err := c.argoprojclientset.ArgoprojV1alpha1().Rollouts(c.rollout.Namespace).Get(context.TODO(), c.rollout.Name, metav1.GetOptions{})
+//				if err != nil {
+//					return fmt.Errorf("error getting rollout %s: %w", c.rollout.Name, err)
+//				}
+//
+//				roCopy := modifyRollout(roGet)
+//				updatedRollout, err = c.argoprojclientset.ArgoprojV1alpha1().Rollouts(c.rollout.Namespace).Update(context.TODO(), roCopy, metav1.UpdateOptions{})
+//				if err != nil {
+//					return err
+//				}
+//
+//				return nil
+//			})
+//			if errRetry != nil {
+//				return nil, errRetry
+//			}
+//		} else {
+//			c.log.WithError(err).Error("Error: updating rollout revision")
+//			return nil, err
+//		}
+//	}
+//	return updatedRollout, nil
+//}
diff --git a/rollout/controller_test.go b/rollout/controller_test.go
@@ -792,6 +792,12 @@ func (f *fixture) expectPatchServiceAction(s *corev1.Service, newLabel string) i
 	return len
 }
 
+func (f *fixture) expectGetReplicaSetAction(r *appsv1.ReplicaSet) int { //nolint:unused
+	len := len(f.kubeactions)
+	f.kubeactions = append(f.kubeactions, core.NewGetAction(schema.GroupVersionResource{Resource: "replicasets"}, r.Namespace, r.Name))
+	return len
+}
+
 func (f *fixture) expectCreateReplicaSetAction(r *appsv1.ReplicaSet) int {
 	len := len(f.kubeactions)
 	f.kubeactions = append(f.kubeactions, core.NewCreateAction(schema.GroupVersionResource{Resource: "replicasets"}, r.Namespace, r))

diff --git a/rollout/ephemeralmetadata.go b/rollout/ephemeralmetadata.go
@@ -2,7 +2,6 @@ package rollout
 
 import (
 	"context"
-	"fmt"
 
 	appsv1 "k8s.io/api/apps/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -82,14 +81,12 @@ func (c *rolloutContext) syncEphemeralMetadata(ctx context.Context, rs *appsv1.R
 	}
 
 	// 2. Update ReplicaSet so that any new pods it creates will have the metadata
-	rs, err = c.kubeclientset.AppsV1().ReplicaSets(modifiedRS.Namespace).Update(ctx, modifiedRS, metav1.UpdateOptions{})
+	rs, err = c.updateReplicaSetFallbackToPatch(ctx, modifiedRS)
 	if err != nil {
-		return fmt.Errorf("error updating replicaset in syncEphemeralMetadata: %w", err)
-	}
-	err = c.replicaSetInformer.GetIndexer().Update(rs)
-	if err != nil {
-		return fmt.Errorf("error updating replicaset informer in syncEphemeralMetadata: %w", err)
+		c.log.Infof("failed to sync ephemeral metadata %v to ReplicaSet %s: %v", podMetadata, rs.Name, err)
+		return err
 	}
+
 	c.log.Infof("synced ephemeral metadata %v to ReplicaSet %s", podMetadata, rs.Name)
 	return nil
 }