Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e2e TestWorkloadRef fails / LabelSelector is "null" & should be optional? #1675

Closed
derjust opened this issue Nov 28, 2021 · 7 comments · Fixed by #1678
Closed

e2e TestWorkloadRef fails / LabelSelector is "null" & should be optional? #1675

derjust opened this issue Nov 28, 2021 · 7 comments · Fixed by #1678
Assignees
Labels
bug Something isn't working

Comments

@derjust
Copy link
Contributor

derjust commented Nov 28, 2021

Summary

Following the docs/CONTRIBUTING.md documentation on master 47d59fa
on OSX, the TestWorkloadRef e2e fails - using make test-e2e E2E_TEST_OPTIONS="-testify.m ^TestWorkloadRef$" for all further descriptions below:
(For context, i found this while working on my PR #1577)

=== RUN   TestFunctionalSuite/TestWorkloadRef
time="2021-11-28T15:51:21-05:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
time="2021-11-28T15:51:25-05:00" level=info msg="service/rollout-bluegreen-active created\n" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment] met" rollout=rollout-ref-deployment
time="2021-11-28T15:51:30-05:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-28T15:51:30-05:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
FAIL  github.com/argoproj/argo-rollouts/test/e2e      22.245s

Diagnostics

What version of Argo Rollouts are you running?
Master

I suspect this is an issue with my setup but i can't point to where it is:

$ go version
go version go1.16.10 darwin/amd64
$ protoc --version
libprotoc 3.17.3

Also all the other software is installed via Homebrew:

$ brew install kubectl kustomize golangci-lint protobuf swagger-codegen

Warning: kubernetes-cli 1.22.4 is already installed and up-to-date.
To reinstall 1.22.4, run:
  brew reinstall kubernetes-cli
Warning: kustomize 4.4.1 is already installed and up-to-date.
To reinstall 4.4.1, run:
  brew reinstall kustomize
Warning: golangci-lint 1.43.0 is already installed and up-to-date.
To reinstall 1.43.0, run:
  brew reinstall golangci-lint
Warning: protobuf 3.17.3 is already installed and up-to-date.
To reinstall 3.17.3, run:
  brew reinstall protobuf
Warning: swagger-codegen 3.0.29 is already installed and up-to-date.
To reinstall 3.0.29, run:
  brew reinstall swagger-codegen

What am I missing?

Two observations:

First observation

Running make codegen in master creates a diff - which i didn't expect:

$ git diff --stat 

pkg/apis/rollouts/v1alpha1/generated.pb.go      | 1520 +++++++++++++++--------
pkg/apis/rollouts/v1alpha1/generated.proto      |   25 +-
pkg/apis/rollouts/v1alpha1/openapi_generated.go |   84 +-
3 files changed, 1114 insertions(+), 515 deletions(-)

all changes are of this style:

  • Changing the index(?) of the fileDescriptor_XXXXXXXX
  • Adding XXX_Unmarshal() / XXX_Marshal() methods
diff --git a/pkg/apis/rollouts/v1alpha1/generated.pb.go b/pkg/apis/rollouts/v1alpha1/generated.pb.go
index 4af51298..2d982911 100644
--- a/pkg/apis/rollouts/v1alpha1/generated.pb.go
+++ b/pkg/apis/rollouts/v1alpha1/generated.pb.go
@@ -48,10 +48,38 @@ var _ = math.Inf
 // proto package needs to be updated.
 const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package

+func (m *ALBStatus) Reset()      { *m = ALBStatus{} }
+func (*ALBStatus) ProtoMessage() {}
+func (*ALBStatus) Descriptor() ([]byte, []int) {
+       return fileDescriptor_e0e705f843545fab, []int{0}
+}
+func (m *ALBStatus) XXX_Unmarshal(b []byte) error {
+       return m.Unmarshal(b)
+}
+func (m *ALBStatus) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+       b = b[:cap(b)]
+       n, err := m.MarshalToSizedBuffer(b)
+       if err != nil {
+               return nil, err
+       }
+       return b[:n], nil
+}
+func (m *ALBStatus) XXX_Merge(src proto.Message) {
+       xxx_messageInfo_ALBStatus.Merge(m, src)
+}
+func (m *ALBStatus) XXX_Size() int {
+       return m.Size()
+}
+func (m *ALBStatus) XXX_DiscardUnknown() {
+       xxx_messageInfo_ALBStatus.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ALBStatus proto.InternalMessageInfo
+
 func (m *ALBTrafficRouting) Reset()      { *m = ALBTrafficRouting{} }
 func (*ALBTrafficRouting) ProtoMessage() {}
 func (*ALBTrafficRouting) Descriptor() ([]byte, []int) {
-       return fileDescriptor_e0e705f843545fab, []int{0}
+       return fileDescriptor_e0e705f843545fab, []int{1}
 }
 func (m *ALBTrafficRouting) XXX_Unmarshal(b []byte) error {
        return m.Unmarshal(b)
@@ -79,7 +107,7 @@ var xxx_messageInfo_ALBTrafficRouting proto.InternalMessageInfo
 func (m *AmbassadorTrafficRouting) Reset()      { *m = AmbassadorTrafficRouting{} }
 func (*AmbassadorTrafficRouting) ProtoMessage() {}
 func (*AmbassadorTrafficRouting) Descriptor() ([]byte, []int) {
-       return fileDescriptor_e0e705f843545fab, []int{1}
+       return fileDescriptor_e0e705f843545fab, []int{2}
 }
 func (m *AmbassadorTrafficRouting) XXX_Unmarshal(b []byte) error {
        return m.Unmarshal(b)
@@ -107,7 +135,7 @@ var xxx_messageInfo_AmbassadorTrafficRouting proto.InternalMessageInfo
 func (m *AnalysisRun) Reset()      { *m = AnalysisRun{} }
 func (*AnalysisRun) ProtoMessage() {}
 func (*AnalysisRun) Descriptor() ([]byte, []int) {
-       return fileDescriptor_e0e705f843545fab, []int{2}
+       return fileDescriptor_e0e705f843545fab, []int{3}
 }

With & without this diff though I looked at the error itself:

Second observation

When the e2e test fails, the controller output is

$ make start-e2e
....
ERRO[2021-11-28T15:48:50-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-28T15:48:50-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-28T15:48:50-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
....


with some additional logging from

diff --git a/rollout/temlateref.go b/rollout/temlateref.go
index 58cdf5d7..64055eca 100644
--- a/rollout/temlateref.go
+++ b/rollout/temlateref.go
@@ -250,7 +250,7 @@ func (r *informerBasedTemplateResolver) updateRolloutsReferenceAnnotation(obj in
                        ro.Spec.Template.Spec.Containers = []corev1.Container{}
                        _, err := r.argoprojclientset.ArgoprojV1alpha1().Rollouts(ro.Namespace).Update(context.TODO(), ro, v1.UpdateOptions{})
                        if err != nil {
-                               log.Errorf("Cannot update the workload-ref/annotation for %s/%s", ro.GetName(), ro.GetNamespace())
+                               log.Errorf("Cannot update the workload-ref/annotation for %s/%s: %v", ro.GetName(), ro.GetNamespace(), err)
                        }
                }
        }

The error message turns into

ERRO[2021-11-28T15:47:09-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default: Rollout.argoproj.io "rollout-ref-deployment" is invalid: spec.selector: Invalid value: "null": spec.selector in body must be of type object: "null"
ERRO[2021-11-28T15:47:09-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default: Rollout.argoproj.io "rollout-ref-deployment" is invalid: spec.selector: Invalid value: "null": spec.selector in body must be of type object: "null"
ERRO[2021-11-28T15:47:09-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default: Rollout.argoproj.io "rollout-ref-deployment" is invalid: spec.selector: Invalid value: "null": spec.selector in body must be of type object: "null"

I'm not sure if the selector should be set or not for the test.
But my understanding is that "null" is marshaled to the Rollouts().Update() call - which is for sure not what is desired.

Altering the type to make the selector optional seems to fix the issue:

diff --git a/pkg/apis/rollouts/v1alpha1/types.go b/pkg/apis/rollouts/v1alpha1/types.go
index 244db01d..8b51aa60 100644
--- a/pkg/apis/rollouts/v1alpha1/types.go
+++ b/pkg/apis/rollouts/v1alpha1/types.go
@@ -44,7 +44,7 @@ type RolloutSpec struct {
        // selected by this will be the ones affected by this rollout.
        // It must match the pod template's labels.
        // +optional
-       Selector *metav1.LabelSelector `json:"selector" protobuf:"bytes,2,opt,name=selector"`
+       Selector *metav1.LabelSelector `json:"selector,omitempty" protobuf:"bytes,2,opt,name=selector"`
        // Template describes the pods that will be created.
        // +optional
        Template corev1.PodTemplateSpec `json:"template" protobuf:"bytes,3,opt,name=template"`

as now the all tests pass.

But that seems to be the wrong solution?! Is this even an optional field?

My kubeconfig is set to use EKS/K8s 1.19

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

All those changes are also available in branch MasterFix


Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

@derjust derjust added the bug Something isn't working label Nov 28, 2021
@alexmt alexmt self-assigned this Nov 29, 2021
@huikang
Copy link
Member

huikang commented Nov 30, 2021

Could you try syncing up with the latest master? the test is passed for me

=== RUN   TestFunctionalSuite
time="2021-11-29T23:59:54-05:00" level=info msg="analysistemplate.argoproj.io/sleep-job unchanged\n"
=== RUN   TestFunctionalSuite/TestWorkloadRef
time="2021-11-29T23:59:54-05:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
W1129 23:59:55.026648   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 23:59:55.032620   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-29T23:59:55-05:00" level=info msg="service/rollout-bluegreen-active created\n" rollout=rollout-ref-deployment
time="2021-11-29T23:59:55-05:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T23:59:55-05:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T23:59:55-05:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T23:59:55-05:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment] met" rollout=rollout-ref-deployment
time="2021-11-29T23:59:55-05:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T23:59:55-05:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-30T00:00:04-05:00" level=info msg="Condition 'status=Healthy' met after 8s" rollout=rollout-ref-deployment
time="2021-11-30T00:00:04-05:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 1' met" rollout=rollout-ref-deployment
time="2021-11-30T00:00:04-05:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:7c69f96b6b] met" rollout=rollout-ref-deployment
time="2021-11-30T00:00:04-05:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-30T00:00:10-05:00" level=info msg="Condition 'status=Degraded' met after 5s" rollout=rollout-ref-deployment
time="2021-11-30T00:00:10-05:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 2 after workload ref updated' met" rollout=rollout-ref-deployment
time="2021-11-30T00:00:10-05:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-30T00:00:12-05:00" level=info msg="Condition 'status=Healthy' met after 2s" rollout=rollout-ref-deployment
time="2021-11-30T00:00:12-05:00" level=info msg="Updated rollout spec: {\"spec\":{\"workloadRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"non-existent-deploy\"}}}" rollout=rollout-ref-deployment
time="2021-11-30T00:00:12-05:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-30T00:00:12-05:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-30T00:00:12-05:00" level=info msg="Updated rollout spec: {\"spec\":{\"workloadRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"rollout-ref-deployment\"}}}" rollout=rollout-ref-deployment
time="2021-11-30T00:00:12-05:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="Condition 'status=Healthy' met after 1s" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:6cd5994b44] met" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="Rollout expectation 'Resolved template not persisted' met" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment configured\n" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="Condition 'status=Healthy' met after 0s" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration must be removed after switch to inline template' met" rollout=rollout-ref-deployment
time="2021-11-30T00:00:14-05:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
W1130 00:00:14.490868   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1130 00:00:14.494212   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
=== RUN   TestFunctionalSuite/TestWorkloadRefTemplate
time="2021-11-30T00:00:14-05:00" level=info msg="Deleting e2e-test-name=TestWorkloadRefTemplate"
W1130 00:00:14.571982   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1130 00:00:14.576693   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-30T00:00:14-05:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-30T00:00:15-05:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-30T00:00:15-05:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-30T00:00:15-05:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-30T00:00:15-05:00" level=info msg="Rollout expectation 'error due to workload ref and template' met" rollout=rollout-ref-deployment
time="2021-11-30T00:00:15-05:00" level=info msg="Deleting e2e-test-name=TestWorkloadRefTemplate"
W1130 00:00:15.267134   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1130 00:00:15.321008   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-30T00:00:15-05:00" level=info msg="Deleting e2e-test-name"
W1130 00:00:16.471126   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1130 00:00:16.522487   47020 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
--- PASS: TestFunctionalSuite (22.10s)
    --- PASS: TestFunctionalSuite/TestWorkloadRef (19.55s)
    --- PASS: TestFunctionalSuite/TestWorkloadRefTemplate (1.11s)
PASS
ok  	github.com/argoproj/argo-rollouts/test/e2e	23.115s


@derjust
Copy link
Contributor Author

derjust commented Nov 30, 2021

Thank you @huikang .
I'm running of the latest main i see:

argo-rollouts git:(47d59fa9) git log

commit 47d59fa9855ab737e40e21ef5938b2b6723cb899 (HEAD, upstream/master, origin/master, origin/HEAD, master)
Author: Siddharth Maloo <[email protected]>
Date:   Sat Nov 20 04:12:57 2021 +0530
...

Can you add logging and see if your selector is null (and it serializes differently) or if there is a selector that somewhat is missing on my end?

I also cleared out my ~/.kube/config and have only my EKS cluster present. The cluster itself was also cleaned and it has nothing running but the AWS system pods

$ kubectl config get-contexts

CURRENT   NAME                                                   CLUSTER                                                AUTHINFO                                               NAMESPACE
*         arn:aws:eks:us-east-1:922437464650:cluster/argo-test   arn:aws:eks:us-east-1:922437464650:cluster/argo-test   arn:aws:eks:us-east-1:922437464650:cluster/argo-test

The same error as before shows up in the e2e run:

$ make test-e2e E2E_TEST_OPTIONS="-testify.m ^TestWorkloadRef$"

go test -timeout 30m -v -count 1 --tags e2e -p 4 --short ./test/e2e -testify.m ^TestWorkloadRef
=== RUN   TestAnalysisSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAnalysisSuite (0.00s)
=== RUN   TestAWSSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAWSSuite (0.00s)
=== RUN   TestBlueGreenSuite
    suite.go:190: warning: no tests to run
--- PASS: TestBlueGreenSuite (0.00s)
=== RUN   TestCanarySuite
    suite.go:190: warning: no tests to run
--- PASS: TestCanarySuite (0.00s)
=== RUN   TestExperimentSuite
    suite.go:190: warning: no tests to run
--- PASS: TestExperimentSuite (0.00s)
=== RUN   TestFunctionalSuite
time="2021-11-29T21:36:43-08:00" level=info msg="analysistemplate.argoproj.io/sleep-job unchanged\n"
=== RUN   TestFunctionalSuite/TestWorkloadRef
time="2021-11-29T21:36:43-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
time="2021-11-29T21:36:48-08:00" level=info msg="service/rollout-bluegreen-active created\n" rollout=rollout-ref-deployment
time="2021-11-29T21:36:50-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T21:36:50-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T21:36:51-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T21:36:51-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment] met" rollout=rollout-ref-deployment
time="2021-11-29T21:36:54-08:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T21:36:54-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T21:37:21-08:00" level=info msg="Condition 'status=Healthy' met after 27s" rollout=rollout-ref-deployment
time="2021-11-29T21:37:22-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 1' met" rollout=rollout-ref-deployment
time="2021-11-29T21:37:22-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:659f6f45f7] met" rollout=rollout-ref-deployment
time="2021-11-29T21:37:22-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
^CFAIL  github.com/argoproj/argo-rollouts/test/e2e      69.771s
make: *** [test-e2e] Error 1

The controller logs show the error as above but before they also complain about a missing deployment

 make start-e2e

go run ./cmd/rollouts-controller/main.go --instance-id argo-rollouts-e2e --loglevel debug
INFO[2021-11-29T21:36:21-08:00] Argo Rollouts starting                        version=v99.99.99+unknown
INFO[2021-11-29T21:36:22-08:00] Creating event broadcaster
INFO[2021-11-29T21:36:22-08:00] Setting up event handlers
INFO[2021-11-29T21:36:22-08:00] Setting up experiments event handlers
INFO[2021-11-29T21:36:22-08:00] Setting up analysis event handlers
INFO[2021-11-29T21:36:23-08:00] Waiting for controller's informer caches to sync
INFO[2021-11-29T21:36:23-08:00] Enqueueing parent of default/rollout-ref-deployment-7c69f96b6b: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:36:23-08:00] Leaderelection get id Sebastians-MacBook-Pro.local_4d8d676f-4864-4aad-b551-2eb4afc29a52
INFO[2021-11-29T21:36:23-08:00] Starting Healthz Server at 0.0.0.0:8080
I1129 21:36:23.436918   88611 leaderelection.go:243] attempting to acquire leader lease kube-system/argo-rollouts-controller-lock...
INFO[2021-11-29T21:36:23-08:00] New leader elected: Sebastians-MacBook-Pro.local_f7ee4a4e-1965-4cd1-9335-efc2ad8931ad
INFO[2021-11-29T21:36:23-08:00] Starting Secondary Metric Server at 0.0.0.0:8090
I1129 21:36:40.288217   88611 leaderelection.go:253] successfully acquired lease kube-system/argo-rollouts-controller-lock
WARN[2021-11-29T21:36:40-08:00] Shutdown Secondary Metrics Server
WARN[2021-11-29T21:36:40-08:00] Starting Secondary Metric Server: http: Server closed
INFO[2021-11-29T21:36:40-08:00] Starting Controllers
INFO[2021-11-29T21:36:40-08:00] Started controller
INFO[2021-11-29T21:36:40-08:00] Starting Metric Server at 0.0.0.0:8090
INFO[2021-11-29T21:36:40-08:00] Starting Ingress workers
INFO[2021-11-29T21:36:40-08:00] Starting Experiment workers
INFO[2021-11-29T21:36:40-08:00] Started Ingress workers
INFO[2021-11-29T21:36:40-08:00] Started Experiment workers
INFO[2021-11-29T21:36:40-08:00] Starting Rollout workers
INFO[2021-11-29T21:36:40-08:00] Started Rollout workers
WARN[2021-11-29T21:36:40-08:00] Controller is running.
INFO[2021-11-29T21:36:40-08:00] Start processing                              resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:40-08:00] Starting Service workers
INFO[2021-11-29T21:36:40-08:00] Starting analysis workers
INFO[2021-11-29T21:36:40-08:00] Started Service workers
INFO[2021-11-29T21:36:40-08:00] Started 30 analysis workers
ERRO[2021-11-29T21:36:40-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:40-08:00] syncing service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:36:40-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"True","type":"Completed"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"ReplicaSet \"rollout-ref-deployment-7c69f96b6b\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-11-30T05:36:41Z","lastUpdateTime":"2021-11-30T05:36:41Z","message":"The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","reason":"InvalidSpec","status":"True","type":"InvalidSpec"}],"message":"InvalidSpec: The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","phase":"Degraded"}}  generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment time_ms=911.8547590000001
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 4 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.202174   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"True","type":"Completed"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"ReplicaSet \"rollout-ref-deployment-7c69f96b6b\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-11-30T05:36:41Z","lastUpdateTime":"2021-11-30T05:36:41Z","message":"The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","reason":"InvalidSpec","status":"True","type":"InvalidSpec"}],"message":"InvalidSpec: The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","phase":"Degraded"}}  generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment time_ms=118.99775799999999
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 6 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.321261   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.5124249999999999
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 7 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.321867   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.6034200000000001
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 8 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.358229   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.6285839999999999
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 9 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.491887   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.598853
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 10 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.753137   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:42-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:42-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:42-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.5721670000000001
ERRO[2021-11-29T21:36:42-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:42-08:00] rollout syncHandler queue retries: 11 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:42.269574   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:43-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:43-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=1.030473
ERRO[2021-11-29T21:36:43-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] rollout syncHandler queue retries: 12 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:43.297678   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:43-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:43-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] rollout deleted                               namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] syncing service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:36:43-08:00] cleaned service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:36:50-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917742 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:36:50Z","lastUpdateTime":"2021-11-30T05:36:50Z","message":"The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","reason":"InvalidSpec","status":"True","type":"InvalidSpec"}],"message":"InvalidSpec: The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","observedGeneration":"1","phase":"Degraded"}}  generation=1 namespace=default resourceVersion=13917742 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917742 rollout=rollout-ref-deployment time_ms=124.02361300000001
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 3 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.788293   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.386567
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 4 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.788728   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.5878209999999999
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 5 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.793829   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.458619
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 6 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.811531   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.5054649999999999
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 7 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.845289   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.566269
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 8 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.910015   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:51-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:51-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.6207349999999999
ERRO[2021-11-29T21:36:51-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] rollout syncHandler queue retries: 9 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:51.043820   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:51-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:51-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.484018
ERRO[2021-11-29T21:36:51-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] rollout syncHandler queue retries: 10 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:51.302380   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:51-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:51-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.5353209999999999
ERRO[2021-11-29T21:36:51-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] rollout syncHandler queue retries: 11 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:51.818618   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:52-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:52-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:52-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.44193699999999997
ERRO[2021-11-29T21:36:52-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:52-08:00] rollout syncHandler queue retries: 12 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:52.846020   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
INFO[2021-11-29T21:36:54-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:54-08:00] Updating replica set 'rollout-ref-deployment-5d466d9b5f' revision from 0 to 1  namespace=default rollout=rollout-ref-deployment
WARN[2021-11-29T21:36:55-08:00] Found a hash collision - bumped collisionCount (0->1) to resolve it  namespace=default rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:55-08:00] roCtx.reconcile err replicasets.apps "rollout-ref-deployment-5d466d9b5f" already exists  generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:55-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=280.29145500000004
ERRO[2021-11-29T21:36:55-08:00] rollout syncHandler error: replicasets.apps "rollout-ref-deployment-5d466d9b5f" already exists  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:55-08:00] rollout syncHandler queue retries: 13 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:55.179836   88611 controller.go:174] replicasets.apps "rollout-ref-deployment-5d466d9b5f" already exists
INFO[2021-11-29T21:36:55-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:55-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917776 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] Updating replica set 'rollout-ref-deployment-7c69f96b6b' revision from 0 to 1  namespace=default rollout=rollout-ref-deployment
WARN[2021-11-29T21:36:59-08:00] Found a hash collision - bumped collisionCount (1->2) to resolve it  namespace=default rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:59-08:00] roCtx.reconcile err replicasets.apps "rollout-ref-deployment-7c69f96b6b" already exists  generation=1 namespace=default resourceVersion=13917776 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917776 rollout=rollout-ref-deployment time_ms=271.331821
ERRO[2021-11-29T21:36:59-08:00] rollout syncHandler error: replicasets.apps "rollout-ref-deployment-7c69f96b6b" already exists  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] rollout syncHandler queue retries: 15 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:59.549857   88611 controller.go:174] replicasets.apps "rollout-ref-deployment-7c69f96b6b" already exists
INFO[2021-11-29T21:36:59-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:59-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917794 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Updating replica set 'rollout-ref-deployment-659f6f45f7' revision from 0 to 1  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Created ReplicaSet rollout-ref-deployment-659f6f45f7  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
DEBU[2021-11-29T21:37:09-08:00] No configured destinations for trigger: on-rollout-updated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Rollout updated to revision 1                 event_reason=RolloutUpdated namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917840", FieldPath:""}): type: 'Normal' reason: 'RolloutUpdated' Rollout updated to revision 1
INFO[2021-11-29T21:37:09-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917840", FieldPath:""}): type: 'Normal' reason: 'NewReplicaSetCreated' Created ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1)
DEBU[2021-11-29T21:37:09-08:00] No configured destinations for trigger: on-new-replica-set-created  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Created ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1)  event_reason=NewReplicaSetCreated namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:09-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Set rollout condition: &RolloutCondition{Type:Progressing,Status:True,LastUpdateTime:2021-11-29 21:37:09.807982 -0800 PST m=+48.190608464,LastTransitionTime:2021-11-29 21:37:09.807982 -0800 PST m=+48.190608568,Reason:NewReplicaSetCreated,Message:Created new replica set "rollout-ref-deployment-659f6f45f7",}  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:09-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:10-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:37:10-08:00] roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again  generation=1 namespace=default resourceVersion=13917794 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:10-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917794 rollout=rollout-ref-deployment time_ms=494.50115300000004
ERRO[2021-11-29T21:37:10-08:00] rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:10-08:00] rollout syncHandler queue retries: 21 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:37:10.046668   88611 controller.go:174] Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again
INFO[2021-11-29T21:37:19-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
DEBU[2021-11-29T21:37:19-08:00] No configured destinations for trigger: on-scaling-replica-set  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Scaled up ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1) from 0 to 1  event_reason=ScalingReplicaSet namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917842", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1) from 0 to 1
INFO[2021-11-29T21:37:19-08:00] New RS 'rollout-ref-deployment-659f6f45f7' is not ready to pause  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] skipping active service switch: New RS 'rollout-ref-deployment-659f6f45f7' is not fully saturated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Skipping analysis: isAborted: false, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: true  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917842", FieldPath:""}): type: 'Normal' reason: 'RolloutCompleted' Rollout completed update to revision 1 (659f6f45f7): Initial deploy
DEBU[2021-11-29T21:37:19-08:00] No configured destinations for trigger: on-rollout-completed  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Rollout completed update to revision 1 (659f6f45f7): Initial deploy  event_reason=RolloutCompleted namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Timed out (true) [last progress check: 2021-11-29 21:37:09 -0800 PST - now: 2021-11-29 21:37:19.890636 -0800 PST m=+58.273425729]  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Patched: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:37:19Z","lastUpdateTime":"2021-11-30T05:37:19Z","message":"ReplicaSet \"rollout-ref-deployment-659f6f45f7\" has timed out progressing.","reason":"ProgressDeadlineExceeded","status":"False","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:37:19Z","lastUpdateTime":"2021-11-30T05:37:19Z","message":"Rollout does not have minimum availability","reason":"AvailableReason","status":"False","type":"Available"}],"currentPodHash":"659f6f45f7","message":"ProgressDeadlineExceeded: ReplicaSet \"rollout-ref-deployment-659f6f45f7\" has timed out progressing.","selector":"app=rollout-ref-deployment","stableRS":"659f6f45f7","workloadObservedGeneration":"1"}}  generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment time_ms=338.75113699999997
INFO[2021-11-29T21:37:20-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:20-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] New RS 'rollout-ref-deployment-659f6f45f7' is not ready to pause  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] skipping active service switch: New RS 'rollout-ref-deployment-659f6f45f7' is not fully saturated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Patched: {"status":{"HPAReplicas":1,"conditions":[{"lastTransitionTime":"2021-11-30T05:37:19Z","lastUpdateTime":"2021-11-30T05:37:19Z","message":"Rollout does not have minimum availability","reason":"AvailableReason","status":"False","type":"Available"},{"lastTransitionTime":"2021-11-30T05:37:20Z","lastUpdateTime":"2021-11-30T05:37:20Z","message":"ReplicaSet \"rollout-ref-deployment-659f6f45f7\" is progressing.","reason":"ReplicaSetUpdated","status":"True","type":"Progressing"}],"message":"updated replicas are still becoming available","phase":"Progressing","replicas":1,"updatedReplicas":1}}  generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment time_ms=138.684145
INFO[2021-11-29T21:37:20-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:20-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] New RS 'rollout-ref-deployment-659f6f45f7' is not ready to pause  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] skipping active service switch: New RS 'rollout-ref-deployment-659f6f45f7' is not fully saturated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Timed out (false) [last progress check: 2021-11-29 21:37:20 -0800 PST - now: 2021-11-29 21:37:20.180558 -0800 PST m=+58.563352235]  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] No status changes. Skipping patch             generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Queueing up rollout for a progress after 4s   namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment time_ms=1.222034
INFO[2021-11-29T21:37:21-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] skipping pause: fast-tracked update           namespace=default rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] No configured destinations for trigger: on-switch-service  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Switched selector for service 'rollout-bluegreen-active' from '' to '659f6f45f7'  event_reason=SwitchService namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917892", FieldPath:""}): type: 'Normal' reason: 'SwitchService' Switched selector for service 'rollout-bluegreen-active' from '' to '659f6f45f7'
INFO[2021-11-29T21:37:21-08:00] Updating active selector ( -> 659f6f45f7)     namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] syncing service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:37:21-08:00] Patched: {"status":{"availableReplicas":1,"blueGreen":{"activeSelector":"659f6f45f7"},"conditions":[{"lastTransitionTime":"2021-11-30T05:37:21Z","lastUpdateTime":"2021-11-30T05:37:21Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"True","type":"Completed"},{"lastTransitionTime":"2021-11-30T05:37:20Z","lastUpdateTime":"2021-11-30T05:37:21Z","message":"ReplicaSet \"rollout-ref-deployment-659f6f45f7\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:37:21Z","lastUpdateTime":"2021-11-30T05:37:21Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"}],"message":null,"phase":"Healthy","readyReplicas":1,"selector":"app=rollout-ref-deployment,rollouts-pod-template-hash=659f6f45f7"}}  generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment time_ms=251.114889
INFO[2021-11-29T21:37:21-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:21-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciling stable ReplicaSet 'rollout-ref-deployment-659f6f45f7'  namespace=default rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] skipping pause: fast-tracked update           namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] No status changes. Skipping patch             generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment time_ms=0.938288
INFO[2021-11-29T21:37:21-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciling stable ReplicaSet 'rollout-ref-deployment-659f6f45f7'  namespace=default rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] skipping pause: fast-tracked update           namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] No status changes. Skipping patch             generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment time_ms=1.3315130000000002
ERRO[2021-11-29T21:37:22-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:37:22-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:37:22-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
^CINFO[2021-11-29T21:37:49-08:00] Shutting down workers
make: *** [start-e2e] Error 1
  • Is Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again an issue or is that just noise?
  • Not sure what the secret is used for from Failed to process: secret "argo-rollouts-notification-secret" not found resource=default/rollout-ref-deployment but i created such an (empty, generic) secret though the message still remains
$ kubectl get secrets -A | grep argo-rollouts

argo-test         argo-rollouts-notification-secret                    Opaque                                0      29m
argo-test         argo-rollouts-token-hspxc                            kubernetes.io/service-account-token   3      49d
argo-test         sh.helm.release.v1.argo-rollouts.v1                  helm.sh/release.v1                    1      49d
default           argo-rollouts-notification-secret                    Opaque                                0      32m
  • The missing deployment from deployments.apps "default/rollout-ref-deployment" not found namespace=default rollout=rollout-ref-deployment is present. Not sure what to make out of that error message?
$ kubectl get deployments -A

NAMESPACE     NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
default       rollout-ref-deployment         0/0     0            0           24m
kube-system   aws-load-balancer-controller   2/2     2            2           49d
kube-system   coredns                        2/2     2            2           49d
  • All those messages 'before' 2021-11-29T21:37:22-08:00 seem to be fine as the e2e is progressing through its asserts - which leaves me with only the Cannot update the workload-ref/annotation for rollout-ref-deployment/default - which with additional logging is the "null" value shown above.
    I'm utterly confused what else could be different

@derjust
Copy link
Contributor Author

derjust commented Nov 30, 2021

Alright - I did another test with minikube - which lets the e2e test succeed:

# Start minikube
$  minikube start
# Install CRDs etc
$ helm repo add argo https://argoproj.github.io/argo-helm
$ helm install argo-rollouts argo/argo-rollouts
# Delete released Argo version so that only the locally running one takes action
$ kubectl delete deployment -n default       argo-rollouts
$ make start-e2e

And in a different terminal run the e2e test successfully

make test-e2e E2E_TEST_OPTIONS="-testify.m ^TestWorkloadRef$"
go test -timeout 30m -v -count 1 --tags e2e -p 4 --short ./test/e2e -testify.m ^TestWorkloadRef
=== RUN   TestAnalysisSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAnalysisSuite (0.00s)
=== RUN   TestAWSSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAWSSuite (0.00s)
=== RUN   TestBlueGreenSuite
    suite.go:190: warning: no tests to run
--- PASS: TestBlueGreenSuite (0.00s)
=== RUN   TestCanarySuite
    suite.go:190: warning: no tests to run
--- PASS: TestCanarySuite (0.00s)
=== RUN   TestExperimentSuite
    suite.go:190: warning: no tests to run
--- PASS: TestExperimentSuite (0.00s)
=== RUN   TestFunctionalSuite
time="2021-11-29T22:15:04-08:00" level=info msg="analysistemplate.argoproj.io/sleep-job created\n"
=== RUN   TestFunctionalSuite/TestWorkloadRef
time="2021-11-29T22:15:04-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
W1129 22:15:04.064097   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:04.066680   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-29T22:15:04-08:00" level=info msg="service/rollout-bluegreen-active created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment] met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Condition 'status=Healthy' met after 4s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 1' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:7c69f96b6b] met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:15-08:00" level=info msg="Condition 'status=Degraded' met after 5s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:15-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 2 after workload ref updated' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:15-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Condition 'status=Healthy' met after 1s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Updated rollout spec: {\"spec\":{\"workloadRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"non-existent-deploy\"}}}" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Updated rollout spec: {\"spec\":{\"workloadRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"rollout-ref-deployment\"}}}" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Condition 'status=Healthy' met after 1s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:6cd5994b44] met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Rollout expectation 'Resolved template not persisted' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment configured\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Condition 'status=Healthy' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration must be removed after switch to inline template' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
W1129 22:15:18.549226   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:18.552968   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
=== RUN   TestFunctionalSuite/TestWorkloadRefTemplate
time="2021-11-29T22:15:18-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRefTemplate"
W1129 22:15:18.638484   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:18.641610   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-29T22:15:18-08:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Rollout expectation 'error due to workload ref and template' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRefTemplate"
W1129 22:15:19.328932   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:19.377819   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-29T22:15:19-08:00" level=info msg="Deleting e2e-test-name"
W1129 22:15:20.527551   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:20.577157   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
--- PASS: TestFunctionalSuite (17.29s)
    --- PASS: TestFunctionalSuite/TestWorkloadRef (14.57s)
    --- PASS: TestFunctionalSuite/TestWorkloadRefTemplate (1.10s)
=== RUN   TestIstioSuite
    suite.go:190: warning: no tests to run
--- PASS: TestIstioSuite (0.00s)
=== RUN   TestSMIIngressSuite
    suite.go:190: warning: no tests to run
--- PASS: TestSMIIngressSuite (0.00s)
=== RUN   TestSMISuite
    suite.go:190: warning: no tests to run
--- PASS: TestSMISuite (0.00s)
PASS
ok      github.com/argoproj/argo-rollouts/test/e2e      17.919s

After this I added a quick logging line

diff --git a/rollout/temlateref.go b/rollout/temlateref.go
index 58cdf5d7..5e1dd0a1 100644
--- a/rollout/temlateref.go
+++ b/rollout/temlateref.go
@@ -247,6 +247,9 @@ func (r *informerBasedTemplateResolver) updateRolloutsReferenceAnnotation(obj in
                if updated {
                        // update the annotation causes the rollout to be requeued and the template will be resolved to the referred
                        // workload during next reconciliation
+                       if ro.Spec.Selector == nil {
+                               log.Errorf(">>> Selector should not be nil! %v", ro)
+                       }
                        ro.Spec.Template.Spec.Containers = []corev1.Container{}
                        _, err := r.argoprojclientset.ArgoprojV1alpha1().Rollouts(ro.Namespace).Update(context.TODO(), ro, v1.UpdateOptions{})
                        if err != nil {

and sure enough, at the timestamp of the e2e the nil that would turn into "null" shows up:

ERRO[2021-11-29T22:19:55-08:00] >>> Selector should not be nil! &Rollout{ObjectMeta:{rollout-ref-deployment  default  1cb497a3-4469-49c9-be9c-ab4e4af16d7d 41748 5 2021-11-29 22:19:45 -0800 PST <nil> <nil> map[argo-rollouts.argoproj.io/controller-instance-id:argo-rollouts-e2e e2e-test-name:TestWorkloadRef] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"argoproj.io/v1alpha1","kind":"Rollout","metadata":{"annotations":{},"labels":{"argo-rollouts.argoproj.io/controller-instance-id":"argo-rollouts-e2e","e2e-test-name":"TestWorkloadRef"},"name":"rollout-ref-deployment","namespace":"default"},"spec":{"progressDeadlineSeconds":5,"replicas":1,"revisionHistoryLimit":2,"strategy":{"blueGreen":{"activeService":"rollout-bluegreen-active"}},"workloadRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"rollout-ref-deployment"}}}
 rollout.argoproj.io/revision:2 rollout.argoproj.io/workload-generation:3] [] []  [{kubectl-client-side-apply Update argoproj.io/v1alpha1 2021-11-29 22:19:45 -0800 PST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:argo-rollouts.argoproj.io/controller-instance-id":{},"f:e2e-test-name":{}}},"f:spec":{".":{},"f:progressDeadlineSeconds":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:strategy":{".":{},"f:blueGreen":{".":{},"f:activeService":{}}},"f:workloadRef":{".":{},"f:apiVersion":{},"f:kind":{},"f:name":{}}}}} {main Update argoproj.io/v1alpha1 2021-11-29 22:19:49 -0800 PST FieldsV1 {"f:metadata":{"f:annotations":{"f:rollout.argoproj.io/revision":{},"f:rollout.argoproj.io/workload-generation":{}}},"f:status":{"f:HPAReplicas":{},"f:availableReplicas":{},"f:blueGreen":{"f:activeSelector":{}},"f:readyReplicas":{},"f:selector":{},"f:stableRS":{}}}} {main Update argoproj.io/v1alpha1 2021-11-29 22:19:49 -0800 PST FieldsV1 {"f:status":{"f:conditions":{},"f:currentPodHash":{},"f:message":{},"f:observedGeneration":{},"f:phase":{},"f:replicas":{},"f:updatedReplicas":{},"f:workloadObservedGeneration":{}}}}]},Spec:RolloutSpec{Replicas:*1,Selector:nil,Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {[] [] [] []  <nil> <nil>  map[]   <nil>  false false false <nil> nil []   nil  [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil>}},MinReadySeconds:0,Strategy:RolloutStrategy{BlueGreen:&BlueGreenStrategy{ActiveService:rollout-bluegreen-active,PreviewService:,PreviewReplicaCount:nil,AutoPromotionEnabled:nil,AutoPromotionSeconds:0,MaxUnavailable:<nil>,ScaleDownDelaySeconds:nil,ScaleDownDelayRevisionLimit:nil,PrePromotionAnalysis:nil,AntiAffinity:nil,PostPromotionAnalysis:nil,PreviewMetadata:nil,ActiveMetadata:nil,AbortScaleDownDelaySeconds:nil,},Canary:nil,},RevisionHistoryLimit:*2,Paused:false,ProgressDeadlineSeconds:*5,RestartAt:<nil>,WorkloadRef:&ObjectRef{APIVersion:apps/v1,Kind:Deployment,Name:rollout-ref-deployment,},Analysis:nil,ProgressDeadlineAbort:false,},Status:RolloutStatus{Abort:false,PauseConditions:[]PauseCondition{},ControllerPause:false,AbortedAt:<nil>,CurrentPodHash:7f859ddcff,CurrentStepHash:,Replicas:2,UpdatedReplicas:1,ReadyReplicas:1,AvailableReplicas:1,CurrentStepIndex:nil,CollisionCount:*1,ObservedGeneration:5,Conditions:[]RolloutCondition{RolloutCondition{Type:Available,Status:True,LastUpdateTime:2021-11-29 22:19:49 -0800 PST,LastTransitionTime:2021-11-29 22:19:49 -0800 PST,Reason:AvailableReason,Message:Rollout has minimum availability,},RolloutCondition{Type:Completed,Status:False,LastUpdateTime:2021-11-29 22:19:49 -0800 PST,LastTransitionTime:2021-11-29 22:19:49 -0800 PST,Reason:RolloutCompleted,Message:RolloutCompleted,},RolloutCondition{Type:Progressing,Status:False,LastUpdateTime:2021-11-29 22:19:55 -0800 PST,LastTransitionTime:2021-11-29 22:19:55 -0800 PST,Reason:ProgressDeadlineExceeded,Message:ReplicaSet "rollout-ref-deployment-7f859ddcff" has timed out progressing.,},},Canary:CanaryStatus{CurrentStepAnalysisRunStatus:nil,CurrentBackgroundAnalysisRunStatus:nil,CurrentExperiment:,Weights:nil,},BlueGreen:BlueGreenStatus{PreviewSelector:,ActiveSelector:7c69f96b6b,ScaleUpPreviewCheckPoint:false,PrePromotionAnalysisRunStatus:nil,PostPromotionAnalysisRunStatus:nil,},HPAReplicas:1,Selector:app=rollout-ref-deployment,rollouts-pod-template-hash=7c69f96b6b,StableRS:7c69f96b6b,RestartedAt:<nil>,PromoteFull:false,Phase:Degraded,Message:ProgressDeadlineExceeded: ReplicaSet "rollout-ref-deployment-7f859ddcff" has timed out progressing.,WorkloadObservedGeneration:2,},}
INFO[2021-11-29T22:19:55-08:00] Start processing                              resource=default/rollout-ref-deployment

So - I guess minikube (and others?) are just more lenient and ignore such a 'broken' selector when sent by Argo but EKS isn't?

@huikang
Copy link
Member

huikang commented Nov 30, 2021

@derjust , thanks for the info. So the problem is reproducible in EKS.

@derjust
Copy link
Contributor Author

derjust commented Nov 30, 2021

Yes - though i don't know if the problem is EKS (rejecting the "null" or if minikube is just more leninent (which it shouldn't be). Leaning towards the first but I'm not sure

@alexmt
Copy link
Contributor

alexmt commented Dec 1, 2021

I think we should use k8s patch to change generation annotation, instead of update. The update might fail due to invalid fields and also due to concurrent modification. Created PR that changes code to use patch #1678

@derjust
Copy link
Contributor Author

derjust commented Dec 3, 2021

Thank you @alexmt !
I tested your branch locally with EKS and it works (as does my PR #1675.

As it is already approved & testes 🟢 - is there timeline to merge your PR?

Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants