Failed
=== RUN   TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress
time="2023-07-31T00:57:02Z" level=info msg="Deleting e2e-test-name=TestALBExperimentStepNoSetWeightMultiIngress"
time="2023-07-31T00:57:02Z" level=info msg="service/alb-rollout-root created\n" rollout=alb-rollout
time="2023-07-31T00:57:03Z" level=info msg="service/alb-rollout-canary created\n" rollout=alb-rollout
time="2023-07-31T00:57:03Z" level=info msg="service/alb-rollout-stable created\n" rollout=alb-rollout
time="2023-07-31T00:57:03Z" level=info msg="ingress.networking.k8s.io/alb-rollout-multi-ingress-1 created\n" rollout=alb-rollout
time="2023-07-31T00:57:04Z" level=info msg="ingress.networking.k8s.io/alb-rollout-multi-ingress-2 created\n" rollout=alb-rollout
time="2023-07-31T00:57:04Z" level=info msg="rollout.argoproj.io/alb-rollout created\n" rollout=alb-rollout
time="2023-07-31T00:57:04Z" level=info msg="Waiting for condition: status=Healthy" rollout=alb-rollout
time="2023-07-31T00:57:05Z" level=info msg="Condition 'status=Healthy' met after 1s" rollout=alb-rollout
time="2023-07-31T00:57:05Z" level=info msg="Experiment expectation 'experiment count == 0' met" rollout=alb-rollout
time="2023-07-31T00:57:05Z" level=info msg="Updated rollout pod spec: 2023-07-31T00:57:05.830937981Z" rollout=alb-rollout
time="2023-07-31T00:57:05Z" level=info msg="Sleeping 10s" rollout=alb-rollout
    require.go:72: 
        	Error Trace:	/home/runner/go/pkg/mod/github.com/tj/[email protected]/require.go:72
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/e2e/aws_test.go:298
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/fixtures/then.go:28
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/e2e/aws_test.go:287
        	Error:      	Not equal: 
        	            	expected: "{\"Type\":\"forward\",\"ForwardConfig\":{\"TargetGroups\":[{\"ServiceName\":\"alb-rollout-canary\",\"ServicePort\":\"80\",\"Weight\":0},{\"ServiceName\":\"\",\"ServicePort\":\"80\",\"Weight\":20},{\"ServiceName\":\"\",\"ServicePort\":\"80\",\"Weight\":20},{\"ServiceName\":\"alb-rollout-stable\",\"ServicePort\":\"80\",\"Weight\":60}]}}"
        	            	actual  : "{\"Type\":\"forward\",\"ForwardConfig\":{\"TargetGroups\":[{\"ServiceName\":\"alb-rollout-canary\",\"ServicePort\":\"80\",\"Weight\":0},{\"ServicePort\":\"80\",\"Weight\":20},{\"ServicePort\":\"80\",\"Weight\":20},{\"ServiceName\":\"alb-rollout-stable\",\"ServicePort\":\"80\",\"Weight\":60}]}}"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-{"Type":"forward","ForwardConfig":{"TargetGroups":[{"ServiceName":"alb-rollout-canary","ServicePort":"80","Weight":0},{"ServiceName":"","ServicePort":"80","Weight":20},{"ServiceName":"","ServicePort":"80","Weight":20},{"ServiceName":"alb-rollout-stable","ServicePort":"80","Weight":60}]}}
        	            	+{"Type":"forward","ForwardConfig":{"TargetGroups":[{"ServiceName":"alb-rollout-canary","ServicePort":"80","Weight":0},{"ServicePort":"80","Weight":20},{"ServicePort":"80","Weight":20},{"ServiceName":"alb-rollout-stable","ServicePort":"80","Weight":60}]}}
        	Test:       	TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress
Name:            alb-rollout
Namespace:       default
Status:          �[94m◌�[0m Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/1
  SetWeight:     0
  ActualWeight:  0
Images:          nginx:1.19-alpine (�[32mstable�[0m, �[0mΣ:experiment-alb-canary�[0m, �[0mΣ:experiment-alb-stable�[0m)
Replicas:
  Desired:       1
  Current:       1
  Updated:       0
  Ready:         1
  Available:     1

NAME                                                                  KIND        STATUS         AGE  INFO
⟳ alb-rollout                                                         Rollout     �[94m◌�[0m Progressing  11s  
├──# revision:2                                                                                       
│  ├──⧉ �[33malb-rollout-799676d889�[0m                                        ReplicaSet  �[0m•�[0m ScaledDown   10s  �[33mcanary�[0m
│  └──Σ �[94malb-rollout-799676d889-2-0�[0m                                    Experiment  �[94m◌�[0m Running      10s  
│     ├──⧉ alb-rollout-799676d889-2-0-experiment-alb-canary           ReplicaSet  �[32m✔�[0m Healthy      10s  
│     │  └──□ alb-rollout-799676d889-2-0-experiment-alb-canary-ppm7w  Pod         �[32m✔�[0m Running      10s  ready:1/1
│     └──⧉ alb-rollout-799676d889-2-0-experiment-alb-stable           ReplicaSet  �[32m✔�[0m Healthy      10s  
│        └──□ alb-rollout-799676d889-2-0-experiment-alb-stable-hnzt4  Pod         �[32m✔�[0m Running      9s   ready:1/1
└──# revision:1                                                                                       
   └──⧉ �[32malb-rollout-d6f4fdcdc�[0m                                         ReplicaSet  �[32m✔�[0m Healthy      11s  �[32mstable�[0m
      └──□ alb-rollout-d6f4fdcdc-nf4xt                                Pod         �[32m✔�[0m Running      11s  ready:1/1

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: "2"
  creationTimestamp: "2023-07-31T00:57:04Z"
  generation: 3
  labels:
    argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e
    e2e-test-name: TestALBExperimentStepNoSetWeightMultiIngress
  name: alb-rollout
  namespace: default
  resourceVersion: "4305"
  uid: d7c8105e-e6eb-4f74-a61e-ef0d99c6ca1e
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alb-rollout
  strategy:
    canary:
      canaryService: alb-rollout-canary
      stableService: alb-rollout-stable
      steps:
      - experiment:
          duration: 15s
          templates:
          - metadata: {}
            name: experiment-alb-canary
            specRef: canary
            weight: 20
          - metadata: {}
            name: experiment-alb-stable
            specRef: stable
            weight: 20
      trafficRouting:
        alb:
          ingresses:
          - alb-rollout-multi-ingress-1
          - alb-rollout-multi-ingress-2
          rootService: alb-rollout-root
          servicePort: 80
  template:
    metadata:
      annotations:
        update: "2023-07-31T00:57:05.830937981Z"
      creationTimestamp: null
      labels:
        app: alb-rollout
    spec:
      containers:
      - image: nginx:1.19-alpine
        name: alb-rollout
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 16Mi
status:
  HPAReplicas: 1
  availableReplicas: 1
  blueGreen: {}
  canary:
    currentExperiment: alb-rollout-799676d889-2-0
    weights:
      additional:
      - weight: 20
      - weight: 20
      canary:
        podTemplateHash: 799676d889
        serviceName: alb-rollout-canary
        weight: 0
      stable:
        podTemplateHash: d6f4fdcdc
        serviceName: alb-rollout-stable
        weight: 60
  conditions:
  - lastTransitionTime: "2023-07-31T00:57:05Z"
    lastUpdateTime: "2023-07-31T00:57:05Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2023-07-31T00:57:05Z"
    lastUpdateTime: "2023-07-31T00:57:05Z"
    message: Rollout is not healthy
    reason: RolloutHealthy
    status: "False"
    type: Healthy
  - lastTransitionTime: "2023-07-31T00:57:05Z"
    lastUpdateTime: "2023-07-31T00:57:05Z"
    message: RolloutCompleted
    reason: RolloutCompleted
    status: "False"
    type: Completed
  - lastTransitionTime: "2023-07-31T00:57:04Z"
    lastUpdateTime: "2023-07-31T00:57:05Z"
    message: Created new replica set "alb-rollout-799676d889"
    reason: NewReplicaSetCreated
    status: "True"
    type: Progressing
  currentPodHash: 799676d889
  currentStepHash: 558cc8bf4d
  currentStepIndex: 0
  message: more replicas need to be updated
  observedGeneration: "3"
  phase: Progressing
  readyReplicas: 1
  replicas: 1
  selector: app=alb-rollout
  stableRS: d6f4fdcdc

2023-07-31T00:57:04Z         Normal    RolloutAddedToInformer    Rollout resource added to informer: default/alb-rollout
2023-07-31T00:57:04Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 1 (d6f4fdcdc)
2023-07-31T00:57:04Z         Normal    RolloutUpdated            Rollout updated to revision 1
2023-07-31T00:57:04Z         Normal    NewReplicaSetCreated      Created ReplicaSet alb-rollout-d6f4fdcdc (revision 1)
2023-07-31T00:57:04Z (x2)    Normal    TrafficWeightUpdated      Traffic weight updated to 0
2023-07-31T00:57:04Z         Normal    ScalingReplicaSet         Scaled up ReplicaSet alb-rollout-d6f4fdcdc (revision 1) from 0 to 1
2023-07-31T00:57:04Z         Normal    RolloutCompleted          Rollout completed update to revision 1 (d6f4fdcdc): Initial deploy
2023-07-31T00:57:05Z         Normal    SwitchService             Switched selector for service 'alb-rollout-stable' from '' to 'd6f4fdcdc'
2023-07-31T00:57:05Z         Normal    SwitchService             Switched selector for service 'alb-rollout-canary' from '' to 'd6f4fdcdc'
2023-07-31T00:57:05Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 2 (799676d889)
2023-07-31T00:57:05Z         Normal    RolloutUpdated            Rollout updated to revision 2
2023-07-31T00:57:05Z         Normal    NewReplicaSetCreated      Created ReplicaSet alb-rollout-799676d889 (revision 2)
2023-07-31T00:57:05Z (x2)    Normal    TrafficWeightUpdated      Traffic weight updated 
2023-07-31T00:57:05Z         Normal    ExperimentCreated         Created Experiment 'alb-rollout-799676d889-2-0'
2023-07-31T00:57:07Z (x2)    Normal    PatchingALBIngress        Updating Ingress `alb-rollout-multi-ingress-1` to desiredWeight '0'
2023-07-31T00:57:07Z (x2)    Normal    PatchingALBIngress        Updating Ingress `alb-rollout-multi-ingress-2` to desiredWeight '0'
2023-07-31T00:57:07Z         Normal    TrafficWeightUpdated      Traffic weight updated additional: [{20  } {20  }]

time="2023-07-31T00:57:15Z" level=info msg="Deleting e2e-test-name=TestALBExperimentStepNoSetWeightMultiIngress"
    --- FAIL: TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress (14.18s)

Failed
=== RUN   TestAWSSuite
--- FAIL: TestAWSSuite (125.41s)

Failed
=== RUN   TestCanarySuite/TestCanaryDynamicStableScale
time="2023-07-31T01:00:37Z" level=info msg="Deleting e2e-test-name=TestCanaryDynamicStableScale"
time="2023-07-31T01:00:38Z" level=info msg="service/dynamic-stable-scale-root created\n" rollout=dynamic-stable-scale
time="2023-07-31T01:00:38Z" level=info msg="service/dynamic-stable-scale-canary created\n" rollout=dynamic-stable-scale
time="2023-07-31T01:00:38Z" level=info msg="service/dynamic-stable-scale-stable created\n" rollout=dynamic-stable-scale
time="2023-07-31T01:00:39Z" level=info msg="ingress.networking.k8s.io/dynamic-stable-scale-ingress created\n" rollout=dynamic-stable-scale
time="2023-07-31T01:00:39Z" level=info msg="rollout.argoproj.io/dynamic-stable-scale created\n" rollout=dynamic-stable-scale
time="2023-07-31T01:00:40Z" level=info msg="Marked 4 revision 1 pods as ready" rollout=dynamic-stable-scale
time="2023-07-31T01:00:40Z" level=info msg="Waiting for condition: status=Healthy" rollout=dynamic-stable-scale
time="2023-07-31T01:00:43Z" level=info msg="Condition 'status=Healthy' met after 2s" rollout=dynamic-stable-scale
time="2023-07-31T01:00:43Z" level=info msg="Updated rollout pod spec: 2023-07-31T01:00:43.08917959Z" rollout=dynamic-stable-scale
time="2023-07-31T01:00:44Z" level=info msg="Marked 1 revision 2 pods as ready" rollout=dynamic-stable-scale
time="2023-07-31T01:00:44Z" level=info msg="Waiting for condition: status=Paused" rollout=dynamic-stable-scale
time="2023-07-31T01:00:44Z" level=info msg="Condition 'status=Paused' met after 0s" rollout=dynamic-stable-scale
time="2023-07-31T01:00:44Z" level=info msg="Sleeping 2s" rollout=dynamic-stable-scale
time="2023-07-31T01:00:46Z" level=info msg="Pod expectation 'revision:1 pod count == 3' met" rollout=dynamic-stable-scale
time="2023-07-31T01:00:46Z" level=info msg="Pod expectation 'revision:2 pod count == 1' met" rollout=dynamic-stable-scale
time="2023-07-31T01:00:46Z" level=info msg="Promoted rollout" rollout=dynamic-stable-scale
time="2023-07-31T01:00:47Z" level=info msg="Marked 2 revision 2 pods as ready" rollout=dynamic-stable-scale
time="2023-07-31T01:00:47Z" level=info msg="Waiting for condition: status.currentStepIndex=3" rollout=dynamic-stable-scale
time="2023-07-31T01:00:49Z" level=info msg="Condition 'status.currentStepIndex=3' met after 1s" rollout=dynamic-stable-scale
time="2023-07-31T01:00:49Z" level=info msg="Sleeping 2s" rollout=dynamic-stable-scale
time="2023-07-31T01:00:51Z" level=info msg="Pod expectation 'revision:1 pod count == 1' met" rollout=dynamic-stable-scale
time="2023-07-31T01:00:51Z" level=info msg="Pod expectation 'revision:2 pod count == 3' met" rollout=dynamic-stable-scale
time="2023-07-31T01:00:51Z" level=info msg="Aborted rollout" rollout=dynamic-stable-scale
time="2023-07-31T01:00:52Z" level=info msg="Marked 2 revision 1 pods as ready" rollout=dynamic-stable-scale
time="2023-07-31T01:00:52Z" level=info msg="Waiting for condition: rev=2 podcount=1" rollout=dynamic-stable-scale
time="2023-07-31T01:00:54Z" level=info msg="Condition 'rev=2 podcount=1' met after 2s" rollout=dynamic-stable-scale
time="2023-07-31T01:00:55Z" level=info msg="Pod expectation 'revision:1 pod count == 4' met" rollout=dynamic-stable-scale
time="2023-07-31T01:00:56Z" level=info msg="Marked 1 revision 1 pods as ready" rollout=dynamic-stable-scale
time="2023-07-31T01:00:56Z" level=info msg="Waiting for condition: rev=2 podcount=0" rollout=dynamic-stable-scale
time="2023-07-31T01:00:56Z" level=info msg="Condition 'rev=2 podcount=0' met after 0s" rollout=dynamic-stable-scale
    canary_test.go:620: Expected dynamic-stable-scale-canary selector: map[app:dynamic-stable-scale rollouts-pod-template-hash:868d98995b]. Actual: map[app:dynamic-stable-scale rollouts-pod-template-hash:68c5f4fff5]
Name:            dynamic-stable-scale
Namespace:       default
Status:          �[31m✖�[0m Degraded
Message:         RolloutAborted: Rollout aborted update to revision 2
Strategy:        Canary
  Step:          0/4
  SetWeight:     0
  ActualWeight:  0
Images:          nginx:1.19-alpine (�[32mstable�[0m)
Replicas:
  Desired:       4
  Current:       4
  Updated:       0
  Ready:         4
  Available:     4

NAME                                              KIND        STATUS         AGE  INFO
⟳ dynamic-stable-scale                            Rollout     �[31m✖�[0m Degraded     17s  
├──# revision:2                                                                   
│  └──⧉ �[33mdynamic-stable-scale-68c5f4fff5�[0m           ReplicaSet  �[0m•�[0m ScaledDown   13s  �[33mcanary�[0m
│     └──□ dynamic-stable-scale-68c5f4fff5-s6vbd  Pod         �[94m◌�[0m Terminating  13s  ready:1/1
└──# revision:1                                                                   
   └──⧉ �[32mdynamic-stable-scale-868d98995b�[0m           ReplicaSet  �[32m✔�[0m Healthy      17s  �[32mstable�[0m
      ├──□ dynamic-stable-scale-868d98995b-8pb24  Pod         �[32m✔�[0m Running      17s  ready:1/1
      ├──□ dynamic-stable-scale-868d98995b-mt9x7  Pod         �[32m✔�[0m Running      5s   ready:1/1
      ├──□ dynamic-stable-scale-868d98995b-td6xv  Pod         �[32m✔�[0m Running      5s   ready:1/1
      └──□ dynamic-stable-scale-868d98995b-xwnqb  Pod         �[32m✔�[0m Running      5s   ready:1/1

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: "2"
  creationTimestamp: "2023-07-31T01:00:39Z"
  generation: 2
  labels:
    argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e
    e2e-test-name: TestCanaryDynamicStableScale
  name: dynamic-stable-scale
  namespace: default
  resourceVersion: "6458"
  uid: 6bebfe45-4871-492e-8c0d-08683fe47cb8
spec:
  replicas: 4
  selector:
    matchLabels:
      app: dynamic-stable-scale
  strategy:
    canary:
      canaryService: dynamic-stable-scale-canary
      dynamicStableScale: true
      stableService: dynamic-stable-scale-stable
      steps:
      - setWeight: 25
      - pause: {}
      - setWeight: 75
      - pause: {}
      trafficRouting:
        alb:
          ingress: dynamic-stable-scale-ingress
          rootService: dynamic-stable-scale-root
          servicePort: 80
  template:
    metadata:
      annotations:
        update: "2023-07-31T01:00:43.08917959Z"
      creationTimestamp: null
      labels:
        app: dynamic-stable-scale
    spec:
      containers:
      - image: nginx:1.19-alpine
        name: dynamic-stable-scale
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 16Mi
      readinessGates:
      - conditionType: argoproj.io/e2e-readiness
status:
  HPAReplicas: 5
  abort: true
  abortedAt: "2023-07-31T01:00:51Z"
  availableReplicas: 5
  blueGreen: {}
  canary:
    weights:
      canary:
        podTemplateHash: 68c5f4fff5
        serviceName: dynamic-stable-scale-canary
        weight: 0
      stable:
        podTemplateHash: 868d98995b
        serviceName: dynamic-stable-scale-stable
        weight: 100
  conditions:
  - lastTransitionTime: "2023-07-31T01:00:43Z"
    lastUpdateTime: "2023-07-31T01:00:43Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2023-07-31T01:00:43Z"
    lastUpdateTime: "2023-07-31T01:00:43Z"
    message: Rollout is not healthy
    reason: RolloutHealthy
    status: "False"
    type: Healthy
  - lastTransitionTime: "2023-07-31T01:00:43Z"
    lastUpdateTime: "2023-07-31T01:00:43Z"
    message: RolloutCompleted
    reason: RolloutCompleted
    status: "False"
    type: Completed
  - lastTransitionTime: "2023-07-31T01:00:49Z"
    lastUpdateTime: "2023-07-31T01:00:49Z"
    message: Rollout is paused
    reason: RolloutPaused
    status: "True"
    type: Paused
  - lastTransitionTime: "2023-07-31T01:00:51Z"
    lastUpdateTime: "2023-07-31T01:00:51Z"
    message: Rollout aborted update to revision 2
    reason: RolloutAborted
    status: "False"
    type: Progressing
  currentPodHash: 68c5f4fff5
  currentStepHash: 7c5b668566
  currentStepIndex: 0
  message: 'RolloutAborted: Rollout aborted update to revision 2'
  observedGeneration: "2"
  phase: Degraded
  readyReplicas: 5
  replicas: 5
  selector: app=dynamic-stable-scale
  stableRS: 868d98995b
  updatedReplicas: 1

2023-07-31T01:00:39Z         Normal    RolloutAddedToInformer    Rollout resource added to informer: default/dynamic-stable-scale
2023-07-31T01:00:39Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 1 (868d98995b)
2023-07-31T01:00:39Z         Normal    RolloutUpdated            Rollout updated to revision 1
2023-07-31T01:00:39Z         Normal    NewReplicaSetCreated      Created ReplicaSet dynamic-stable-scale-868d98995b (revision 1)
2023-07-31T01:00:39Z         Normal    PatchingALBIngress        Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '0'
2023-07-31T01:00:39Z (x2)    Normal    TrafficWeightUpdated      Traffic weight updated to 0
2023-07-31T01:00:39Z         Normal    ScalingReplicaSet         Scaled up ReplicaSet dynamic-stable-scale-868d98995b (revision 1) from 0 to 4
2023-07-31T01:00:39Z         Normal    RolloutCompleted          Rollout completed update to revision 1 (868d98995b): Initial deploy
2023-07-31T01:00:43Z         Normal    SwitchService             Switched selector for service 'dynamic-stable-scale-stable' from '' to '868d98995b'
2023-07-31T01:00:43Z         Normal    SwitchService             Switched selector for service 'dynamic-stable-scale-canary' from '' to '868d98995b'
2023-07-31T01:00:43Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 2 (68c5f4fff5)
2023-07-31T01:00:43Z         Normal    RolloutUpdated            Rollout updated to revision 2
2023-07-31T01:00:43Z         Normal    NewReplicaSetCreated      Created ReplicaSet dynamic-stable-scale-68c5f4fff5 (revision 2)
2023-07-31T01:00:43Z (x3)    Normal    TrafficWeightUpdated      Traffic weight updated 
2023-07-31T01:00:43Z         Normal    ScalingReplicaSet         Scaled up ReplicaSet dynamic-stable-scale-68c5f4fff5 (revision 2) from 0 to 1
2023-07-31T01:00:44Z         Normal    SwitchService             Switched selector for service 'dynamic-stable-scale-canary' from '868d98995b' to '68c5f4fff5'
2023-07-31T01:00:44Z         Normal    PatchingALBIngress        Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '25'
2023-07-31T01:00:44Z         Normal    TrafficWeightUpdated      Traffic weight updated from 0 to 25
2023-07-31T01:00:44Z         Normal    RolloutStepCompleted      Rollout step 1/4 completed (setWeight: 25)
2023-07-31T01:00:44Z         Normal    ScalingReplicaSet         Scaled down ReplicaSet dynamic-stable-scale-868d98995b (revision 1) from 4 to 3
2023-07-31T01:00:44Z         Normal    RolloutPaused             Rollout is paused (CanaryPauseStep)
2023-07-31T01:00:46Z         Normal    RolloutStepCompleted      Rollout step 2/4 completed (pause)

time="2023-07-31T01:00:57Z" level=info msg="Deleting e2e-test-name=TestCanaryDynamicStableScale"
    --- FAIL: TestCanarySuite/TestCanaryDynamicStableScale (20.11s)

Failed
=== RUN   TestCanarySuite
time="2023-07-31T01:00:37Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n"
--- FAIL: TestCanarySuite (248.77s)

Failed
=== RUN   TestFunctionalSuite/TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds
time="2023-07-31T01:13:10Z" level=info msg="Deleting e2e-test-name=TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds"
time="2023-07-31T01:13:11Z" level=info msg="rollout.argoproj.io/rollout-canary created\n" rollout=rollout-canary
time="2023-07-31T01:13:11Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-canary
time="2023-07-31T01:13:13Z" level=info msg="Condition 'status=Healthy' met after 2s" rollout=rollout-canary
time="2023-07-31T01:13:13Z" level=info msg="Updated rollout pod spec: 2023-07-31T01:13:13.556697949Z" rollout=rollout-canary
    when.go:353: unexpected status 
Name:            rollout-canary
Namespace:       default
Status:          �[94m◌�[0m Progressing
Message:         waiting for rollout spec update to be observed
Strategy:        Canary
  Step:          3/3
  SetWeight:     100
  ActualWeight:  100
Images:          nginx:1.19-alpine (�[32mstable�[0m)
Replicas:
  Desired:       3
  Current:       3
  Updated:       3
  Ready:         3
  Available:     3

NAME                                        KIND        STATUS         AGE   INFO
⟳ rollout-canary                            Rollout     �[94m◌�[0m Progressing  2m2s  
└──# revision:1                                                              
   └──⧉ �[32mrollout-canary-5c8ff9c59d�[0m           ReplicaSet  �[32m✔�[0m Healthy      2m2s  �[32mstable�[0m
      ├──□ rollout-canary-5c8ff9c59d-68zkd  Pod         �[32m✔�[0m Running      2m2s  ready:1/1
      ├──□ rollout-canary-5c8ff9c59d-lw67n  Pod         �[32m✔�[0m Running      2m2s  ready:1/1
      └──□ rollout-canary-5c8ff9c59d-zttsz  Pod         �[32m✔�[0m Running      2m2s  ready:1/1

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: "1"
  creationTimestamp: "2023-07-31T01:13:11Z"
  generation: 2
  labels:
    argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e
    e2e-test-name: TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds
  name: rollout-canary
  namespace: default
  resourceVersion: "12924"
  uid: bc521610-fc7f-47d3-8c24-a66b3dfc43f7
spec:
  progressDeadlineSeconds: 5
  replicas: 3
  selector:
    matchLabels:
      app: rollout-canary
  strategy:
    canary:
      steps:
      - setWeight: 32
      - pause:
          duration: 30s
      - setWeight: 67
  template:
    metadata:
      annotations:
        update: "2023-07-31T01:13:13.556697949Z"
      creationTimestamp: null
      labels:
        app: rollout-canary
    spec:
      containers:
      - image: nginx:1.19-alpine
        name: rollouts-demo
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 16Mi
status:
  HPAReplicas: 3
  availableReplicas: 3
  blueGreen: {}
  canary: {}
  conditions:
  - lastTransitionTime: "2023-07-31T01:13:11Z"
    lastUpdateTime: "2023-07-31T01:13:11Z"
    message: RolloutCompleted
    reason: RolloutCompleted
    status: "True"
    type: Completed
  - lastTransitionTime: "2023-07-31T01:13:13Z"
    lastUpdateTime: "2023-07-31T01:13:13Z"
    message: Rollout is healthy
    reason: RolloutHealthy
    status: "True"
    type: Healthy
  - lastTransitionTime: "2023-07-31T01:13:11Z"
    lastUpdateTime: "2023-07-31T01:13:13Z"
    message: ReplicaSet "rollout-canary-5c8ff9c59d" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-07-31T01:13:13Z"
    lastUpdateTime: "2023-07-31T01:13:13Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  currentPodHash: 5c8ff9c59d
  currentStepHash: fd7c94c7c
  currentStepIndex: 3
  observedGeneration: "1"
  phase: Healthy
  readyReplicas: 3
  replicas: 3
  selector: app=rollout-canary
  stableRS: 5c8ff9c59d
  updatedReplicas: 3

2023-07-31T01:13:11Z    Normal    RolloutAddedToInformer    Rollout resource added to informer: default/rollout-canary
2023-07-31T01:13:11Z    Normal    RolloutNotCompleted       Rollout not completed, started update to revision 1 (5c8ff9c59d)
2023-07-31T01:13:11Z    Normal    RolloutUpdated            Rollout updated to revision 1
2023-07-31T01:13:11Z    Normal    NewReplicaSetCreated      Created ReplicaSet rollout-canary-5c8ff9c59d (revision 1)
2023-07-31T01:13:11Z    Normal    ScalingReplicaSet         Scaled up ReplicaSet rollout-canary-5c8ff9c59d (revision 1) from 0 to 3
2023-07-31T01:13:11Z    Normal    RolloutCompleted          Rollout completed update to revision 1 (5c8ff9c59d): Initial deploy

time="2023-07-31T01:15:13Z" level=info msg="Deleting e2e-test-name=TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds"
    --- FAIL: TestFunctionalSuite/TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds (123.95s)

Failed
=== RUN   TestFunctionalSuite
time="2023-07-31T01:08:13Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n"
--- FAIL: TestFunctionalSuite (463.16s)

Failed
=== RUN   TestRollbackSuite/TestRollbackAnalysisOutsideWindow
time="2023-07-31T01:14:42Z" level=info msg="Deleting e2e-test-name=TestRollbackAnalysisOutsideWindow"
W0731 01:14:42.289574   18938 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0731 01:14:42.319701   18938 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2023-07-31T01:14:42Z" level=info msg="rollout.argoproj.io/window configured\n" rollout=window
time="2023-07-31T01:14:42Z" level=info msg="Waiting for condition: status=Healthy" rollout=window
time="2023-07-31T01:14:50Z" level=info msg="Condition 'status=Healthy' met after 7s" rollout=window
time="2023-07-31T01:14:50Z" level=info msg="Updated rollout spec: {\"metadata\":{\"labels\":{\"rev\":\"two\"}},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"rev\":\"two\"}}}}}" rollout=window
time="2023-07-31T01:14:50Z" level=info msg="Waiting for condition: status=Healthy" rollout=window
time="2023-07-31T01:14:56Z" level=info msg="Condition 'status=Healthy' met after 6s" rollout=window
time="2023-07-31T01:14:56Z" level=info msg="AnalysisRun expectation 'analysisrun count == 1' met" rollout=window
time="2023-07-31T01:14:56Z" level=info msg="Updated rollout spec: {\"metadata\":{\"labels\":{\"rev\":\"three\"}},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"rev\":\"three\"}}}}}" rollout=window
time="2023-07-31T01:14:56Z" level=info msg="Waiting for condition: status=Healthy" rollout=window
time="2023-07-31T01:14:56Z" level=info msg="Condition 'status=Healthy' met after 0s" rollout=window
time="2023-07-31T01:14:56Z" level=error msg="AnalysisRun expectation 'analysisrun count == 2' failed" rollout=window
Name:            window
Namespace:       default
Status:          �[94m◌�[0m Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/2
  SetWeight:     50
  ActualWeight:  0
Images:          nginx:1.19-alpine (�[32mstable�[0m)
Replicas:
  Desired:       2
  Current:       2
  Updated:       0
  Ready:         2
  Available:     2

NAME                                                         KIND         STATUS         AGE  INFO
⟳ window                                                     Rollout      �[94m◌�[0m Progressing  15s  
├──# revision:3                                                                               
│  └──⧉ �[33mwindow-6458bcd6fb�[0m                                    ReplicaSet   �[94m◌�[0m Progressing  1s   �[33mcanary�[0m
│     └──□ window-6458bcd6fb-7xtlf                           Pod          �[94m◌�[0m Pending      1s   ready:0/1
├──# revision:2                                                                               
│  ├──⧉ �[32mwindow-5658c69dbb�[0m                                    ReplicaSet   �[32m✔�[0m Healthy      7s   �[32mstable�[0m
│  │  ├──□ window-5658c69dbb-dgqvc                           Pod          �[32m✔�[0m Running      7s   ready:1/1
│  │  └──□ window-5658c69dbb-cklc7                           Pod          �[32m✔�[0m Running      2s   ready:1/1
│  └──α �[0mwindow-5658c69dbb-2-1�[0m                                AnalysisRun  �[32m✔�[0m Successful   6s   �[32m✔�[0m 1
│     └──⊞ �[0m32e8c8d2-7dac-4e94-824c-ebdbdf6feea3.sleep-job.1�[0m  Job          �[32m✔�[0m Successful   6s   
└──# revision:1                                                                               
   └──⧉ window-c64cf84f6                                     ReplicaSet   �[0m•�[0m ScaledDown   15s  
      └──□ window-c64cf84f6-5nk2f                            Pod          �[94m◌�[0m Terminating  15s  ready:1/1

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: "2"
  creationTimestamp: "2023-07-31T01:14:42Z"
  generation: 6
  labels:
    argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e
    e2e-test-name: TestRollbackAnalysisOutsideWindow
    rev: three
  name: window
  namespace: default
  resourceVersion: "16312"
  uid: 4f73ea97-5305-44c0-9163-566aa9175695
spec:
  replicas: 2
  rollbackWindow:
    revisions: 2
  selector:
    matchLabels:
      app: window
  strategy:
    canary:
      steps:
      - setWeight: 50
      - analysis:
          analysisRunMetadata: {}
          args:
          - name: rev
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['rev']
          templates:
          - templateName: sleep-job
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: window
        rev: three
    spec:
      containers:
      - image: nginx:1.19-alpine
        name: basic
        resources:
          requests:
            cpu: 1m
            memory: 16Mi
status:
  HPAReplicas: 2
  availableReplicas: 2
  blueGreen: {}
  canary: {}
  conditions:
  - lastTransitionTime: "2023-07-31T01:14:49Z"
    lastUpdateTime: "2023-07-31T01:14:49Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2023-07-31T01:14:56Z"
    lastUpdateTime: "2023-07-31T01:14:56Z"
    message: Rollout is not healthy
    reason: RolloutHealthy
    status: "False"
    type: Healthy
  - lastTransitionTime: "2023-07-31T01:14:42Z"
    lastUpdateTime: "2023-07-31T01:14:56Z"
    message: Rollout "window" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-07-31T01:14:56Z"
    lastUpdateTime: "2023-07-31T01:14:56Z"
    message: RolloutCompleted
    reason: RolloutCompleted
    status: "False"
    type: Completed
  currentPodHash: 6458bcd6fb
  currentStepHash: 7dd87d6d74
  currentStepIndex: 0
  message: more replicas need to be updated
  observedGeneration: "6"
  phase: Progressing
  readyReplicas: 2
  replicas: 2
  selector: app=window
  stableRS: 5658c69dbb

2023-07-31T01:14:42Z    Normal    RolloutAddedToInformer    Rollout resource added to informer: default/window
2023-07-31T01:14:42Z    Normal    RolloutNotCompleted       Rollout not completed, started update to revision 1 (c64cf84f6)
2023-07-31T01:14:42Z    Normal    RolloutUpdated            Rollout updated to revision 1
2023-07-31T01:14:42Z    Normal    NewReplicaSetCreated      Created ReplicaSet window-c64cf84f6 (revision 1)
2023-07-31T01:14:42Z    Normal    ScalingReplicaSet         Scaled up ReplicaSet window-c64cf84f6 (revision 1) from 0 to 2
2023-07-31T01:14:42Z    Normal    RolloutCompleted          Rollout completed update to revision 1 (c64cf84f6): Initial deploy
2023-07-31T01:14:50Z    Normal    RolloutNotCompleted       Rollout not completed, started update to revision 2 (5658c69dbb)
2023-07-31T01:14:50Z    Normal    RolloutUpdated            Rollout updated to revision 2
2023-07-31T01:14:50Z    Normal    NewReplicaSetCreated      Created ReplicaSet window-5658c69dbb (revision 2)
2023-07-31T01:14:50Z    Normal    ScalingReplicaSet         Scaled up ReplicaSet window-5658c69dbb (revision 2) from 0 to 1
2023-07-31T01:14:51Z    Normal    ScalingReplicaSet         Scaled down ReplicaSet window-c64cf84f6 (revision 1) from 2 to 1
2023-07-31T01:14:51Z    Normal    RolloutStepCompleted      Rollout step 1/2 completed (setWeight: 50)
2023-07-31T01:14:51Z    Normal    AnalysisRunRunning        Step Analysis Run 'window-5658c69dbb-2-1' Status New: 'Running' Previous: ''
2023-07-31T01:14:55Z    Normal    AnalysisRunSuccessful     Step Analysis Run 'window-5658c69dbb-2-1' Status New: 'Successful' Previous: 'Running'
2023-07-31T01:14:55Z    Normal    RolloutStepCompleted      Rollout step 2/2 completed (analysis)
2023-07-31T01:14:55Z    Normal    ScalingReplicaSet         Scaled up ReplicaSet window-5658c69dbb (revision 2) from 1 to 2
2023-07-31T01:14:56Z    Normal    ScalingReplicaSet         Scaled down ReplicaSet window-c64cf84f6 (revision 1) from 1 to 0
2023-07-31T01:14:56Z    Normal    RolloutCompleted          Rollout completed update to revision 2 (5658c69dbb): Completed all 2 canary steps
2023-07-31T01:14:56Z    Normal    RolloutNotCompleted       Rollout not completed, started update to revision 3 (6458bcd6fb)
2023-07-31T01:14:56Z    Normal    RolloutUpdated            Rollout updated to revision 3
2023-07-31T01:14:56Z    Normal    NewReplicaSetCreated      Created ReplicaSet window-6458bcd6fb (revision 3)
2023-07-31T01:14:56Z    Normal    ScalingReplicaSet         Scaled up ReplicaSet window-6458bcd6fb (revision 3) from 0 to 1

time="2023-07-31T01:14:57Z" level=info msg="Deleting e2e-test-name=TestRollbackAnalysisOutsideWindow"
W0731 01:14:57.347079   18938 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0731 01:14:57.367354   18938 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
    --- FAIL: TestRollbackSuite/TestRollbackAnalysisOutsideWindow (15.32s)

Failed
=== RUN   TestRollbackSuite
time="2023-07-31T01:14:41Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n"
time="2023-07-31T01:14:42Z" level=info msg="rollout.argoproj.io/window created\n"
--- FAIL: TestRollbackSuite (42.57s)