Unexpected rollout of all already deployed canary during upgrade from < 1.16.1 to >= 1.16.1

[tejaspbajaj]

Describe the bug

When upgrading flagger from version 1.14.0 to 1.19.0, it triggered canary rollouts. On narrowing it down further it seemed to happen when going from 1.16.0 to 1.16.1.
Curious on why this happens and if this could happen in future.

Canary manifest used:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: testapp
  namespace: testapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name:  testapp-server
  progressDeadlineSeconds: 600
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: testapp-server
  service:
    name: test-app
    port: 80
    targetPort: http
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: "request-success-rate"
      thresholdRange:
        min: 99
      interval: 1m
    - name: "request-duration"
      thresholdRange:
        max: 20
      interval: 1m

To Reproduce

Start at flagger 1.16.0 then upgrade to 1.16.1, notice that the canaries start progressing after upgrade

➜ kubectl get canaries -A | grep test
default     testapp    Initialized   0        2022-05-18T20:09:59Z

➜  helm list -n istio-system | grep flagger
flagger                        istio-system 12       2022-05-18 13:12:49.584183 -0700 PDT    deployed flagger-1.16.0                        1.16.0

➜  helm upgrade -i flagger flagger/flagger --namespace=istio-system --version=v1.16.1                        
Release "flagger" has been upgraded. Happy Helming!
NAME: flagger
LAST DEPLOYED: Wed May 18 13:14:45 2022
NAMESPACE: istio-system
STATUS: deployed
REVISION: 13
TEST SUITE: None
NOTES:Flagger installed

➜  kubectl get canaries -A | grep test                                                                                                          
default     testapp    Initialized   0        2022-05-18T20:09:59Z

➜  kubectl get canaries -A | grep test                                                                                                         
default     testapp    Progressing   0        2022-05-18T20:15:18Z

➜ helm list -n istio-system | grep flagger
flagger                        istio-system 13       2022-05-18 13:14:45.8087 -0700 PDT      deployed flagger-1.16.1                        1.16.1

Expected behavior

Flagger should not induce rollouts unless called out in the release notes.

Additional context

• Flagger version: 1.16.1
• Kubernetes version: 1.21.9
• Service Mesh provider: Istio 1.13.3
• Ingress provider: Envoy

[stefanprodan]

This was due to Kubernetes packages upgrades, the computed checksum of the canary changed due to how the objects are serialised. We'll need to add some tests around this, in case it changes again in the future.

[tejaspbajaj]

Thanks Stefan, sounds good.