Kustomization not removing attributes added by kubectl edit

[nicjohnson145]

Describe the bug

When new keys are added through kubectl edit to a ConfigMap managed by flux, the subsequent reconciliation loops do not remove the keys not present in git.

Steps to reproduce

1. Create a ConfigMap in the flux repo

❯ ll
total 192
-rw-r--r-- 1 NJohnso157 staff    106 Nov 16 16:06 configmap.yaml
-rw-r--r-- 1 NJohnso157 staff 182776 Nov 16 16:04 gotk-components.yaml
-rw-r--r-- 1 NJohnso157 staff    553 Nov 16 16:04 gotk-sync.yaml
-rw-r--r-- 1 NJohnso157 staff    132 Nov 16 16:05 kustomization.yaml
❯ cat configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: flux-repro
  namespace: default
data:
  key1: value1
❯ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
- configmap.yaml

2. Push the changes and trigger a reconciliation

❯ flux reconcile source git flux-system                                         
► annotating GitRepository flux-system in flux-system namespace                 
✔ GitRepository annotated                                                       
◎ waiting for GitRepository reconciliation                                      
✔ fetched revision main/52962ac2e9dcb8877653efe1eaeb7ee3ad3fc845 

3. Edit the ConfigMap through kubectl edit to add a new key not present in the git repository

❯ kc get configmap -n default flux-repro -o yaml
apiVersion: v1
data:
  key1: value1
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-16T22:06:38Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-repro
  namespace: default
  resourceVersion: "1579"
  uid: 37c08ec3-083e-45ae-ad85-d1c1436c8693
❯ kc edit configmap -n default flux-repro
configmap/flux-repro edited
❯ kc get configmap -n default flux-repro -o yaml
apiVersion: v1
data:
  key1: value1
  new_key: new_value
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-16T22:06:38Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-repro
  namespace: default
  resourceVersion: "1689"
  uid: 37c08ec3-083e-45ae-ad85-d1c1436c8693

4. Force a reconciliation

❯ flux reconcile kustomization flux-system
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main/52962ac2e9dcb8877653efe1eaeb7ee3ad3fc845

5. Observe that the ConfigMap does not match what is stored in the git repository, and still contains the manually added key

❯ kc get configmap -n default flux-repro -o yaml
apiVersion: v1
data:
  key1: value1
  new_key: new_value
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-16T22:06:38Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-repro
  namespace: default
  resourceVersion: "1689"
  uid: 37c08ec3-083e-45ae-ad85-d1c1436c8693

Expected behavior

The ConfigMap should be updated to match what is stored in Git, i.e the manually added key should be removed

Screenshots and recordings

No response

OS / Distro

macOS

Flux version

flux: v0.23.0

Flux check

❯ flux check
► checking prerequisites
✔ Kubernetes 1.22.2 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.13.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.18.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.18.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.18.0
✔ all checks passed

Git provider

Gitlab

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Duplicate of fluxcd/kustomize-controller#486

[nicjohnson145]

The workaround mentioned in this issue does not resolve the issue. Are there any alternatives?

❯ kc get configmap -n default flux-repro -o jsonpath='{.metadata.managedFields}' | jq                                                                           
[                                                                               
  {                                                                                                                                                             
    "apiVersion": "v1",                                                         
    "fieldsType": "FieldsV1",                                                                                                                                   
    "fieldsV1": {                                                               
      "f:data": {                                                                                                                                               
        "f:key1": {}                                                            
      },                                                                                                                                                        
      "f:metadata": {                                                           
        "f:labels": {                                                                                                                                           
          "f:foo": {},                                                          
          "f:kustomize.toolkit.fluxcd.io/name": {},                                                                                                             
          "f:kustomize.toolkit.fluxcd.io/namespace": {}                         
        }                                                                                                                                                       
      }                                                                                                                                                         
    },                                                                                                                                                          
    "manager": "kustomize-controller",                                          
    "operation": "Apply",                                                       
    "time": "2021-11-17T17:23:46Z"                                              
  },                                                                            
  {                                                                             
    "apiVersion": "v1",                                                         
    "fieldsType": "FieldsV1",                                                                                                                                   
    "fieldsV1": {                                                                                                                                               
      "f:data": {                       
        "f:new_key": {}                                                                                                                                         
      }                                                                         
    },                                                                          
    "manager": "kubectl-edit",                                                  
    "operation": "Update",                                                      
    "time": "2021-11-16T22:07:24Z"                                              
  }                                                                             
]                                                                                                                                                      
❯ kc patch cm -n default flux-repro --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/1"}]'                                                    
configmap/flux-repro patched            
❯ kc get configmap -n default flux-repro -o jsonpath='{.metadata.managedFields}' | jq                                                                           
[                                                                                                                                                               
  {                                                                                                                                                             
    "apiVersion": "v1",                                                                                                                                         
    "fieldsType": "FieldsV1",                                                                                                                                   
    "fieldsV1": {                                                                                                                                               
      "f:data": {                                                               
        "f:key1": {}                                                                                                                                            
      },                                                                                                                                                        
      "f:metadata": {                                                                                                                                           
        "f:labels": {                                                                                                                                           
          "f:foo": {},                                                                                                                                          
          "f:kustomize.toolkit.fluxcd.io/name": {},                                                                                                             
          "f:kustomize.toolkit.fluxcd.io/namespace": {}                                                                                                         
        }                               
      }                                                                         
    },                                                                                                                                                          
    "manager": "kustomize-controller",  
    "operation": "Apply",                                                                                                                                       
    "time": "2021-11-17T17:23:46Z"                                              
  }                                                                                                                                                             
]                                                                                                                                                               
❯ flux reconcile kustomization flux-system                                      
► annotating Kustomization flux-system in flux-system namespace                 
✔ Kustomization annotated                                                       
◎ waiting for Kustomization reconciliation                                                                                                                      
✔ Kustomization reconciliation completed                                        
✔ reconciled revision main/52962ac2e9dcb8877653efe1eaeb7ee3ad3fc845
❯ kc get configmap -n default flux-repro -o yaml                                
apiVersion: v1                                                                                                                                                  
data:                                                                                                                                                           
  key1: value1                          
  new_key: new_value                                                                                                                                            
kind: ConfigMap                                                                 
metadata:                                                                       
  creationTimestamp: "2021-11-16T22:06:38Z"                                     
  labels:                                                                       
    foo: bar                                                                    
    kustomize.toolkit.fluxcd.io/name: flux-system                               
    kustomize.toolkit.fluxcd.io/namespace: flux-system                          
  name: flux-repro                                                                                                                                              
  namespace: default                                                                                                                                            
  resourceVersion: "30623"              
  uid: 37c08ec3-083e-45ae-ad85-d1c1436c8693
❯ kc patch cm -n default flux-repro --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0"}]'                                                    
configmap/flux-repro patched                                                                                                                                    
❯ kc get configmap -n default flux-repro -o jsonpath='{.metadata.managedFields}' | jq                                                                           
❯ flux reconcile kustomization flux-system                                                                                                                      
► annotating Kustomization flux-system in flux-system namespace                 
✔ Kustomization annotated                                                                                                                                       
◎ waiting for Kustomization reconciliation                                                                                                                      
✔ Kustomization reconciliation completed                                                                                                                        
✔ reconciled revision main/52962ac2e9dcb8877653efe1eaeb7ee3ad3fc845                                                                                             
❯ kc get configmap -n default flux-repro -o yaml                                                                                                                
apiVersion: v1                                                                                                                                                  
data:                                                                                                                                                           
  key1: value1                          
  new_key: new_value                                                            
kind: ConfigMap                                                                                                                                                 
metadata:                               
  creationTimestamp: "2021-11-16T22:06:38Z"                                                                                                                     
  labels:                                                                       
    foo: bar                                                                                                                                                    
    kustomize.toolkit.fluxcd.io/name: flux-system                                                                                                               
    kustomize.toolkit.fluxcd.io/namespace: flux-system                          
  name: flux-repro                                                              
  namespace: default                                                            
  resourceVersion: "30802"                                                                                                                                      
  uid: 37c08ec3-083e-45ae-ad85-d1c1436c8693

[stefanprodan]

Hmm this is really strange, I have a test here for detecting drift of keys added directly to the cluster https://github.com/fluxcd/pkg/blob/main/ssa/manager_diff_test.go#L181

I'll try this on a real cluster tomorrow and get back to you.

[somtochiama]

I tried this on a real cluster and was able to reproduce this.
It happens when applying an object(not managed by flux) with Flux.
It is a result of how server-side apply works. It doesn't delete keys with no manager.

$ kubectl apply --server-side -f test-cm.yaml
$ kubectl edit cm test-ssa
$ kubectl apply --server-side -f test-cm.yaml

$ kubectl get cm test-ssa -oyaml
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-19T08:17:26Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key1: {}
    manager: kubectl
    operation: Apply
    time: "2021-11-19T08:17:26Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key2: {}
    manager: kubectl-edit
    operation: Update
    time: "2021-11-19T08:18:25Z"
  name: test-ssa
  namespace: default
  resourceVersion: "46165"
  uid: ea2bfffd-fd2b-41ea-8854-7143dc3a049a

Then after

$ k patch cm -n default test-ssa --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/1"}]'  
$ kubectl apply --server-side -f test-cm.yaml

The added field still exists:

$ kubectl get cm test-ssa -oyaml
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-19T08:17:26Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key1: {}
    manager: kubectl
    operation: Apply
    time: "2021-11-19T08:17:26Z"
  name: test-ssa
  namespace: default
  resourceVersion: "46689"
  uid: ea2bfffd-fd2b-41ea-8854-7143dc3a049a

[stefanprodan]

I wander if this is by design or if it’s a bug in the Kubernetes API.

Regardless of this, we can’t do anything in Flux if Kubernetes server-side apply doesn’t remove fields without a manager.

I think we should document this behavior in https://github.com/fluxcd/kustomize-controller/blob/main/docs/spec/v1beta2/kustomization.md#reconciliation

[superbrothers]

If kubectl field-manager does not get ownership of the field you want to delete beforehand, you will not be able to delete the field even if you delete the managedField.

The easiest way to get ownership of all fields is to use the following command:

kubectl get cm test-ssa -o yaml | kubectl apply --server-side -f-

The complete steps that can solve the problem are as follows。

$ kubectl apply --server-side -f test-cm.yaml
configmap/test-ssa serverside-applied

$ k edit cm test-ssa
configmap/test-ssa edited

$ kubectl apply --server-side -f test-cm.yaml
configmap/test-ssa serverside-applied

$ kubectl get cm test-ssa --show-managed-fields -o yaml
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-30T08:27:44Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key1: {}
    manager: kubectl
    operation: Apply
    time: "2021-11-30T08:27:44Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key2: {}
    manager: kubectl-edit
    operation: Update
    time: "2021-11-30T08:27:56Z"
  name: test-ssa
  namespace: default
  resourceVersion: "47057"
  uid: 341de5b0-40c4-43d9-b90d-f6ed3b6d4438

Get ownership of all fields of test-ssa cm by kubectl field-manager.

$ kubectl get cm test-ssa -o yaml | kubectl apply --server-side -f-
configmap/test-ssa serverside-applied

$ kubectl get cm test-ssa --show-managed-fields -o yaml
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-30T08:27:44Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key1: {}
        f:key2: {}
    manager: kubectl
    operation: Apply
    time: "2021-11-30T08:28:49Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key2: {}
    manager: kubectl-edit
    operation: Update
    time: "2021-11-30T08:27:56Z"
  name: test-ssa
  namespace: default
  resourceVersion: "47148"
  uid: 341de5b0-40c4-43d9-b90d-f6ed3b6d4438

Remove the managedField of kubectl-edit field-manager.

$ kubectl patch cm -n default test-ssa --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/1"}]'
configmap/test-ssa patched

$ kubectl get cm test-ssa --show-managed-fields -o yaml
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-30T08:27:44Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key1: {}
        f:key2: {}
    manager: kubectl
    operation: Apply
    time: "2021-11-30T08:28:49Z"
  name: test-ssa
  namespace: default
  resourceVersion: "47222"
  uid: 341de5b0-40c4-43d9-b90d-f6ed3b6d4438

$ kubectl apply --server-side -f test-cm.yaml
configmap/test-ssa serverside-applied

$ kubectl get cm test-ssa --show-managed-fields -o yaml
apiVersion: v1
data:
  key1: value1
kind: ConfigMap
metadata:
  creationTimestamp: "2021-11-30T08:27:44Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:key1: {}
    manager: kubectl
    operation: Apply
    time: "2021-11-30T08:29:55Z"
  name: test-ssa
  namespace: default
  resourceVersion: "47263"
  uid: 341de5b0-40c4-43d9-b90d-f6ed3b6d4438

For more information about ownership, please refer to the following document.

• https://kubernetes.io/docs/reference/using-api/server-side-apply/#transferring-ownership

[superbrothers]

We are also facing this problem, but we expect to solve it with the following steps.

1. Stop kustomize-controller
2. Change ownership of all fields to kustomize-controller manager
3. Remove managedFields owned by kubectl-client-side-apply or kubectl-edit field-manager etc. (consider carefully which managedFields are not needed for you)
4. Start kustomize-controller

We can delete the fields by kustomize-controller in the above steps.

However, it does not mean that kustomize-controller will always be able to remove fields added by humans with kubectl apply. We often suspend ks to temporarily apply manifests manually and verify the behavior of the application. Therefore, we would like kustomize-controller to remove fields managed by kubectl-client-side-apply field-manager.

We are considering the following solutions to this problem.

1. Stop kustomize-controller
2. Remove managedFields managed by kubectl-client-side-apply field-manage
3. Make kustomize-controller field manager gets ownership of all fields of all objects managed by Flux
4. Rename the manager name of managedFields managed by kustomize-controller field-manager to kubectl-client-side-apply
5. Rename the field-manager used by kustomize-controller to kubectl-client-side-apply (need to fork it at this time)

The above steps will allow kustomize-controller to remove fields added by humans with kubectl apply. If it works as expected, I would like to add an option to explicitly specify the field-manager name in kustomize-controller.