ArgoCD losing some of its settings after re-running cluster.yml

[spantaleev]

What happened?

After re-running cluster.yml (with or without --tags=argocd), ArgoCD gets reinstalled each time from the upstream all-in-one install.yaml file:

argocd_install_url: "https://raw.githubusercontent.com/argoproj/argo-cd/{{ argocd_version }}/manifests/install.yaml"

install.yaml contains various resource definitions, among which several ConfigMap resources. Some of these (e.g. argocd-ssh-known-hosts-cm, argocd-cm) are used for storing applications state which the ArgoCD administrator may wish to change.

argocd-ssh-known-hosts-cm is certainly used by people who wish to connect ArgoCD to their own private git repositories running on a self-hosted server.

argocd-cm may be adjusted by people with more special Ingress, to work around Why is my application stuck in Progressing state?, as described here and here.

Re-running Kubespray having the side-effect of rebuilding these ConfigMap objects and losing data is unexpected.

What did you expect to happen?

This may be a bit of an upstream documentation problem as well. The Upgrading guide for ArgoCD does not state that one is supposed to backup the ArgoCD settings each time versions are switched (or the same version is re-applied). It hints that during major version upgrades it's recommended to take a backup as described on the Disaster Recovery documentation. It's not explicitly clear that ConfigMap-stored settings will always be lost when running install.yaml, regardless of whether you're upgrading at all or reinstalling the same version.

It seems like the proper way to upgrade ArgoCD is to always:

1. Do a backup, as described in Disaster Recovery
2. Run install.yaml for the new version
3. Restore the backup, as described in Disaster Recovery

Unless I'm misunderstanding something, I believe the ArgoCD Upgrading/Overview documentation should be updated to reflect this.

Nevertheless, given that Kubespray supports an ArgoCD addon, it also becomes Kubespray's problem how this addon is managed.

Because this ArgoCD addon is tricky to upgrade and Kubespray includes it as a supported component, I believe one of the following needs to be done:

1. Either: remove support for the ArgoCD addon and let people deploy it as they see fit and worry about upgrading by themselves. At least re-running cluster.yml would not cause data loss and it won't be Kubespray's problem anymore.
2. Or: add Kubespray documentation related to how the ArgoCD addon is to be used. I couldn't find any documentation about this addon in Kubespray yet. Perhaps one should enable the ArgoCD addon once, install via Kubespray and then disable it to avoid data loss? To upgrade to a new version in the future, perhaps one should be instructed to first create an ArgoCD backup as described in Disaster Recovery, then re-enable the addon, re-run the playbook, import the ArgoCD backup data and finally disable the addon again to avoid accidental data loss.
3. Or: Kubespray could be made to do all this automatically. If it could do so, there's value-add in having the Kubespray playbook install ArgoCD as opposed to installing it the regular way. Implementation ideas follow. During addon installation, addon manifests (*.yaml) are dumped to the first control plane node (/etc/kubernetes/addons/..). This node already has a kubeconfig file with admin access. The playbook may detect the existence of ArgoCD, perform backup procedures by running containers as described in Disaster Recovery, then run ArgoCD's all-in-one install.yaml like it does now, then restore from backup and finally clean up the local backups.

How can we reproduce it (as minimally and precisely as possible)?

1. Enable the ArgoCD addon (argocd_enabled: true + argocd_admin_password: ...)
2. Run cluster.yml (or cluster.yml --tags=argocd) to get it installed
3. Expose it locally on port 1234: kubectl port-forward -n argocd service/argocd-server 1234:80
4. Log in to http://localhost:1234 with admin and the password specified in argocd_admin_password
5. Go to Settings -> Repository certificates and known hosts, and add the Example SSH host key found in my message below
6. Verify that the custom SSH host key made it to the argocd-ssh-known-hosts-cm ConfigMap: kubectl -n argocd describe cm argocd-ssh-known-hosts-cm | grep example.com
7. Run cluster.yml (or cluster.yml --tags=argocd) once again. We have not changed the ArgoCD version in the inventory or done any other changes.
8. After re-running the playbook, certain resources (like the argocd-ssh-known-hosts-cm ConfigMap) got reset to their default initial values as seen in the upstream install.yaml all-in-one file.
9. Verify that our custom SSH host key is now indeed gone: kubectl -n argocd describe cm argocd-ssh-known-hosts-cm | grep example.com

---

Example SSH host key that may be added during this test:

example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=

This example SSH host key is actually one of the github.com SSH host keys. We have to use some vaid base64-encoded data for ArgoCD to accept it, so using something real seemed like the best choice.

OS

Irrelevant

Version of Ansible

I'm running the container image for the v2.24 release (quay.io/kubespray/kubespray:v2.24):

ansible [core 2.15.8]
  config file = /kubespray/ansible.cfg
  configured module search path = ['/kubespray/library']
  ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

Version of Python

Python 3.10.12

Version of Kubespray (commit)

64447e7

Network plugin used

calico

Full inventory with variables

No response

Command used to invoke ansible

No response

Output of ansible run

No response

Anything else we need to know

No response

[VannTen]

Because this ArgoCD addon is tricky to upgrade and Kubespray includes it as a supported component, I believe one of the following needs to be done:

1. Either: remove support for the ArgoCD addon and let people deploy it as they see fit and worry about upgrading by themselves. At least re-running cluster.yml would not cause data loss and it won't be Kubespray's problem anymore.

2. Or: add Kubespray documentation related to how the ArgoCD addon is to be used. I couldn't find any documentation about this addon in Kubespray yet. Perhaps one should enable the ArgoCD addon once, install via Kubespray and then disable it to avoid data loss? To upgrade to a new version in the future, perhaps one should be instructed to first create an ArgoCD backup as described in Disaster Recovery, then re-enable the addon, re-run the playbook, import the ArgoCD backup data and finally disable the addon again to avoid accidental data loss.

3. Or: Kubespray could be made to do all this automatically. If it could do so, there's value-add in having the Kubespray playbook install ArgoCD as opposed to installing it the regular way. Implementation ideas follow. During addon installation, addon manifests (*.yaml) are dumped to the first control plane node (/etc/kubernetes/addons/..). This node already has a kubeconfig file with admin access. The playbook may detect the existence of ArgoCD, perform backup procedures by running containers as described in Disaster Recovery, then run ArgoCD's all-in-one install.yaml like it does now, then restore from backup and finally clean up the local backups.

I personally would go with 1, in the direction of #6400 . Maybe server apply could help us in this case, but that's not available in the custom kube.py module of kubespray.

@floryut @yankay : I propose the following: let's give us some delay to see if anyone come forward to support properly the ArgoCD addon, if not, delete it. (before 2.25 at the latest).

Wdyt ?

[VannTen]

#10909 (comment)
@mzaian (replying here)
No argument that handling argocd is useful. However, causing data loss is somehow worse than stop supporting argocd, don't you think ? (For existing users, that would mean no upgrade of argocd, it would not be removed)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[vaibhav2107]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.