Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secret key can not be omitted #1936

Open
dtaniwaki opened this issue Jul 14, 2019 · 26 comments
Open

Secret key can not be omitted #1936

dtaniwaki opened this issue Jul 14, 2019 · 26 comments
Labels
bug Something isn't working

Comments

@dtaniwaki
Copy link
Member

Describe the bug

Although the comment in secret.yaml indicates server.secretkey can be auto-generated if it's missing.

# Autogenerated when missing.

However, the code doesn't allow empty server.secretkey.

errs = append(errs, &incompleteSettingsError{message: "server.secretkey is missing"})

Which is the expected behavior?

To Reproduce

Just after installing ArgoCD and set admin.password (although this is not mentioned in the docs).

Expected behavior

Asking the expected behavior.

Screenshots

N/A

Version

$ argocd version
argocd: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:36Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:03Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

Logs

time="2019-07-14T11:45:39Z" level=info msg="Starting configmap/secret informers"
time="2019-07-14T11:45:39Z" level=info msg="Configmap/secret informer synced"
time="2019-07-14T11:45:39Z" level=fatal msg="server.secretkey is missing"

Have you thought about contributing a fix yourself?

I'm not sure which is the expected behavior.

@dtaniwaki dtaniwaki added the bug Something isn't working label Jul 14, 2019
@jessesuen
Copy link
Member

The API server is supposed to generate this. During new installations, we will sometimes see application-controller crash because it started before the api-server had a chance to generate it.

Which service are your logs coming from?

@dtaniwaki
Copy link
Member Author

The log came from the application controller.

@dtaniwaki
Copy link
Member Author

Uh, the api server was not running because of PSP. I'll try it and update you again.

@dtaniwaki
Copy link
Member Author

dtaniwaki commented Aug 29, 2019

I confirmed it is created automatically if the api server is running.

@emidevi
Copy link

emidevi commented Jan 24, 2022

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

@alkdese
Copy link

alkdese commented Jun 21, 2022

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

In my situation I started having CrashLoopBackoff on argocd-dex-server in the logs I have "server.secretkey is missing" after deleting argocd namespace and creating it from scratch. I just now deleted already 3 times and recreated and still same problem. Any advice?

@hyerim-gn
Copy link

hyerim-gn commented Jul 10, 2022

I solved the problem with node scale out due to the problem caused by the API server not being executed.

@emirot
Copy link
Contributor

emirot commented Jul 20, 2022

Seeing this issue after doing:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.7/manifests/install.yaml

Then CrashLoopBackOff on argocd-dex-server

kubectl logs argocd-dex-server-56d495844c-l2lxc -n argocd
time="2022-07-20T22:45:06Z" level=info msg="Starting configmap/secret informers"
time="2022-07-20T22:45:06Z" level=info msg="Configmap/secret informer synced"
time="2022-07-20T22:45:06Z" level=fatal msg="server.secretkey is missing"

@crenshaw-dev
Copy link
Member

@emirot can you confirm that the API server is successfully coming up?

@emirot
Copy link
Contributor

emirot commented Jul 26, 2022

@crenshaw-dev sorry about the noise, I figured it out, it's not related, it is due to Open Policy Agent

@brianpooe
Copy link

@emirot i have the exact same issue how did you resolve yours?

@emirot
Copy link
Contributor

emirot commented Oct 3, 2022

@brianpooe In my case I had to do add to add --set global.networkPolicy.create=true in my helm install

@winston0410
Copy link

I have the same issue when I install argocd with Core Install. @dtaniwaki Can we reopen this issue?

@yevhen-harmonizehr
Copy link

i do have same issue after my redis pods restarted. My argo redis do not have PVC, so this might be related.

@TeamDman
Copy link

TeamDman commented Nov 1, 2022

#10831 might be related

I had this

$ kubectl get all -n argocd
NAME                                                 READY   STATUS             RESTARTS      AGE
pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx   1/1     Running            0             17m
pod/argocd-argo-cd-repo-server-9565b9f86-5956t       1/1     Running            0             17m
pod/argocd-argo-cd-server-685dfc4dd7-kgl8r           0/1     CrashLoopBackOff   7 (33s ago)   11m
pod/argocd-redis-master-0                            1/1     Running            0             12m

NAME                                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/argocd-argo-cd-app-controller   ClusterIP   10.0.189.90   <none>        8082/TCP         17m
service/argocd-argo-cd-repo-server      ClusterIP   10.0.80.110   <none>        8081/TCP         17m
service/argocd-argo-cd-server           ClusterIP   10.0.42.249   <none>        80/TCP,443/TCP   17m
service/argocd-redis-headless           ClusterIP   None          <none>        6379/TCP         17m
service/argocd-redis-master             ClusterIP   10.0.51.188   <none>        6379/TCP         17m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-argo-cd-app-controller   1/1     1            1           17m
deployment.apps/argocd-argo-cd-repo-server      1/1     1            1           17m
deployment.apps/argocd-argo-cd-server           0/1     1            0           17m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-argo-cd-app-controller-6fbb4ccf8d   1         1         1       17m
replicaset.apps/argocd-argo-cd-repo-server-9565b9f86       1         1         1       17m
replicaset.apps/argocd-argo-cd-server-685dfc4dd7           1         1         0       17m

NAME                                   READY   AGE
statefulset.apps/argocd-redis-master   1/1     17m
$ kubectl logs -n argocd pod/argocd-argo-cd-server-685dfc4dd7-kgl8r
Defaulted container "argocd-server" out of: argocd-server, wait-for-redis (init)
 19:37:13.78 
 19:37:13.79 Welcome to the Bitnami argo-cd container
 19:37:13.79 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:37:13.79 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:37:13.79 
 19:37:13.80 INFO  ==> Configuring libnss_wrapper

time="2022-11-01T19:37:13Z" level=info msg="ArgoCD API Server is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd port=8080 version=v99.99.99+unknown
time="2022-11-01T19:37:13Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="Initialized server signature"
time="2022-11-01T19:37:14Z" level=info msg="admin disabled"
time="2022-11-01T19:37:14Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="configmap informer cancelled"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="secrets informer cancelled"
panic: server.secretkey is missing

goroutine 1 [running]:
github.com/argoproj/argo-cd/v2/util/session.NewSessionManager(0xc00004d980, {0x44c4aa0, 0xc000fcc580}, {0x34aec23, 0x16}, 0xc000aee5e0?, {0x44ddf10, 0xc0005a1f80})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/util/session/sessionmanager.go:124 +0x3fc
github.com/argoproj/argo-cd/v2/server.NewServer({_, _}, {0x0, 0x0, 0x1, {0x7ffc91ef54aa, 0x18}, 0x1f90, 0x1f93, {0xc000aee5e0, ...}, ...})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/server/server.go:267 +0x5af
github.com/argoproj/argo-cd/v2/cmd/argocd-server/commands.NewCommand.func1(0xc000afea00?, {0x347c98d?, 0xb?, 0xb?})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/argocd-server/commands/argocd_server.go:192 +0xfa8
github.com/spf13/cobra.(*Command).execute(0xc000afea00, {0xc0000ba010, 0xb, 0xb})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/[email protected]/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0xc000afea00)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/[email protected]/command.go:918
main.main()
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/main.go:57 +0x2c5
$ kubectl logs -n argocd pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx
Defaulted container "controller" out of: controller, wait-for-redis (init)
 19:21:17.44 
Welcome to the Bitnami argo-cd container
 19:21:17.44 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:21:17.45 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:21:17.45 
INFO  ==> Configuring libnss_wrapper
level=info msg="ArgoCD Application Controller is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd version=v99.99.99+unknown
level=info msg="Processing all cluster shards"
level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
level=info msg="Starting configmap/secret informers"
level=info msg="Configmap/secret informer synced"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=info msg="0xc000ff02a0 subscribed to settings updates"
level=info msg="Starting secretInformer forcluster"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Failed to save clusters info: server.secretkey is missing"
level=info msg="Notifying 1 settings subscribers: [0xc000ff02a0]"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"

Workaround

Manually specifying server.secretKey as an empty string in argocd-secret seems to have fixed the problem without having to manually restart anything after helm applying...
idk if everything is actually working, but the pod stopped crashing and logs look healthy now

@gaeljw
Copy link
Contributor

gaeljw commented Nov 17, 2022

For anyone coming here for similar issue where server.secretKey is not defined, here's what I did to make it work back:

  • delete the secret
  • re-create the secret but empty
  • restart argocd-server

@ali-de7
Copy link

ali-de7 commented Aug 9, 2023

I fixed it by simply restarting the server.
Here's what I did to fix it:

kubectl rollout restart deploy/argocd-server

@crenshaw-dev
Copy link
Member

Reopening because I think this is almost definitely a persistent problem with core install.

@crenshaw-dev crenshaw-dev reopened this Aug 9, 2023
@pgpx
Copy link
Contributor

pgpx commented Sep 13, 2023

Could we split argocd-secret into 2 or more secrets? Leave one for the autogenerated values, another for webhook secrets, and maybe another for the admin password (which is optionally autogenerated)? I'm encountering the same issue trying to introduce a webhook.gitlab.secret via Helm. And/or use server-side-apply for the autogenerated values?

@GuillaumeTC1
Copy link

GuillaumeTC1 commented Dec 11, 2023

Hello, I face a related issue on ArgoCD 2.9 managed with Helm Charts. Argo goes out of sync because of required secrets config. If I sync I loose access right away and need to re add manually secrets in Azure.

Capture d’écran 2023-12-11 150738

Am I the only one to encounter this issue ? Do you think it would be possible to fix this without having to ignore argocd-secret sync in argo-cm ?

@jdubs
Copy link

jdubs commented Jan 4, 2024

I'm also having this same issue with self managed argo. This virtually makes the api/ui not usable, it makes me think that it should fail the health check so that it can be restarted automatically with the new secret.

@fstr
Copy link

fstr commented Jan 26, 2024

I ran into the same issue as #1936 (comment) It's also simple to reproduce together with the ArgoCD Helm chart and ArgoCD managing itself.

Add a github.webhook.secret:

argo-cd:
  configs:
    secret:
       githubSecret: foobar

Sync it using ArgoCD. Successfully setup. Now your secret will look like this or similar:

apiVersion: v1
data:
  accounts.argoreadonly.tokens: +++++
  admin.password: +++++
  admin.passwordMtime: +++++
  server.secretkey: +++++
  webhook.github.secret: +++++

Now remove the github.webhook.secret by removing the configs: block from your value file again. The argocd-secret will now have an empty data: block when you run helm template because you removed the only value managed by the chart. If you now go into ArgoCD and sync, it will remove all the properties within data property of the argocd-secret. Including all those auto-generated values from the server. After that, ArgoCD stops working. You need to follow the steps from #1936 (comment) to resurrect it.

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in #16262 obsolete.

@MrFreezeex
Copy link
Contributor

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in #16262 obsolete.

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

@fstr
Copy link

fstr commented Jan 29, 2024

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

How I got into this topic is also by the need to provide the secret via external-secrets. If the webhook secret would be stored in a separate secret, then we could simply provision that via external-secrets and there's no more need to reference it via the special $<secret-name>.<property> syntax. Your PR will bridge the gap until then though.

@RobinsonZ
Copy link
Contributor

This is also causing an issue for us when argocd-secret is itself managed by argocd, and (as mentioned above) it's compounded by the fact that the argocd-server healthcheck will still succeed even when this error is occurring and making the interface unusable.

@gerardnico
Copy link

I got this problem after using external-secrets to provide a github webhook token

In case, you are in the same situation, the workaround is to:

  • use the creationPolicy: 'Merge'. Example:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: argocd-secret
spec:
  secretStoreRef:
    name: vault-external-secret-store
    kind: ClusterSecretStore
  target:
    name: argocd-secret # Secret name in Kubernetes
    creationPolicy: 'Merge'
    template:
      data:
        webhook.github.secret: "{{ .github_webhook_token }}"
      metadata:
        annotations:
          description: "The WebHook Secret that permits to authenticate the GitHub Webhook"
  # Mapping to local secret from remote secret
  data:
    # https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/#2-configure-argo-cd-with-the-webhook-secret-optional
    - secretKey: github_webhook_token # Prop Name in the secret
      remoteRef:
        key: argocd # Name of the remote secret
        property: github-webhook-token # Prop Name in the remote secret
  • Rerun the install to get a clean argocd-secret
  • Restart the argo-server deployment to get the missing secrets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests