Consul-server-acl-init found no secret of type 'kubernetes.io/service-account-token' associated with the consul-auth-method service account #1768
Labels
type/bug
Something isn't working
Comments
2 tasks
2 tasks
|
Just as a note you should use API Gateway 0.5.0 with Consul 1.14.0 otherwise you will likely see errors. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Community Note
Overview of the Issue
The pod consul-server-acl-init logs "2022-12-02T16:54:25.759Z [ERROR] found no secret of type 'kubernetes.io/service-account-token' associated with the consul-auth-method service account" for OpenShift environment with Client Version 4.10.16, Server Version 4.11.12, Kubernetes version v1.24.6+5157800, Consul 1.14.1, Envoy Proxy 1.24.0, and Consul K8s Control Plane 1.0.1.
Reproduction Steps
Steps to reproduce this issue, eg:
--->
Logs
POD
consul-client/client-acl-init
consul-connect-injector/sidecar-injector
consul-server-acl-init
2022-12-02T21:50:47.028Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul.svc: lookup consul-server.consul.svc on 10.11.0.10:53: no such host" 2022-12-02T21:50:47.507Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul.svc: lookup consul-server.consul.svc on 10.11.0.10:53: no such host" 2022-12-02T21:50:48.612Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul.svc: lookup consul-server.consul.svc on 10.11.0.10:53: no such host" 2022-12-02T21:50:50.139Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul.svc: lookup consul-server.consul.svc on 10.11.0.10:53: no such host" 2022-12-02T21:50:51.074Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul.svc: lookup consul-server.consul.svc on 10.11.0.10:53: no such host" 2022-12-02T21:50:53.540Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul.svc: lookup consul-server.consul.svc on 10.11.0.10:53: no such host" 2022-12-02T21:50:57.500Z [INFO] No bootstrap token from previous installation found, continuing on to bootstrapping 2022-12-02T21:51:01.820Z [INFO] Success: bootstrapping ACLs - PUT /v1/acl/bootstrap 2022-12-02T21:51:01.824Z [INFO] Success: writing bootstrap Secret "consul-bootstrap-acl-token" 2022-12-02T21:51:01.824Z [INFO] Setting Consul server tokens 2022-12-02T21:51:01.827Z [INFO] Success: creating agent policy - PUT /v1/acl/policy 2022-12-02T21:51:01.830Z [INFO] Success: creating server token for {10.121.6.108 } - PUT /v1/acl/token 2022-12-02T21:51:01.831Z [INFO] Success: updating server token for {10.121.6.108 } - PUT /v1/agent/token/agent 2022-12-02T21:51:01.831Z [INFO] consul-server-connection-manager: trying to connect to a Consul server 2022-12-02T21:51:01.832Z [INFO] consul-server-connection-manager: discovered Consul servers: addresses=[10.121.6.108:8502] 2022-12-02T21:51:01.832Z [INFO] consul-server-connection-manager: current prioritized list of known Consul servers: addresses=[10.121.6.108:8502] 2022-12-02T21:51:02.034Z [INFO] consul-server-connection-manager: connected to Consul server: address=10.121.6.108:8502 2022-12-02T21:51:02.034Z [INFO] consul-server-connection-manager: updated known Consul servers from watch stream: addresses=[10.121.6.108:8502] 2022-12-02T21:51:02.037Z [INFO] Success: calling /agent/self to get datacenter 2022-12-02T21:51:02.037Z [INFO] Current datacenter: datacenter=dc1 primaryDC=dc1 2022-12-02T21:51:02.039Z [INFO] Success: getting consul-auth-method ServiceAccount 2022-12-02T21:51:02.042Z [INFO] Success: getting consul-auth-method-dockercfg-ncdqg Secret 2022-12-02T21:51:02.042Z [ERROR] found no secret of type 'kubernetes.io/service-account-token' associated with the consul-auth-method service account 2022-12-02T21:51:02.042Z [INFO] consul-server-connection-manager: stoppingSECRETS
CODE
// createAuthMethodTmpl sets up the auth method template based on the connect-injector's service account // jwt token. It is common for both the connect inject auth method and the component auth method // with the option to add namespace specific configuration to the auth method template via `useNS`. func (c *Command) createAuthMethodTmpl(authMethodName string, useNS bool) (api.ACLAuthMethod, error) { // Get the Secret name for the auth method ServiceAccount. var authMethodServiceAccount *apiv1.ServiceAccount serviceAccountName := c.withPrefix("auth-method") err := c.untilSucceeds(fmt.Sprintf("getting %s ServiceAccount", serviceAccountName), func() error { var err error authMethodServiceAccount, err = c.clientset.CoreV1().ServiceAccounts(c.flagK8sNamespace).Get(c.ctx, serviceAccountName, metav1.GetOptions{}) return err }) if err != nil { return api.ACLAuthMethod{}, err } var saSecret *apiv1.Secret var secretNames []string if len(authMethodServiceAccount.Secrets) == 0 { // In Kube 1.24+ there is no automatically generated long term JWT token for a ServiceAccount. // Furthermore, there is no reference to a Secret in the ServiceAccount. Instead we have deployed // a Secret in Helm which references the ServiceAccount and contains a permanent JWT token. secretNames = append(secretNames, c.withPrefix("auth-method")) } else { // ServiceAccounts always have a SecretRef in Kubernetes < 1.24. The Secret contains the JWT token. for _, secretRef := range authMethodServiceAccount.Secrets { secretNames = append(secretNames, secretRef.Name) } } // Because there could be multiple secrets attached to the service account, // we need pick the first one of type "kubernetes.io/service-account-token". // We will fetch the Secrets regardless of whether we created the Secret or Kubernetes did automatically. for _, secretName := range secretNames { var secret *apiv1.Secret err = c.untilSucceeds(fmt.Sprintf("getting %s Secret", secretName), func() error { var err error secret, err = c.clientset.CoreV1().Secrets(c.flagK8sNamespace).Get(c.ctx, secretName, metav1.GetOptions{}) return err }) if secret != nil && secret.Type == apiv1.SecretTypeServiceAccountToken { saSecret = secret break } } if err != nil { return api.ACLAuthMethod{}, err } // This is unlikely to happen since we now deploy the secret through Helm, but should catch any corner-cases // where the secret is not deployed for some reason. if saSecret == nil { return api.ACLAuthMethod{}, fmt.Errorf("found no secret of type 'kubernetes.io/service-account-token' associated with the %s service account", serviceAccountName) }Expected behavior
Find "consul-auth-method kubernetes.io/service-account-token" or "consul-auth-method-token-s7w87 kubernetes.io/service-account-token" instead of "consul-auth-method-dockercfg-ncdqg kubernetes.io/dockercfg ."
Environment details
OpenShift environment with
Additional Context
Consul UI can be logged into with the Bootstrap Token if Access Controls need to be modified.
The text was updated successfully, but these errors were encountered: