Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

consul-connect-inject-init failing with transparent proxy #568

Closed
andriktr opened this issue Jul 22, 2021 · 10 comments
Closed

consul-connect-inject-init failing with transparent proxy #568

andriktr opened this issue Jul 22, 2021 · 10 comments
Labels
type/bug Something isn't working

Comments

@andriktr
Copy link

andriktr commented Jul 22, 2021 •
edited
Loading

Hello,
We upgraded our consul cluster and k8s agent till 1.10.0.
We now tried to use a transparent proxy which is enabled by default, however when we try to deploy app the consul-connect-inject-init container is failing with following errors

2021-07-22T15:01:41.747Z [INFO]  Consul login complete
2021-07-22T15:01:41.750Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:42.751Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:43.752Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:44.753Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:45.754Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:46.755Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:47.756Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:48.758Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:49.759Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:50.760Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:51.761Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:52.762Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:53.763Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:54.764Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:55.765Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:56.767Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:57.816Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:58.817Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:01:59.818Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:00.820Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:01.821Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:02.822Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:03.823Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:04.825Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:05.826Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:06.828Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:07.828Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:08.829Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:09.831Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:10.832Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:02:11.838Z [INFO]  Registered service has been detected: service=backend
2021-07-22T15:02:11.838Z [INFO]  Registered service has been detected: service=backend-sidecar-proxy
2021-07-22T15:02:11.838Z [INFO]  Connect initialization completed
==> Failed to create configuration to apply traffic redirection rules: failed to fetch proxy service from Consul Agent: Unexpected response code: 403 (Permission denied)

After the ~7 restarts consul-connect-inject-init container logs output changes to the following:

2021-07-22T15:15:46.620Z [INFO]  Consul login complete
2021-07-22T15:15:46.623Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:47.624Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:48.625Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:49.626Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:50.627Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:51.628Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:52.630Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:53.631Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:54.632Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:55.633Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:56.634Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:57.635Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:58.636Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:15:59.638Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:00.639Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:01.640Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:02.642Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:03.644Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:04.645Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:05.646Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:06.647Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:07.648Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:08.649Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:09.650Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:10.652Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:11.653Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:12.654Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:13.655Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:14.657Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:15.658Z [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)"
2021-07-22T15:16:16.661Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:17.716Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:18.718Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:19.719Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:20.722Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:21.723Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:22.725Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:23.726Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:24.728Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:25.730Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:25.730Z [INFO]  Check to ensure a Kubernetes service has been created for this application.
2021-07-22T15:16:26.731Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:27.733Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:28.740Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:29.741Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:30.744Z [INFO]  Unable to find registered services; retrying
2021-07-22T15:16:31.745Z [INFO]  Unable to find registered services; retrying

Sure I double checked and a k8s service which is required for all connect services since transparent proxy feature was added exist for my app. Here is a full yaml for my deployment

apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend
  namespace: infra
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: infra
spec:
  selector:
    app: backend
  type: ClusterIP
  ports:
  - name: http
    protocol: TCP
    port: 9090
    targetPort: 9090
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name:  backend
  namespace: infra
  labels:
    app:  backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels: 
        app: backend
      annotations: 
        "consul.hashicorp.com/connect-inject": "true"
    spec:
      serviceAccountName: backend
      containers:
      - name: backend
        image: nicholasjackson/fake-service:v0.22.4
        ports:
        - containerPort: 9090
        env:
        - name: "LISTEN_ADDR"
          value: "0.0.0.0:9090"
        - name: "UPSTREAM_URIS"    
          value: "http://backend.infra.svc.cluster.local"
        - name: "NAME"
          value: "backend"
        - name: "MESSAGE"
          value: "Response from Backend"
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
          limits:
            cpu: "200m"
            memory: "200Mi"

I also noticed that each container restart generate a new token for the pod:
image

K8S node agents show the following in the logs

%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:17.082Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%22backend-557d5d9cb4-pqkbv%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:18.083Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%22backend-557d5d9cb4-pqkbv%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:19.085Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%22backend-557d5d9cb4-pqkbv%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:20.087Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%22backend-557d5d9cb4-pqkbv%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:21.088Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%22backend-557d5d9cb4-pqkbv%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:22.089Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%22backend-557d5d9cb4-pqkbv%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:23.091Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%22backend-557d5d9cb4-pqkbv%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22infra%22 from=10.162.208.31:54082 error="ACL not found"
2021-07-22T15:07:24.792Z [WARN]  agent: Check socket connection failed: check=default/service:backend-557d5d9cb4-pqkbv-backend-sidecar-proxy:1 error="dial tcp 10.162.208.31:20000: connect: connection refused"
2021-07-22T15:07:24.792Z [WARN]  agent: Check is now critical: check=default/service:backend-557d5d9cb4-pqkbv-backend-sidecar-proxy:1
2021-07-22T15:07:32.318Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/service/backend-557d5d9cb4-pqkbv-backend-sidecar-proxy from=10.162.208.31:54586 error="Permission denied"
2021-07-22T15:07:34.794Z [WARN]  agent: Check socket connection failed: check=default/service:backend-557d5d9cb4-pqkbv-backend-sidecar-proxy:1 error="dial tcp 10.162.208.31:20000: connect: connection refused"

The service status in consul is:
image

I don't see any useful info on our consul server agent (which are outside of K8S).

Also if I'm disabling transparent proxy by adding annotation consul.hashicorp.com/transparent-proxy": "false" to my deployment everything works fine.

Obviously it something related to the ACL , but currently can't catch what is wrong.

Thanks in advance.
@andriktr andriktr added the type/bug Something isn't working label Jul 22, 2021
@andriktr andriktr changed the title consul-connect-inject-init failing with tranparent proxy consul-connect-inject-init failing with transparent proxy Jul 22, 2021
@ishustava
Copy link
Contributor

Hey @andriktr

could you also provide your Helm values so we can reproduce this issue?

I also noticed that each container restart generate a new token for the pod:
image

Sorry about that! this is a regression that was introduced in that release that we're currently working on fixing.

@ishustava ishustava added the waiting-reply Waiting on the issue creator for a response before taking further action label Jul 22, 2021
@ishustava
Copy link
Contributor

ishustava commented Jul 22, 2021 •
edited
Loading

Could you also share the status of your pods and logs from the servers? We suspect it might be related to the health of the servers and them not being in the consistent state that's causing this flakey ACL behavior.

@andriktr
Copy link
Author

Hello @ishustava,
Thank for quick reply.
I'am attaching helm values file yaml is not supported in git so it's in txt
values_redacted.txt

From which pods should I collect the logs?
Here is the status of consul pods on the k8s cluster:
image

Consul server cluster itself deployed to outside of k8s on 3 servers:
Here the lates logs from them:
consul-test-vm-2:

Jul 22 19:37:53 consul-test-vm-2 consul[27866]: 2021-07-22T19:37:53.346+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.52:8300 0 consul-test-vm-2.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.52:8300: operation was canceled". Reconnecting...
Jul 22 19:37:53 consul-test-vm-2 consul[27866]: 2021-07-22T19:37:53.346+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.52:8300 0 consul-test-vm-2.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.52:8300: operation was canceled". Reconnecting...
Jul 22 19:38:20 consul-test-vm-2 consul[27866]: 2021-07-22T19:38:20.141+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:38:20 consul-test-vm-2 consul[27866]: 2021-07-22T19:38:20.141+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:40:35 consul-test-vm-2 consul[27866]: 2021-07-22T19:40:35.105+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:40:35 consul-test-vm-2 consul[27866]: 2021-07-22T19:40:35.105+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:41:00 consul-test-vm-2 consul[27866]: 2021-07-22T19:41:00.758+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:41:00 consul-test-vm-2 consul[27866]: 2021-07-22T19:41:00.758+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:50:35 consul-test-vm-2 consul[27866]: 2021-07-22T19:50:35.528+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:50:35 consul-test-vm-2 consul[27866]: 2021-07-22T19:50:35.528+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:51:53 consul-test-vm-2 consul[27866]: 2021-07-22T19:51:53.423+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:51:53 consul-test-vm-2 consul[27866]: 2021-07-22T19:51:53.423+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:58:36 consul-test-vm-2 consul[27866]: 2021-07-22T19:58:36.215+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:58:36 consul-test-vm-2 consul[27866]: 2021-07-22T19:58:36.215+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.458+0300 [INFO]  agent.fsm: snapshot created: duration=67.705µs
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.458+0300 [INFO]  agent.server.raft: starting snapshot up to: index=2340186
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.458+0300 [INFO]  agent.server.snapshot: creating new snapshot: path=/opt/consul/raft/snapshots/70-2340186-1626973223458.tmp
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.458+0300 [INFO]  agent.fsm: snapshot created: duration=67.705µs
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.458+0300 [INFO]  agent.server.raft: starting snapshot up to: index=2340186
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.458+0300 [INFO]  agent.server.snapshot: creating new snapshot: path=/opt/consul/raft/snapshots/70-2340186-1626973223458.tmp
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.545+0300 [INFO]  agent.server.snapshot: reaping snapshot: path=/opt/consul/raft/snapshots/70-2339571-1626969622497
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.545+0300 [INFO]  agent.server.snapshot: reaping snapshot: path=/opt/consul/raft/snapshots/70-2339571-1626969622497
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.545+0300 [INFO]  agent.server.raft: compacting logs: from=2329634 to=2329946
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.545+0300 [INFO]  agent.server.raft: compacting logs: from=2329634 to=2329946
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.558+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2340186
Jul 22 20:00:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:00:23.558+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2340186
Jul 22 20:11:02 consul-test-vm-2 consul[27866]: 2021-07-22T20:11:02.434+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:11:02 consul-test-vm-2 consul[27866]: 2021-07-22T20:11:02.434+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:12:30 consul-test-vm-2 consul[27866]: 2021-07-22T20:12:30.624+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:12:30 consul-test-vm-2 consul[27866]: 2021-07-22T20:12:30.624+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:21:47 consul-test-vm-2 consul[27866]: 2021-07-22T20:21:47.178+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 20:21:47 consul-test-vm-2 consul[27866]: 2021-07-22T20:21:47.178+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 20:22:38 consul-test-vm-2 consul[27866]: 2021-07-22T20:22:38.773+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:22:38 consul-test-vm-2 consul[27866]: 2021-07-22T20:22:38.773+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:30:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:23.981+0300 [INFO]  agent.fsm: snapshot created: duration=65.004µs
Jul 22 20:30:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:23.982+0300 [INFO]  agent.server.raft: starting snapshot up to: index=2340496
Jul 22 20:30:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:23.982+0300 [INFO]  agent.server.snapshot: creating new snapshot: path=/opt/consul/raft/snapshots/70-2340496-1626975023982.tmp
Jul 22 20:30:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:23.981+0300 [INFO]  agent.fsm: snapshot created: duration=65.004µs
Jul 22 20:30:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:23.982+0300 [INFO]  agent.server.raft: starting snapshot up to: index=2340496
Jul 22 20:30:23 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:23.982+0300 [INFO]  agent.server.snapshot: creating new snapshot: path=/opt/consul/raft/snapshots/70-2340496-1626975023982.tmp
Jul 22 20:30:24 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:24.047+0300 [INFO]  agent.server.snapshot: reaping snapshot: path=/opt/consul/raft/snapshots/70-2339873-1626971423004
Jul 22 20:30:24 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:24.047+0300 [INFO]  agent.server.snapshot: reaping snapshot: path=/opt/consul/raft/snapshots/70-2339873-1626971423004
Jul 22 20:30:24 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:24.048+0300 [INFO]  agent.server.raft: compacting logs: from=2329947 to=2330256
Jul 22 20:30:24 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:24.048+0300 [INFO]  agent.server.raft: compacting logs: from=2329947 to=2330256
Jul 22 20:30:24 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:24.064+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2340496
Jul 22 20:30:24 consul-test-vm-2 consul[27866]: 2021-07-22T20:30:24.064+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2340496
Jul 22 20:43:02 consul-test-vm-2 consul[27866]: 2021-07-22T20:43:02.587+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:43:02 consul-test-vm-2 consul[27866]: 2021-07-22T20:43:02.587+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:43:07 consul-test-vm-2 consul[27866]: 2021-07-22T20:43:07.789+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:43:07 consul-test-vm-2 consul[27866]: 2021-07-22T20:43:07.789+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0

consul-test-vm-1

Jul 22 18:02:08 consul-test-vm-1 consul[12187]: 2021-07-22T18:02:08.258+0300 [INFO]  agent.server.raft: compacting logs: from=2312321 to=2328718
Jul 22 18:02:08 consul-test-vm-1 consul[12187]: 2021-07-22T18:02:08.258+0300 [INFO]  agent.server.raft: compacting logs: from=2312321 to=2328718
Jul 22 18:02:08 consul-test-vm-1 consul[12187]: 2021-07-22T18:02:08.319+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2338958
Jul 22 18:02:08 consul-test-vm-1 consul[12187]: 2021-07-22T18:02:08.319+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2338958
Jul 22 18:08:25 consul-test-vm-1 consul[12187]: 2021-07-22T18:08:25.987+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:08:25 consul-test-vm-1 consul[12187]: 2021-07-22T18:08:25.987+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:15:39 consul-test-vm-1 consul[12187]: 2021-07-22T18:15:39.581+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:15:39 consul-test-vm-1 consul[12187]: 2021-07-22T18:15:39.581+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:28:32 consul-test-vm-1 consul[12187]: 2021-07-22T18:28:32.951+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:28:32 consul-test-vm-1 consul[12187]: 2021-07-22T18:28:32.951+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:36:04 consul-test-vm-1 consul[12187]: 2021-07-22T18:36:04.991+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 18:36:04 consul-test-vm-1 consul[12187]: 2021-07-22T18:36:04.991+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 18:46:24 consul-test-vm-1 consul[12187]: 2021-07-22T18:46:24.326+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:46:24 consul-test-vm-1 consul[12187]: 2021-07-22T18:46:24.326+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:49:30 consul-test-vm-1 consul[12187]: 2021-07-22T18:49:30.515+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:49:30 consul-test-vm-1 consul[12187]: 2021-07-22T18:49:30.515+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:59:34 consul-test-vm-1 consul[12187]: 2021-07-22T18:59:34.398+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:59:34 consul-test-vm-1 consul[12187]: 2021-07-22T18:59:34.398+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:20:38 consul-test-vm-1 consul[12187]: 2021-07-22T19:20:38.330+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:20:38 consul-test-vm-1 consul[12187]: 2021-07-22T19:20:38.330+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:28:09 consul-test-vm-1 consul[12187]: 2021-07-22T19:28:09.246+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:28:09 consul-test-vm-1 consul[12187]: 2021-07-22T19:28:09.246+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:30:33 consul-test-vm-1 consul[12187]: 2021-07-22T19:30:33.779+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:30:33 consul-test-vm-1 consul[12187]: 2021-07-22T19:30:33.779+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:41:30 consul-test-vm-1 consul[12187]: 2021-07-22T19:41:30.033+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:41:30 consul-test-vm-1 consul[12187]: 2021-07-22T19:41:30.033+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:51:31 consul-test-vm-1 consul[12187]: 2021-07-22T19:51:31.261+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:51:31 consul-test-vm-1 consul[12187]: 2021-07-22T19:51:31.261+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:01:52 consul-test-vm-1 consul[12187]: 2021-07-22T20:01:52.877+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:01:52 consul-test-vm-1 consul[12187]: 2021-07-22T20:01:52.877+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:02:26 consul-test-vm-1 consul[12187]: 2021-07-22T20:02:26.081+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:02:26 consul-test-vm-1 consul[12187]: 2021-07-22T20:02:26.081+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:09:01 consul-test-vm-1 consul[12187]: 2021-07-22T20:09:01.137+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:09:01 consul-test-vm-1 consul[12187]: 2021-07-22T20:09:01.137+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:11:59 consul-test-vm-1 consul[12187]: 2021-07-22T20:11:59.632+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:11:59 consul-test-vm-1 consul[12187]: 2021-07-22T20:11:59.632+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:14:03 consul-test-vm-1 consul[12187]: 2021-07-22T20:14:03.987+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:14:03 consul-test-vm-1 consul[12187]: 2021-07-22T20:14:03.987+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:19:18 consul-test-vm-1 consul[12187]: 2021-07-22T20:19:18.192+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:19:18 consul-test-vm-1 consul[12187]: 2021-07-22T20:19:18.192+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:21:28 consul-test-vm-1 consul[12187]: 2021-07-22T20:21:28.414+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:21:28 consul-test-vm-1 consul[12187]: 2021-07-22T20:21:28.414+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:22:21 consul-test-vm-1 consul[12187]: 2021-07-22T20:22:21.866+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:22:21 consul-test-vm-1 consul[12187]: 2021-07-22T20:22:21.866+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:29:28 consul-test-vm-1 consul[12187]: 2021-07-22T20:29:28.848+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:29:28 consul-test-vm-1 consul[12187]: 2021-07-22T20:29:28.848+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:31:47 consul-test-vm-1 consul[12187]: 2021-07-22T20:31:47.618+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:31:47 consul-test-vm-1 consul[12187]: 2021-07-22T20:31:47.618+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:49:59 consul-test-vm-1 consul[12187]: 2021-07-22T20:49:59.288+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:49:59 consul-test-vm-1 consul[12187]: 2021-07-22T20:49:59.288+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0

consul-test-vm-3

Jul 22 15:54:14 consul-test-vm-3 consul[9948]: 2021-07-22T15:54:14.113+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 15:54:14 consul-test-vm-3 consul[9948]: 2021-07-22T15:54:14.113+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:02:36 consul-test-vm-3 consul[9948]: 2021-07-22T16:02:36.967+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 16:02:36 consul-test-vm-3 consul[9948]: 2021-07-22T16:02:36.967+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 16:04:49 consul-test-vm-3 consul[9948]: 2021-07-22T16:04:49.392+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:04:49 consul-test-vm-3 consul[9948]: 2021-07-22T16:04:49.392+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:23:09 consul-test-vm-3 consul[9948]: 2021-07-22T16:23:09.567+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 16:23:09 consul-test-vm-3 consul[9948]: 2021-07-22T16:23:09.567+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 16:35:12 consul-test-vm-3 consul[9948]: 2021-07-22T16:35:12.429+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:35:12 consul-test-vm-3 consul[9948]: 2021-07-22T16:35:12.429+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:45:27 consul-test-vm-3 consul[9948]: 2021-07-22T16:45:27.499+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:45:27 consul-test-vm-3 consul[9948]: 2021-07-22T16:45:27.499+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:55:44 consul-test-vm-3 consul[9948]: 2021-07-22T16:55:44.680+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:55:44 consul-test-vm-3 consul[9948]: 2021-07-22T16:55:44.680+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 16:56:20 consul-test-vm-3 consul[9948]: 2021-07-22T16:56:20.107+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 16:56:20 consul-test-vm-3 consul[9948]: 2021-07-22T16:56:20.107+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:04:08 consul-test-vm-3 consul[9948]: 2021-07-22T17:04:08.583+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:04:08 consul-test-vm-3 consul[9948]: 2021-07-22T17:04:08.583+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:06:04 consul-test-vm-3 consul[9948]: 2021-07-22T17:06:04.431+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:06:04 consul-test-vm-3 consul[9948]: 2021-07-22T17:06:04.431+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:07:16 consul-test-vm-3 consul[9948]: 2021-07-22T17:07:16.264+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 17:07:16 consul-test-vm-3 consul[9948]: 2021-07-22T17:07:16.264+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 17:16:08 consul-test-vm-3 consul[9948]: 2021-07-22T17:16:08.633+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:16:08 consul-test-vm-3 consul[9948]: 2021-07-22T17:16:08.633+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:24:37 consul-test-vm-3 consul[9948]: 2021-07-22T17:24:37.421+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:24:37 consul-test-vm-3 consul[9948]: 2021-07-22T17:24:37.421+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:26:55 consul-test-vm-3 consul[9948]: 2021-07-22T17:26:55.101+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:26:55 consul-test-vm-3 consul[9948]: 2021-07-22T17:26:55.101+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:37:52 consul-test-vm-3 consul[9948]: 2021-07-22T17:37:52.943+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:37:52 consul-test-vm-3 consul[9948]: 2021-07-22T17:37:52.943+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:47:43 consul-test-vm-3 consul[9948]: 2021-07-22T17:47:43.777+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:47:43 consul-test-vm-3 consul[9948]: 2021-07-22T17:47:43.777+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 17:48:10 consul-test-vm-3 consul[9948]: 2021-07-22T17:48:10.442+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:48:10 consul-test-vm-3 consul[9948]: 2021-07-22T17:48:10.442+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 17:58:11 consul-test-vm-3 consul[9948]: 2021-07-22T17:58:11.563+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 17:58:11 consul-test-vm-3 consul[9948]: 2021-07-22T17:58:11.563+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 18:08:29 consul-test-vm-3 consul[9948]: 2021-07-22T18:08:29.396+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:08:29 consul-test-vm-3 consul[9948]: 2021-07-22T18:08:29.396+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:10:49 consul-test-vm-3 consul[9948]: 2021-07-22T18:10:49.541+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 18:10:49 consul-test-vm-3 consul[9948]: 2021-07-22T18:10:49.541+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 18:18:27 consul-test-vm-3 consul[9948]: 2021-07-22T18:18:27.222+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:18:27 consul-test-vm-3 consul[9948]: 2021-07-22T18:18:27.222+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:18:36 consul-test-vm-3 consul[9948]: 2021-07-22T18:18:36.440+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:18:36 consul-test-vm-3 consul[9948]: 2021-07-22T18:18:36.440+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:25:45 consul-test-vm-3 consul[9948]: 2021-07-22T18:25:45.738+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:25:45 consul-test-vm-3 consul[9948]: 2021-07-22T18:25:45.738+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:29:01 consul-test-vm-3 consul[9948]: 2021-07-22T18:29:01.428+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:29:01 consul-test-vm-3 consul[9948]: 2021-07-22T18:29:01.428+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:39:10 consul-test-vm-3 consul[9948]: 2021-07-22T18:39:10.372+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:39:10 consul-test-vm-3 consul[9948]: 2021-07-22T18:39:10.372+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:49:28 consul-test-vm-3 consul[9948]: 2021-07-22T18:49:28.835+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:49:28 consul-test-vm-3 consul[9948]: 2021-07-22T18:49:28.835+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:54:41 consul-test-vm-3 consul[9948]: 2021-07-22T18:54:41.153+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 18:54:41 consul-test-vm-3 consul[9948]: 2021-07-22T18:54:41.153+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.53:8300 0 consul-test-vm-1.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.53:8300: operation was canceled". Reconnecting...
Jul 22 18:56:42 consul-test-vm-3 consul[9948]: 2021-07-22T18:56:42.402+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:56:42 consul-test-vm-3 consul[9948]: 2021-07-22T18:56:42.402+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:59:29 consul-test-vm-3 consul[9948]: 2021-07-22T18:59:29.076+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:59:29 consul-test-vm-3 consul[9948]: 2021-07-22T18:59:29.076+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 18:59:48 consul-test-vm-3 consul[9948]: 2021-07-22T18:59:48.760+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 18:59:48 consul-test-vm-3 consul[9948]: 2021-07-22T18:59:48.760+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:04:40 consul-test-vm-3 consul[9948]: 2021-07-22T19:04:40.977+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 19:04:40 consul-test-vm-3 consul[9948]: 2021-07-22T19:04:40.977+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 19:07:15 consul-test-vm-3 consul[9948]: 2021-07-22T19:07:15.044+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:07:15 consul-test-vm-3 consul[9948]: 2021-07-22T19:07:15.044+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:10:04 consul-test-vm-3 consul[9948]: 2021-07-22T19:10:04.732+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:10:04 consul-test-vm-3 consul[9948]: 2021-07-22T19:10:04.732+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:17:43 consul-test-vm-3 consul[9948]: 2021-07-22T19:17:43.063+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:17:43 consul-test-vm-3 consul[9948]: 2021-07-22T19:17:43.063+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.922+0300 [INFO]  agent.fsm: snapshot created: duration=152.204µs
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.922+0300 [INFO]  agent.server.raft: starting snapshot up to: index=2339843
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.922+0300 [INFO]  agent.server.snapshot: creating new snapshot: path=/opt/consul/raft/snapshots/70-2339843-1626971250922.tmp
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.922+0300 [INFO]  agent.fsm: snapshot created: duration=152.204µs
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.922+0300 [INFO]  agent.server.raft: starting snapshot up to: index=2339843
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.922+0300 [INFO]  agent.server.snapshot: creating new snapshot: path=/opt/consul/raft/snapshots/70-2339843-1626971250922.tmp
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.968+0300 [INFO]  agent.server.snapshot: reaping snapshot: path=/opt/consul/raft/snapshots/70-2307073-1626781512156
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.968+0300 [INFO]  agent.server.snapshot: reaping snapshot: path=/opt/consul/raft/snapshots/70-2307073-1626781512156
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.968+0300 [INFO]  agent.server.raft: compacting logs: from=2313220 to=2329603
Jul 22 19:27:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:30.968+0300 [INFO]  agent.server.raft: compacting logs: from=2313220 to=2329603
Jul 22 19:27:31 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:31.000+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2339843
Jul 22 19:27:31 consul-test-vm-3 consul[9948]: 2021-07-22T19:27:31.000+0300 [INFO]  agent.server.raft: snapshot complete up to: index=2339843
Jul 22 19:30:32 consul-test-vm-3 consul[9948]: 2021-07-22T19:30:32.808+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:30:32 consul-test-vm-3 consul[9948]: 2021-07-22T19:30:32.808+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 19:48:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:48:30.232+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 19:48:30 consul-test-vm-3 consul[9948]: 2021-07-22T19:48:30.232+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:00:51 consul-test-vm-3 consul[9948]: 2021-07-22T20:00:51.351+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:00:51 consul-test-vm-3 consul[9948]: 2021-07-22T20:00:51.351+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:08:43 consul-test-vm-3 consul[9948]: 2021-07-22T20:08:43.842+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:08:43 consul-test-vm-3 consul[9948]: 2021-07-22T20:08:43.842+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:32:05 consul-test-vm-3 consul[9948]: 2021-07-22T20:32:05.146+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:32:05 consul-test-vm-3 consul[9948]: 2021-07-22T20:32:05.146+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:32:47 consul-test-vm-3 consul[9948]: 2021-07-22T20:32:47.619+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:32:47 consul-test-vm-3 consul[9948]: 2021-07-22T20:32:47.619+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:33:03 consul-test-vm-3 consul[9948]: 2021-07-22T20:33:03.934+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:33:03 consul-test-vm-3 consul[9948]: 2021-07-22T20:33:03.934+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:39:51 consul-test-vm-3 consul[9948]: 2021-07-22T20:39:51.518+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:39:51 consul-test-vm-3 consul[9948]: 2021-07-22T20:39:51.518+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-terminating-gateway-0
Jul 22 20:41:45 consul-test-vm-3 consul[9948]: 2021-07-22T20:41:45.971+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:41:45 consul-test-vm-3 consul[9948]: 2021-07-22T20:41:45.971+0300 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...
Jul 22 20:42:02 consul-test-vm-3 consul[9948]: 2021-07-22T20:42:02.574+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0
Jul 22 20:42:02 consul-test-vm-3 consul[9948]: 2021-07-22T20:42:02.574+0300 [WARN]  agent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=dev-ingress-gateway-0

Seems the only real warning on consul servers is

 [WARN]  agent: grpc: addrConn.createTransport failed to connect to {10.162.34.54:8300 0 consul-test-vm-3.azure-dc-test <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.162.34.54:8300: operation was canceled". Reconnecting...

Not sure does it have impact for the issue.

consul members returns the following:
image

@andriktr
Copy link
Author

Hey @ishustava, do you have any insights or suggestions regarding this issue?
I did an upgrade for another consul environment which has pretty much similar configuration and here we also experiencing similar behaviour. Should I register official support ticket for this issue?

@ishustava
Copy link
Contributor

ishustava commented Jul 27, 2021 •
edited
Loading

Hey @andriktr

I think your problem is due to #570.

The only workaround for now is to enable DNS by setting dns.enabled to true in your helm values. This will set up anonymous token to be able to read services.

@andriktr
Copy link
Author

@ishustava Thanks for clarification will wait for the fix then. Hopefully it will come soon 😉.

ishustava added a commit that referenced this issue Jul 27, 2021
@ishustava
connect: redirect-traffic command should pass ACL token when ACL are …
2dfffed 
…enabled

Fixes #570,#568
@ishustava ishustava mentioned this issue Jul 27, 2021
Merged
2 tasks
@ishustava ishustava removed the waiting-reply Waiting on the issue creator for a response before taking further action label Jul 27, 2021
ishustava added a commit that referenced this issue Jul 30, 2021
@ishustava
connect: redirect-traffic command should pass ACL token when ACL are …
4a50fda 
…enabled (#576)

Fixes #570,#568
@ishustava
Copy link
Contributor

Closing as the fix is now merged with #576

@chenjinkai
Copy link

@ishustava I have used the image “hashicorpdev/consul-k8s:2dfffed”, and turn off the dns.enabled=false. but i still got the problem as following.
2021-08-18T02:34:53.420Z [ERROR] Consul login failed; retrying: error="error logging in: Unexpected response code: 403 (rpc error making call: Permission denied)"
2021-08-18T02:34:54.446Z [ERROR] Consul login failed; retrying: error="error logging in: Unexpected response code: 403 (rpc error making call: rpc error making call: Permission denied)"
2021-08-18T02:34:55.461Z [ERROR] Consul login failed; retrying: error="error logging in: Unexpected response code: 403 (rpc error making call: rpc error making call: Permission denied)"
2021-08-18T02:34:56.474Z [ERROR] Consul login failed; retrying: error="error logging in: Unexpected response code: 403 (rpc error making call: Permission denied)"
2021-08-18T02:34:56.474Z [ERROR] Hit maximum retries for consul login: error="error logging in: Unexpected response code: 403 (rpc error making call: Permission denied)"

my configurations

@lkysow
Copy link
Member

lkysow commented Aug 18, 2021

@chenjinkai sorry you're having some issues. Would you be okay opening up a separate issue with the same information you commented above? That will make it easier to track your specific problem.

@chenjinkai
Copy link

@lkysow ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lkysow @chenjinkai @ishustava @andriktr