bug: always reports etcd /v3/watch connection timeout 900s after apisix startup

[DroidEye2ONGU]

Current Behavior

There are two servers , one for single etcd and the other for apisix. When I started apisix, everything looks fine, but onec you get past 900s, the apisix log will keep saying this
[error] 41073#41073: *777193 upstream timed out (110: Connection timed out) while reading upstream, client: unix:, server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1" server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1"

When this error started, every apisix functions work fine, no matter proxy or update config through apisix-dashboard, and etcd can also receive requests. I can also send request from the apisix server to etcd server.

Expected Behavior

I hope this error won't appear

Error Logs

[error] 41073#41073: *777193 upstream timed out (110: Connection timed out) while reading upstream, client: unix:, server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1" server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1"

Steps to Reproduce

1. deploy single etcd node on server.
2. install apisix by apisix installation-guide using rpm. By the way, the server is online, it cannot touch internet.
3. change some apisix config, mainly ext java-plugin, additional nginx config, and etcd address.
   etcd:
   host:
   • "http://etcdhost:2379"
     prefix: /apisix
     use_grpc: false
     timeout: 30
4. start apisix using systemctl start apisix

Environment

• APISIX version (run apisix version): 3.3.0
• Operating system (run uname -a): Linux archive2 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0 (apisix said it's 3.5.0 but it's actually 3.5.4 )
• APISIX Dashboard version, if relevant: 3.0.1
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[shreemaan-abhishek]

works fine for me.

[shreemaan-abhishek]

are you running apisix on your laptop?

[DroidEye2ONGU]

No.I have two environments running apisix. One of them running on kubernetes, and it works fine. The other is running on a project production server, installed by offline rpm. Although there are differences in deployment modes, every function is works fine, just output the error log, so it's weird.

[wklken]

#9336

[shreemaan-abhishek]

@DroidEye2ONGU, are you using gRPC for etcd connection?

[DroidEye2ONGU]

@DroidEye2ONGU, are you using gRPC for etcd connection?

I have tried either disable rpc or enable, the error log always happended

[kingluo]

The conf server is removed, so the etcd communication rolls back to what 2.x versions behave like, i.e. no matter CP or DP, connects to etcd directly. Closed by #10012