Caddy stops serving http and grpc under heavy grpc-web traffic

[pokaichen]

Caddy 0.11.0, compiled for armv7 with grpc-web plugin.

Steps to reproduce:

1. Start with example code from improbable-eng:
   https://github.com/improbable-eng/grpc-web/blob/master/example/ts/src/index.ts
2. Replace the call to queryBooks(); with a recursive call to
   if (status !== 14)
   getBook();
   Change IP to caddy's server IP.
3. Start grpc service (server)
4. Run caddy with config
   0.0.0.0:8080, localhost:8080 {
   }
   0.0.0.0:55552, localhost:55552 {
   grpc 10.0.4.32:9090 {
   backend_is_insecure
   backend_tls_noverify
   }
   }
   At this point, both the http server and grpc-web proxy is up.
5. Open browser and load index.html.
6. After ~1061 calls to getBook(), http server and grpc-web proxy will no longer respond. Specific number of requests (including index.html and bundle.js) until we get a return status of 14:
   1016,951,1061,1061,1061,1020,1016.
7. At this point, the caddy process is still running, but neither the http server or the grpc-web server works.
   getBook responds with:
   getBook.onEnd.status 14 all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:60000: socket: too many open files"
   

[met-pub]

maybe this is the same reason: It seems caddy still hold the connection to server.