okhttp: detect transport executors with no remaining threads

[ejona86]

We've seen some cases where users are providing fixed-size thread pools to OkHttpChannelBuilder.transportExecutor(). gRPC provides no guarantees how many threads it may need, and queuing a Runnable can prevent transports from functioning.

In particular, grpc-okhttp needs two threads per transport: a reading thread and a sending thread. The reading thread runs for the life of the transport, but the sending thread is only used when something is actively needing to be written to the socket. During handshaking, however, only the reading thread is used, as there's no need to read and write simultaneously. This means the transport can start, but then no RPCs could actually be sent on it, because there are no more threads available.

It seems worthwhile to detect when the executor is misconfigured with too few threads, especially when happy eyeballs is enabled. To do this, during handshaking, we can start a second ("unnecessary") thread and have it blocked doing nothing until the handshake is complete. In addition, the handshake would be blocked on that thread getting scheduled (the Runnable.run() being executed). If it isn't scheduled in X time, then we abort the handshake and kill the transport. If the transport executor is only used for grpc-okhttp, this would guarantee there's at least one free thread to use for writing once the transport is ready and RPCs write data amongst all the transports.