Allow setting a min delay before termination is initiated, e.g. --min-delay-before-term

[miknet]

Feature Description

Imagine an application that is running in Kubernetes pod with Cloud SQL proxy (as a side-car container) to connect to the DB-server. To allow the application to shutdown gracefully, it's important to finish all active client requests before existing. Cloud SQL proxy has a flag -term_timeout that makes proxy to wait for active connections to close before termination, and it might help in some cases. However consider the scenario when the application:

1. Performs some operations on a client's request.
2. Open DB-connection and writes the result there.
3. Returns some result to the client.

If termination of pod is requested when step 1 is running, there will be no active DB-connections and the container with Cloud SQL proxy will be stopped, however when the application proceeds to step 2, it will immediately fail due to unavailability of Cloud SQL proxy.

Alternatives Considered

It's possible to use Alpine- or Debian-based Docker images and configure preStop hook for the container with Cloud SQL proxy to run something like "sleep 10", however the "default" image is distroless, and it would be good if such functionality can be achieved without using external commands.

Another option is to use preStop hook in the container with the application itself. Since preStop hooks are executed before Kubernetes sends SIGTERM to the pod's containers, the application can exit before the Cloud SQL proxy termination is initiated.

[enocom]

Maybe I'm missing something, but wouldn't term_timeout handle this, especially if set to a large value?

For example,

1. Request comes in (proxy sent SIGTERM here)
2. Application connects to DB, reads result
3. Result is returned to client
4. Proxy shuts down.

Possibly related to #828.

[miknet]

It would if an application kept at least 1 active connection for its lifetime.
Otherwise, no matter how large the term_timeout is, the proxy will immediately exit if there are no open connections.
For example,

1. Request comes in, and the app starts to process it. For now it doesn't have any open connection as it does some computational tasks or communicates with other services.
2. Kubernetes starts pod termination and sends SIGTERM to all containers.
3. The Cloud SQL proxy exists immediately as there are no open connections at the moment.
4. The application continues to process the request. Now it needs to read/write from/to the DB and tries to connect to the DB-server, however it fails because the Cloud SQL proxy already exited.

[enocom]

Are you using a client side connection pooling library? That would probably alleviate the problems you're seeing.

[miknet]

Well, I'm aware of potential solutions to this problem, and using connection pool is one of them.
Anyway I'm thinking having such flag might be a good idea. If you disagree, please close the ticket.

[enocom]

In almost all cases we recommend using a client side connection pool -- that would open connections for your app, maintain a pool of connections, and abstract away the concern of connection pool management from your application code (or deployment). I'd suggest giving that a try, and reporting back here if you're still running into issues. Hope that helps.