Provide a way to customize the graceful shutdown delay of Netty EventLoopGroups

[trustin]

Server and ClientFactory currently calls EventLoopGroup.shutdownGracefully() to terminate Netty EventLoopGroups when they stop or close. shutdownGracefully() incurs 2-3 second delay and it can make the applications suffer when it needs to start and stop frequently.

It'd be nice if Armeria provides a user to customize the graceful shutdown delay of EventLoopGroups, like it does for Server requests, so that they can minimize the amount of time required for tearing down a Server or a ClientFactory.

We could, for example, allow a user to specify a config instead of boolean shutdownOnStop in ServerBuilder and ClientFactoryBuilder methods, e.g.

public class GracefulShutdownSpec {
    // External resource managed by the caller - don't shut down
    public static GracefulShutdownSpec ofExternal() { ... }
    // Don't shut down gracefully - shut down immediately
    public static GracefulShutdownSpec ofImmediate() { ... }
    // Shut down gracefully
    public static GracefulShutdownSpec of(quiet period, hard timeout) { ... }
}

Server
  .builder()
  .workerGroup(..., GracefulShutdownSpec.of...)
  // Could be reused for the existing graceful shutdown settings
  // Note: `ofExternal` is not allowed here.
  .gracefulShutdownTimeout(GracefulShutdownSpec.of...)
  ...

[novoj]

Observations from current state:

If all pools are configured this way:

serverBuilder
            .gracefulShutdownTimeout(Duration.ZERO, Duration.ZERO)
            .blockingTaskExecutor(EventLoopGroups.newEventLoopGroup(apiOptions.serviceWorkerGroupThreadsAsInt()), false)
            .serviceWorkerGroup(EventLoopGroups.newEventLoopGroup(apiOptions.serviceWorkerGroupThreadsAsInt()), false)
            .workerGroup(EventLoopGroups.newEventLoopGroup(apiOptions.workerGroupThreadsAsInt()), false);

It still takes about 2 seconds to shut down the server. I've traced the issue down and it seems, that it originates here: com/linecorp/armeria/server/Server.java:678:

bossGroup.shutdownGracefully();

Which then invokes in io.netty.util.concurrent.AbstractEventExecutorGroup:

@Override
public Future<?> shutdownGracefully() {
   return shutdownGracefully(DEFAULT_SHUTDOWN_QUIET_PERIOD, DEFAULT_SHUTDOWN_TIMEOUT, TimeUnit.SECONDS);
}

Where:

static final long DEFAULT_SHUTDOWN_QUIET_PERIOD = 2;
static final long DEFAULT_SHUTDOWN_TIMEOUT = 15;

It seems, that current settings set in gracefulShutdownTimeout(Duration.ZERO, Duration.ZERO) are not propagated to this place.

I have no opinion on newly proposed configuration API change - I'm just trying to document current behavior which prolongs my automated tests. Thank you.