All requests to DynamoDB fail with Connection pool shut down

[IDUN-BogdanPi]

Describe the bug

Without any apparent reason, sometimes I get this exception java.lang.IllegalStateException: Connection pool shut down and all requests to DynamoDB fail.

I am using DynamoDB Enhanced Client:

    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>dynamodb-enhanced</artifactId>
    </dependency>

And have a single client running in the app configured with Spring @bean

  @Bean
    public DynamoDbEnhancedClient dynamoDbClient(
            @Value("${dynamodb.use-localstack:false}") boolean useLocalStack,
            @Value("${LOCALSTACK_CUSTOM_ENDPOINT:}") String customEndpoint) {
        DynamoDbClientBuilder builder = DynamoDbClient.builder();

        if (useLocalStack) {
            // Point to LocalStack endpoint
            builder.endpointOverride(URI.create(customEndpoint))
                    .region(Region.of("eu-central-1")); // LocalStack doesn't care much about the region
        } else {
            // AWS Configuration
            builder.region(Region.EU_CENTRAL_1);
        }

        DynamoDbClient dynamoDbClient = builder.build();
        DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
                .dynamoDbClient(dynamoDbClient)
                .build();

        return enhancedClient;
    }

Expected Behavior

I expect that no exceptions arise.

Current Behavior

java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
	at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75)
	at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57)
	...

stacktrace_2.txt
stacktrace_1.txt

Reproduction Steps

This happens very rarely, without an apparent reason. Sometimes even months in between. I could not find what causes it.

Possible Solution

No response

Additional Information/Context

I am performing some queries in Async context using completable future:

CompletableFuture<Void> f = CompletableFuture
                        .runAsync(()

AWS Java SDK version used

software.amazon.awssdk 2.21.37

JDK version used

amazoncorretto:21.0.1

Operating System and version

amazoncorretto:21.0.1

[debora-ito]

Check our Troubleshooting Guide:

FAQ: How do I fix "java.lang.IllegalStateException: Connection pool shut down" error?

[IDUN-BogdanPi]

Thanks for the reply. I did check everything in the docs. There don't seem to be any Errors thrown. I checked the memory usage and it appeared to be relatively low.
DefaultCredentialsProvider we don't use. The SDK lifecycle is managed by Spring via @bean.

So I'm out of ideas. I'll keep digging / profiling, but if there is anything else I could try, please let me know.

[debora-ito]

The only thing I can think of is Spring Bean may be automatically calling close(), see Bean#destroyMethod(). Try disabling the destroy method.

[IDUN-BogdanPi]

Thanks! I'll try that.