AWS ElastiCache failover causes "Connection is closed" exception to bubble up to the application

[GFriedrich]

Hey,
I've recently tried the ElastiCache failover on AWS and was expecting to see that all reconnections are handled internally in Lettuce.
Unfortunately I found though, that a "Connection is closed" exception was bubbling up to the application layer.
Is there anything I'm doing wrong here? Or is this expected or even a bug?
Looking at the stacktrace it seems there is already a retry happening but this fails because the connection is closed. So I'm wondering if I'm missing any essential configuration or something.
Any help is very much appreciated.

I'm using Lettuce 6.3.2 and this is wrapped by Spring Data Redis 3.2.7
The application is configured via the following to use the enabled cluster setup in AWS:

spring:
  data:
    redis:
      password: <PASSWORD>
      cluster:
        nodes: <URL>
      lettuce:
        cluster:
          refresh:
            adaptive: true

And the client options are also configured with:

                clientOptionsBuilder
                    .validateClusterNodeMembership(false)
                    .nodeFilter {
                        !it.is(NodeFlag.FAIL) &&
                            !it.is(NodeFlag.EVENTUAL_FAIL) &&
                            !it.is(NodeFlag.HANDSHAKE) &&
                            !it.is(NodeFlag.NOADDR)
                    }

The exception I'm getting still looks like:

io.lettuce.core.RedisException: Connection is closed
    at io.lettuce.core.protocol.DefaultEndpoint.validateWrite(DefaultEndpoint.java:284)
    at io.lettuce.core.protocol.DefaultEndpoint.write(DefaultEndpoint.java:180)
    at io.lettuce.core.protocol.DefaultEndpoint$RetryListener.requeueCommands(DefaultEndpoint.java:1072)
    at io.lettuce.core.protocol.DefaultEndpoint$RetryListener.lambda$potentiallyRequeueCommands$0(DefaultEndpoint.java:1059)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run

org.springframework.data.redis.RedisSystemException: Redis exception
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:72)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:40)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:38)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:308)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.await(LettuceConnection.java:1026)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$3(LettuceConnection.java:445)
    at org.springframework.data.redis.connection.lettuce.LettuceInvoker$Synchronizer.invoke(LettuceInvoker.java:673)
    at org.springframework.data.redis.connection.lettuce.LettuceInvoker$DefaultManyInvocationSpec.toSet(LettuceInvoker.java:639)
    at org.springframework.data.redis.connection.lettuce.LettuceInvoker$ManyInvocationSpec.toSet(LettuceInvoker.java:434)
    at org.springframework.data.redis.connection.lettuce.LettuceZSetCommands.zRevRangeByScore(LettuceZSetCommands.java:223)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.zRevRangeByScore(DefaultedRedisConnection.java:1188)
    at org.springframework.data.redis.connection.RedisZSetCommands.zRevRangeByScore(RedisZSetCommands.java:736)
    at org.springframework.data.redis.core.DefaultZSetOperations.lambda$reverseRangeByScore$24(DefaultZSetOperations.java:303)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:396)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:363)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97)
    at org.springframework.data.redis.core.DefaultZSetOperations.reverseRangeByScore(DefaultZSetOperations.java:303)

[tishun]

Hey @GFriedrich ,

could you check the comments in #2530 ?
I think it is basically the same situation, could you verify?

[GFriedrich]

I'm not sure if that actually helps. Expecting that the connection to the node fails while the command is being executed it causes the existing RetryListener of the channel to be called. This in turn will try to requeue the command which then finally ends up in the very same place as the stacktrace. At no point the disconnectedBehaviour will have an effect here.
I'm also wondering why it is not possible for Lettuce to requeue the command from scratch. I mean I do understand that it is not possible to send it to the same node, but why not simply starting over and looking up the node for the command and send it via this new channel?

[tishun]

I'm not sure if that actually helps. Expecting that the connection to the node fails while the command is being executed it causes the existing RetryListener of the channel to be called. This in turn will try to requeue the command which then finally ends up in the very same place as the stacktrace. At no point the disconnectedBehaviour will have an effect here.

My bad, I misunderstood your problem. Yes, you would still end up with Connection is closed messages.

I'm also wondering why it is not possible for Lettuce to requeue the command from scratch. I mean I do understand that it is not possible to send it to the same node, but why not simply starting over and looking up the node for the command and send it via this new channel?

As Mark mentioned in #2530:

Conceptually, after sending a command to a particular connection (node), we do not know why the command was routed there (by intent, by ASK/MOVED redirection, by routing rules).

We can't be sure that executing these commands on the new topology would have the desired effect.

[GFriedrich]

We can't be sure that executing these commands on the new topology would have the desired effect.

I mean that makes sense, but I'm wondering: Why not bubbling up the exception to the place where you do know about it. Because at some point you know whether there was an intended routing or not. And then decide on whether a retry makes sense or not.
My thinking is that the lookup for the node happens in some place, so why not going back until the point where this happened and then check something like: Was the routing previously forced, bubble up - otherwise retry the command including the lookup.

[tishun]

Why not bubbling up the exception to the place where you do know about it. Because at some point you know whether there was an intended routing or not. And then decide on whether a retry makes sense or not. My thinking is that the lookup for the node happens in some place

TBH I am not sure I can identify such a place myself.
I am willing to entertain a suggestion if you have one.

, so why not going back until the point where this happened and then check something like: Was the routing previously forced, bubble up - otherwise retry the command including the lookup.

I do agree that this is perhaps not the best failover experience.
I will leave this open and will try to give it some more thought when there are some free cycles.

[GFriedrich]

Thanks @tishun
I've for now wrapped all the places where Redis is used with Spring Retry. Definitely not ideal as this clutters the code quite a lot, but I guess the best I can get for now.