Redis driver unable to recover after one of the redis cluster master is killed

[mayankgoyal]

I am using Redis driver 4.0.3 with a redis cluster (4.0.10 - 3 master/3 slaves) and have noticed that the driver caches cluster information early on a fails to update when a cluster master is killed and is failover by a slave. The driver still tries to connect to the older node indefinitely if the master never comes back up.

Scenario:

1. Start a Redis cluster as described here https://redis.io/topics/cluster-tutorial#creating-and-using-a-redis-cluster
2. Connect driver with all 1 to 6 nodes in the cluster

::Redis.new({replica: true, cluster: [{host: "localhost", port: 7000}]})

3. Read/write some random keys in an infinite loop from the cluster with a rescue to make sure ::Redis::CannotConnectError is handled and the script does not crash.
4. Kill a master node
5. Notice the client is stuck and cannot connect to the shard master.
6. Notice the redis cluster has elected the slave as master.
7. Notice the client is still trying to connect to the original master node and not to the slave.

I would assume that this would have worked with a CLUSTER FAILOVER command was issued, but not for crashes as the cluster state is never refreshed in this scenario?

[kitsirota]

@mayankgoyal as far as I am aware, cluster support is rolling out with 4.1.0. There is a beta build for this client here https://rubygems.org/gems/redis/versions/4.1.0.beta1

Here is the PR #716

Let me know if this helps. Also please post back if you run into issues with the 4.1.0.beta1 client, I've been waiting for cluster support for some time and cant wait to roll it out also.

[byroot]

@supercaracal if you have some time to look into this, your help would be very welcome.

If you can't repro we can close.

[supercaracal]

Sure. I will try to look into this issue.

[supercaracal]

to: @byroot
cc: @mayankgoyal

As kitsirota says, the clustering is supported since 4.1.0. I think this issue is reproduced by the followings.

$ bin/console 
irb(main):001:0> Redis.new(foo: :bar).ping
Redis::CannotConnectError (Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED))

Simply, non-effective option like the above will be ignored. So it seems this issue is not a bug.

[byroot]

Thanks @supercaracal