Skip to content

activity cache replication

David edited this page Aug 20, 2015 · 4 revisions

If there is a communications fault for the master redis instance, normally activity-cache0, haproxy will automatically start using the slave, activity-cache1. However when the redis master re-joins the cluster haproxy does not switch back to the master.

This procedure should reset the slave-master setup to normal, activity-cache0=master (now called 0) and activity-cache1=slave (now called 1).

Obviously check the for the private IPs of activity-cache{0,1} in case they've changed for some reason.

  1. first swap master/slave relationship;

    on 1...

    redis-cli slaveof no one

    on 0...

    redis-cli slaveof 10.224.14.104 6379
    redis-cli sync
  2. wait until replication is normalized

    on 0

    redis-cli info | grep master_

    if master_sync_in_progress=0 then no sync is in progress

    if master_last_io_seconds_ago>0 but only a few seconds then a sync has been done recently

    A better definitive check would be nice

  3. swap master/slave back again

    on 0...

    redis-cli slaveof no one

    on 1...

    redis-cli slaveof 10.224.14.103 6379
    redis-cli sync
  4. reload haproxy

    on proxy0...

    oldpid=$(ps -A | awk '/redis/ { print $1 }')
    service haproxy reload
    kill $oldpid

Checking the activity cache is easy, log in and upload a doc and it should appear in your activity stream.

Clone this wiki locally