improvement to zstor behaviors when some meta backends down

[iwanbk]

Based on #123 discussion, i want to do full check & improvements (if needed) regarding the behavior when some of the meta backends down

• zstor monitor must be started (discussion : Zstor doesn't start when connection to any Zdb times out #123, PR: fix(meta): metastore availability improvements #128)
• when the meta backend up, zstor monitor automatically reconnect to that backend (fix(meta): refresh metastore if there is zdb up/down #129)
• zstor retrieve must work if there are >= 2 available meta backends (Zstor doesn't start when connection to any Zdb times out #123 (comment))
• zstor store must fail if one of the meta backend down (fix(meta): metastore availability improvements #128)
• reload zstor config using SIGUSR1 must update the meta backends properly fix(meta): refresh metastore if there is zdb up/down #129
• check ReplaceMetaStore behavior fix(meta): refresh metastore if there is zdb up/down #129
• document these behaviors on the README

[iwanbk]

Hi @LeeSmet
What is the plan for ReplaceBackends at

0-stor_v2/zstor/src/actors/backends.rs

Line 82 in b81264c

struct ReplaceBackends;

I can see the Handler, but it is not called at all

[LeeSmet]

That's a leftover from old functionality. A long time ago there was functionality to automatically replace backends if they were full/failed. Since the feature was not used and caused a lot of complexity for maintenance, it was removed. The handler is an artifact that wasn't removed properly. If you want it can be removed.

[iwanbk]

The handler is an artifact that wasn't removed properly. If you want it can be removed.

OK, i'll keep it for now as reference.

[iwanbk]

zstor retrieve must work if there are >= 2 available meta backends

tested on both master and #129

[iwanbk]

check ReplaceMetaStore behavior

also fixed on #129
By adding writeable flag in the message

[iwanbk]

reload zstor config using SIGUSR1 must update the meta backends properly

also fixed on #129.

By re-creating the metastore when there is change in the meta zdb backends