How is Redis used

[dsmiley]

I'm looking for any technical information on how exactly Redis is used. The high level JuiceFS description doesn't really say anything other than "metadata". If there is no such information, perhaps someone can point me at an interesting source file or two?

I'm wondering if any of the developers have considered ZooKeeper or Etcd as a substitute and wether that seems plausible or if they were deemed very unsuitable for some reason.

[davies]

@dsmiley We are sorry that there is no detailed docs for this part right now, please check the source code here: https://github.com/juicedata/juicefs/blob/main/pkg/meta/redis.go. The interface we are implementing is here: https://github.com/juicedata/juicefs/blob/main/pkg/meta/interface.go#L129

In order to implement the interface for metadata, there are lots of bookkeeping stuff, the data structures (hash map, list, and set, counter and so on) in Redis are very useful here. Also, the transaction and batch support in Redis is good to reduce latency for meta requests. ZooKeeper and Etcd don't have those powerful API to work with, they are also not intended to store hundreds of millions of small items, it will be easier to run into scalability issue later on.

BTW, we are also try to use Lua script to reduce the number of requests to Redis to improve latency, which is a great addon.

[dsmiley]

Thanks!