Enable static consistent hash ring

[JiamingMai]

By default, we build a dynamic consistent hash with the live worker list that comes from master or ETCD. Sometimes we want to build a static consistent hash ring to make sure we won't write data to other worker node when a worker node is offline temporarily (especially when other worker nodes are running out of disk space).

This PR provides allows us to build a static consistent hash ring by setting alluxio.user.dynamic.consistent.hash.ring.enabled=false. In this case, client will read from UFS if the worker where the specified file locate is down.

[lucyge2022]

By default, we build a dynamic consistent hash with the live worker list that comes from master or ETCD. Sometimes we want to build a static consistent hash ring to make sure we won't write data to other worker node when a worker node is offline temporarily (especially when other worker nodes are running out of disk space).

This PR provides allows us to build a static consistent hash ring by setting alluxio.user.dynamic.consistent.hash.ring.enabled=false. In this case, client will read from UFS if the worker where the specified file locate is down.

actually if its etcd it has already been using a static consistent hash ring.

[JiamingMai]

By default, we build a dynamic consistent hash with the live worker list that comes from master or ETCD. Sometimes we want to build a static consistent hash ring to make sure we won't write data to other worker node when a worker node is offline temporarily (especially when other worker nodes are running out of disk space).
This PR provides allows us to build a static consistent hash ring by setting alluxio.user.dynamic.consistent.hash.ring.enabled=false. In this case, client will read from UFS if the worker where the specified file locate is down.

actually if its etcd it has already been using a static consistent hash ring.

@lucyge2022 This PR makes us use dynamic consistent hash ring by default (no matter using ETCD or master). In this PR, I change the logic to get getLiveMembers instead of getAllMembers from ETCD by default. This allows users to choose the way they want.

[yyongycy]

@jja725 wondering if the hash ring change impacts the logic of distributedMv related?

[jja725]

@yyongycy this doesn't affect cp/mv, but probably affect distributed load, have to take a closer look.

[JiamingMai]

@yyongycy this doesn't affect cp/mv, but probably affect distributed load, have to take a closer look.

Actually, we do want to affect distributed load. That is why we want to have this change. If there is a worker that is offline temporarily, static consistent hash ring allows user to avoid writing data to other worker nodes when executing load command and reading a file cached on the offline worker.

[jja725]

@yyongycy this doesn't affect cp/mv, but probably affect distributed load, have to take a closer look.

Actually, we do want to affect distributed load. That is why we want to have this change. If there is a worker that is offline temporarily, static consistent hash ring allows user to avoid writing data to other worker nodes when executing load command and reading a file cached on the offline worker.

Then we can update DefaultWorkerProvider.getWorkerInfos

[yyongycy]

Overall LGTM, looks like not a big changes. Master based worker registration would eventually be etcd based worker registration.

just need some data to see "time of the consistent hash ring building given it has 400K (100 physical nodes) vnodes"
Rest optimization can be done later.

[JiamingMai]

@yyongycy this doesn't affect cp/mv, but probably affect distributed load, have to take a closer look.

Actually, we do want to affect distributed load. That is why we want to have this change. If there is a worker that is offline temporarily, static consistent hash ring allows user to avoid writing data to other worker nodes when executing load command and reading a file cached on the offline worker.

Then we can update DefaultWorkerProvider.getWorkerInfos

I updated DefaultWorkerProvider.getWorkerInfos. Please take a look when you have time. @jja725

[yyongycy]

btw, please do test it.

[yyongycy]

lgtm based on test done

[apc999]

my understanding is that "master" mode is only used for backward purpose, will not be used for membership in the future.
correct?

[JiamingMai]

my understanding is that "master" mode is only used for backward purpose, will not be used for membership in the future. correct?

Yes, it is only used for backward purpose. But some features still depends on master, for example load command requires master which has Scheduler for distributed load.

[jja725]

LGTM, But I'm still a little fuzzy what's the exact behavior/exception when we lost a worker which is still running the task or send a request to lost worker. Do you think we can add some comment on those behaviors?

[JiamingMai]

LGTM, But I'm still a little fuzzy what's the exact behavior/exception when we lost a worker which is still running the task or send a request to lost worker. Do you think we can add some comment on those behaviors?

The request will throw an exception after timeout. If we want to add a comment for this case, we need to identify the exact type of exception.

[JiamingMai]

alluxio-bot, merge this please