-
Notifications
You must be signed in to change notification settings - Fork 859
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
osc/rdma: adjustment on btl selection logic
This patch makes the following adjustments to the btl selection logic. First, when selecting a primary btl, this patch requires the primary btl to support remote completion. Second, when selecting alternate btls, this patch allowes any btl to be used as an alternate btl. Prior to this patch, only a list of pre-defined btls can be used as alternate btls. Finally, when a btl is used as an atlernate btl, this patch disables its native atomic and RDMA support, and always use active message RDMA/atomics with it. The reason for these changes are: First, these changes ensured the selected btls of osc/rdma always support remote completion (because active message RDMA/atomics supports remote completion when an btl's native RDMA is disabled). Remote completion is essential for several key components of osc/rdma: the usage of cpu atomics to update peer's state, the usage of local leader to update peer's state and its fence implementation. Therefore the assurance of btl's support of remote completion greatly simplified osc/rdma's implementation. Second, these changes eliminated the need to save and exchange more than 1 memory reigstration, because active message RDMA/atomic does not require explicit memory registration. Third, these changes ensured the correctness of atomic operations. When multiple alternate btls are used, atomicity cannot be guarenteed accross each btl's native implementation of atomics. Finally, these changes allowed more btls to be used as alternate btls, especially btl/self. Signed-off-by: Wei Zhang <[email protected]>
- Loading branch information
Showing
1 changed file
with
51 additions
and
46 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters