Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implement a lock-free work-stealing deque
This adds an implementation of the Chase-Lev work-stealing deque to libstd under std::rt::deque. I've been unable to break the implementation of the deque itself, and it's not super highly optimized just yet (everything uses a SeqCst memory ordering). The major snag in implementing the chase-lev deque is that the buffers used to store data internally cannot get deallocated back to the OS. In the meantime, a shared buffer pool (synchronized by a normal mutex) is used to deallocate/allocate buffers from. This is done in hope of not overcommitting too much memory. It is in theory possible to eventually free the buffers, but one must be very careful in doing so. I was unable to get some good numbers from src/test/bench tests (I don't think many of them are slamming the work queue that much), but I was able to get some good numbers from one of my own tests. In a recent rewrite of select::select(), I found that my implementation was incredibly slow due to contention on the shared work queue. Upon switching to the parallel deque, I saw the contention drop to 0 and the runtime go from 1.6s to 0.9s with the most amount of time spent in libuv awakening the schedulers (plus allocations). Closes #4877
- Loading branch information
a70f9d7
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
saw approval from pcwalton
at alexcrichton@a70f9d7
a70f9d7
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
merging alexcrichton/rust/issue-4877 = a70f9d7 into auto
a70f9d7
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
alexcrichton/rust/issue-4877 = a70f9d7 merged ok, testing candidate = dd1184e
a70f9d7
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all tests pass:
success: http://buildbot.rust-lang.org/builders/auto-mac-32-opt/builds/2702
success: http://buildbot.rust-lang.org/builders/auto-mac-32-nopt-c/builds/548
success: http://buildbot.rust-lang.org/builders/auto-mac-32-nopt-t/builds/548
success: http://buildbot.rust-lang.org/builders/auto-mac-64-opt/builds/2706
success: http://buildbot.rust-lang.org/builders/auto-mac-64-nopt-c/builds/1812
success: http://buildbot.rust-lang.org/builders/auto-mac-64-nopt-t/builds/1811
success: http://buildbot.rust-lang.org/builders/auto-linux-32-opt/builds/2719
success: http://buildbot.rust-lang.org/builders/auto-linux-32-nopt-c/builds/1811
success: http://buildbot.rust-lang.org/builders/auto-linux-32-nopt-t/builds/1812
success: http://buildbot.rust-lang.org/builders/auto-linux-64-opt/builds/2721
success: http://buildbot.rust-lang.org/builders/auto-linux-64-nopt-c/builds/1811
success: http://buildbot.rust-lang.org/builders/auto-linux-64-nopt-t/builds/1811
success: http://buildbot.rust-lang.org/builders/auto-linux-64-x-android/builds/1887
success: http://buildbot.rust-lang.org/builders/auto-win-32-opt/builds/2707
success: http://buildbot.rust-lang.org/builders/auto-win-32-nopt-c/builds/1811
success: http://buildbot.rust-lang.org/builders/auto-win-32-nopt-t/builds/1813
success: http://buildbot.rust-lang.org/builders/auto-bsd-64-opt/builds/2483
a70f9d7
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fast-forwarding master to auto = dd1184e