-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add an arena/pool to enable tensor buffer re-use during graph execution
Improve buffer re-use during graph execution by adding a pool from which operators can allocate output buffers, and into which buffers are added when their ref count drops to zero (ie. when they are no longer needed by subsequent graph execution steps). This significantly reduces how often execution needs to allocate "fresh" buffers from the system allocator and free them back. In this initial implementation, a reference to the pool is passed to all operators via `Operator::run`, but only a subset actually use the pool. This subset was chosen to benefit the YOLOv8 example. - Add `pool` argument to `Operator::run`, specifying a pool from which operators should allocate their outputs - Create a pool at the start of graph execution and release it at the end. Intermediate values that are no longer needed are added to the pool after each operator runs. - Report the number of allocations from the pools and the hit rate (how often the pool was able to satisfy allocations) as part of timing info. - Modify an initial subset of allocators to allocate from the pool, based on what helps the YOLOv8 example.
- Loading branch information
1 parent
a3e2dd9
commit 312c665
Showing
28 changed files
with
747 additions
and
274 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.