Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert remaining operators to use tensor pool #109

Closed
robertknight opened this issue Apr 23, 2024 · 1 comment
Closed

Convert remaining operators to use tensor pool #109

robertknight opened this issue Apr 23, 2024 · 1 comment
Labels
performance Issues that affect model inference or loading performance

Comments

@robertknight
Copy link
Owner

#108 introduced a buffer pool to enable re-use of tensor buffers that are no longer needed, as outputs for future non-mutating operations. Only a subset of operators currently allocate from the pool. To get the full benefit the majority of ops need to be converted. Non-converted ops can be found by searching for operators which ignore their pool argument. This is indicated by having an underscore-prefixed _pool argument in their Operator::run implementation.

@robertknight
Copy link
Owner Author

This is now done except for a few operators where the output size is data-dependent (NonZero, Range) and the RNN operators. For the RNN ops, I need to complete #95 first to avoid conflicts.

robertknight added a commit that referenced this issue Apr 27, 2024
Even though I'm planning to significantly rework the internals of these
operators soon [1], converting them to use the pool now allows
#109 to be resolved. Also using the
pool now only requires smaller code changes which won't lead to major merge
conflicts.

[1] #85
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Issues that affect model inference or loading performance
Projects
None yet
Development

No branches or pull requests

1 participant