-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert remaining operators to use tensor pool #109
Labels
performance
Issues that affect model inference or loading performance
Comments
robertknight
added
the
performance
Issues that affect model inference or loading performance
label
Apr 23, 2024
This was referenced Apr 26, 2024
This is now done except for a few operators where the output size is data-dependent ( |
This was referenced Apr 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
#108 introduced a buffer pool to enable re-use of tensor buffers that are no longer needed, as outputs for future non-mutating operations. Only a subset of operators currently allocate from the pool. To get the full benefit the majority of ops need to be converted. Non-converted ops can be found by searching for operators which ignore their pool argument. This is indicated by having an underscore-prefixed
_pool
argument in theirOperator::run
implementation.The text was updated successfully, but these errors were encountered: