ExecutionProvider API refactor - move allocator from EP level to SessionState level and indexed by OrtDevice #15833
Conversation
onnxruntime/core/optimizer/transpose_optimizer/optimizer_utils.h
Outdated
Show resolved
Hide resolved
jslhcl
requested review from
adithyaselv,
jywu-msft and
pranavsharma
and removed request for
adithyaselv
There was a problem hiding this comment.
Can you modify this error message to say that you can call the new API for other EPs?
|
/azp run Android CI Pipeline |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@jslhcl FYI, it looks like this PR broke the CANN EP build. see https://github.com/Ascend/onnxruntime/actions/runs/5322372781/jobs/9638730400#step:3:3026 /root/Git.d/onnxruntime/onnxruntime/core/providers/cann/cann_execution_provider.cc:1450:7: error: ‘cann_device’ was not declared in this scope |
+@FFFrog ,@zhangsibo1129 FYI, this change broke the CANN EP build. can you advise/help @jslhcl fix it since he doesn't have an environment to build/test? |
|
I fixed the CANN EP build for this #15833 (comment), @jywu-msft would you take a look at this PR. Thanks a lot! |
thank you, @zhangsibo1129! |
### Description Remove AllocatorManager class ### Motivation and Context After the refactor PR #15833 is in, AllocatorManager class is not referenced anymore.
Fix memory leak issue which comes from TRT EP's allocator object not
being released upon destruction.
Following is the log from valgrind:
```
==1911860== 100,272 (56 direct, 100,216 indirect) bytes in 1 blocks are definitely lost in loss record 1,751 of 1,832
==1911860== at 0x483CFA3: operator new(unsigned long) (vg_replace_malloc.c:472)
==1911860== by 0x315DC2: std::_MakeUniq<onnxruntime::OrtAllocatorImplWrappingIAllocator>::__single_object std::make_unique<onnxruntime::OrtAllocatorImplWrappingIAllocator, std::shared_ptr<onnxruntime::IAllocator> >(std::shared_ptr<onnxruntime::IAllocator>&&) (unique_ptr.h:857)
==1911860== by 0x30EE7B: OrtApis::KernelContext_GetAllocator(OrtKernelContext const*, OrtMemoryInfo const*, OrtAllocator**) (custom_ops.cc:121)
==1911860== by 0x660D115: onnxruntime::TensorrtExecutionProvider::Compile(std::vector<onnxruntime::IExecutionProvider::FusedNodeAndGraph, std::allocator<onnxruntime::IExecutionProvider::FusedNodeAndGraph> > const&, std::vector<onnxruntime::NodeComputeInfo, std::allocator<onnxruntime::NodeComputeInfo> >&)::{lambda(void*, OrtApi const*, OrtKernelContext*)#3}::operator()(void*, OrtApi const*, OrtKernelContext*) const (tensorrt_execution_provider.cc:2223)
```
This issue happens after this [EP allocator
refactor](#15833)
) Modify CANN EP `CANNExecutionProvider::CreatePreferredAllocators`, `CANNExecutionProvider::CreateCannAllocator` to align with the EP API refactor and fix CANN CI for #15833 (comment) in this [PR](#15833)
### Description clean unused parameter in ORT_UNUSED_PARAMETER ### Motivation and Context clean unused parameters in ORT_UNUSED_PARAMETER which are introduced from #15833
…t allocation in ORT API (#17030) This addresses a DML performance regression from the following PR resulting in allocations not being rounded and pooled in the DML execution provider. #15833 This also fixes a pre-existing limitation that allocations during session initialization (primarily large weights and persistent resources) only bypassed rounding and pooling while using the Winml API. The allocator now also respects a caller's rounding mode parameter when provided.
…t allocation in ORT API (#17030) This addresses a DML performance regression from the following PR resulting in allocations not being rounded and pooled in the DML execution provider. #15833 This also fixes a pre-existing limitation that allocations during session initialization (primarily large weights and persistent resources) only bypassed rounding and pooling while using the Winml API. The allocator now also respects a caller's rounding mode parameter when provided.
…t allocation in ORT API (microsoft#17030) This addresses a DML performance regression from the following PR resulting in allocations not being rounded and pooled in the DML execution provider. microsoft#15833 This also fixes a pre-existing limitation that allocations during session initialization (primarily large weights and persistent resources) only bypassed rounding and pooling while using the Winml API. The allocator now also respects a caller's rounding mode parameter when provided.
Description
This PR is to refactor ExecutionProvider API for memory management, which is to move allocators from EP level to SessionState level and indexed by OrtDevice
Motivation and Context
This PR is to refactor ExecutionProvider API for memory management, which is to move allocators from EP level to SessionState level and indexed by OrtDevice. By this change, EP level will shift the burden of maintaining allocators, which will be user friendly for EP developers