[Tracking Issue][ONNX] Quantized operator support in ONNX importer

[tmoreau89]

Summary

This issue will be used to track ONNX importer coverage progress of standard and non-standard quantized ops in TVM, and can be used to coordinate distributed efforts on improving quantized importer coverage work across organizations.

Status

To this day (Aug 24th 2021) we'd like to account for both standard ONNX quantized ops and non-standard quantized contrib ops introduced by ONNXRT, as shown in the table below:

Shortlist of ops that are emitted by ONNXRT static quantization (higher priority), based on https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/registry.py:

Operator	Standard	Opset	Status	Owner	PR
ConvInteger	Y	10	Supported	@jwfromm	#8456
DequantizeLinear	Y	13, 10	Supported	@mbrookhart	#7802
MatMulInteger	Y	10	WIP	@WenheLI
QLinearConv	Y	10	Supported	@huochaitiantang	#8007
QLinearMatMul	Y	10	Supported	@cconvey	#8952
QuantizeLinear	Y	13, 10	Supported	@mbrookhart	#7802
com.microsoft.QAttention	N	n/a	TODO
com.microsoft.QLinearAdd	N	n/a	Supported	@mbrookhart	#8305
com.microsoft.QLinearAveragePool	N	n/a	Supported	@quic-sanirudh	#9017
com.microsoft.QLinearConcat	N	n/a	Supported	@anwang2009	#8907
com.microsoft.QLinearGlobalAveragePool	N	n/a	Supported	@quic-sanirudh	#9017
com.microsoft.QLinearLeakyRelu	N	n/a	Supported	@gayatripk1	#9063
com.microsoft.QLinearMul	N	n/a	Supported	@anwang2009	#8773
com.microsoft.QLinearSigmoid	N	n/a	Supported	@arangasa	#9028

Ops for supporting ORT dynamic quantization:

Operator	Standard	Opset	Status	Owner	PR
DynamicQuantizeLinear	Y	11	Supported	@mbrookhart	#7802
com.microsoft.DynamicQuantizeLSTM	N	n/a	TODO
com.microsoft.DynamicQuantizeMatMul	N	n/a	WIP	@quic-sanirudh

Other integer ops that might be relevant:

Operator	Standard	Opset	Status	Owner	PR
com.microsoft.MatMulInteger16	Y	n/a	Supported	@abhikran-quic	#9186
com.microsoft.MatMulIntegerToFloat	N	n/a	WIP	@onkar-sima-ai
com.microsoft.MulInteger	N	n/a	WIP	@tasmia-rahman
com.microsoft.ReduceSumInteger	N	n/a	WIP	@FranckQC

Other ops:

Operator	Standard	Opset	Status	Owner	PR
com.microsoft.QGemm	N	n/a	WIP	@rasagna-quic
com.microsoft.QLinearReduceMean	N	n/a	WIP	@avquicinc

Coordination

Improving the importers can be a good onboarding task for engineers that would like to get a more in-depth exposure to the TVM stack. The goal is that if folks want to claim an operator they can feel reassured that their work won't be deprecated by work that is in flight.

We provide reference PRs that can serve as a template to adding a quantized standard op: #7802 by @mbrookhart. As well as non-standard op from the Microsoft ContribOperators set in ONNXRT: #8773 by @anwang2009.

Please comment in this issue if you'd like to add to Relay ONNX importer coverage so I can update the table.

cc @KJlaccHoeUM9l @ehsanmok

[tmoreau89]

Adding @cconvey to the table who is working on QLinearMatMul

[WenheLI]

@tmoreau89 - Can I try to implement MatMulInteger this OP?

[tmoreau89]

@WenheLI absolutely, I'll mark you down

[tmoreau89]

Also CC- @kparzysz-quic

[kparzysz-quic]

Could you add
@avquicinc for com.microsoft.QLinearReduceMean
@quic-sanirudh for com.microsoft.QAveragePool, com.microsoft.QGlobalAveragePool

Edit: we also have volunteers for com.microsoft.QGlobalAveragePool, com.microsoft.QGemm. You can add my name there, since folks need to create github accounts.

[anwang2009]

@anwang2009 for com.microsoft.QLinearConcat

[tmoreau89]

Yes, yes, and yes! Thanks folks!

[kparzysz-quic]

Please add @tasmia-rahman for com.microsoft.MulInteger.

[kparzysz-quic]

Please add @FranckQC for com.microsoft.ReduceSumInteger

[FranckQC]

Hi everyone.
Could you please add me for com.microsoft.ReduceSumInteger if it's still available?
I might not be able to start working on it right now, but I'd definitely like to work on it as soon as I'm done with something else!

[tmoreau89]

Done and done, thanks all!

[arangasa]

Hi All, May I sign up for QLinearSigmoid? Thank you.

[abhikran-quic]

Hi @tmoreau89, I would like to sign up for com.microsoft.MatMulInteger16. Could you please assign it on my name? Thanks.

[rasagna-quic]

Hi @tmoreau89, could you please add me for com.microsoft.QGemm?

Could you add
@avquicinc for com.microsoft.QLinearReduceMean
@quic-sanirudh for com.microsoft.QAveragePool, com.microsoft.QGlobalAveragePool

Edit: we also have volunteers for com.microsoft.QGlobalAveragePool, com.microsoft.QGemm. You can add my name there, since folks need to create github accounts.

[arangasa]

Hi All, May I sign up for QLinearSigmoid? Thank you.

Hi @tmoreau89, May I sign up for com.microsoft.QLinearLeakyRelu as well? (This is for a team member who needs to create github accounts).

[gayatripk1]

Hi All, May I sign up for QLinearSigmoid? Thank you.

Hi @tmoreau89, May I sign up for com.microsoft.QLinearLeakyRelu as well? (This is for a team member who needs to create github accounts).

Could you please assign the op QLinearLeakyRelu on my name

[kparzysz-quic]

Hi @tmoreau89, could you please add me for com.microsoft.QGemm?

Yes, please reassign com.microsoft.QGemm to @rasagna-quic.

[arangasa]

Hi All, May I sign up for QLinearSigmoid? Thank you.

Hi @tmoreau89, May I sign up for com.microsoft.QLinearLeakyRelu as well? (This is for a team member who needs to create github accounts).

Could you please assign the op QLinearLeakyRelu on my name

Yes, @tmoreau89, could you please assign com.microsoft.QLinearLeakyRelu to
gayatripk1? Thank you.

[mbrookhart]

I think I have it updated with the assignments as volunteered in the comments. Thanks, everyone!

[jcoplin-quic]

I'll grab com.microsoft.QuantizeLinear @tmoreau89 @kparzysz-quic

[tmoreau89]

Thanks @jcoplin-quic I've added you

[cconvey]

QLinearMatMul now has a PR: #8952

[tmoreau89]

Thanks @cconvey - I've updated the table!

[avquicinc]

I am referring this file to see what all ops would get quantized by ONNXRT. https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/registry.py

@mbrookhart : As per your suggestion I have added this link, if you meant this resource to be added in the ticket please add it there as well.
Thanks!

[tmoreau89]

@avquicinc excellent point and many thanks for digging into the registry. I agree that the focus here is to shortlist these ops to the set defined in the registry. To use the example of QLinearReduceMean that gets emitted as QLinearDequantzie->ReduceMean->QLinearQuantize, there is no need for TVM to support the QLinearReduceMean for now. I will therefore as a next task emphasize the short list of ops that we should support to enable any ONNXRT quantized models.

[arangasa]

Hi @tmoreau89, I've placed a PR #9028 for QLinearSigmoid. Just FYI.

[tmoreau89]

Thanks @arangasa - I've added it to the table!

[tmoreau89]

Also happy to share that com.microsoft.QLinearGlobalAveragePool and com.microsoft.QLinearAveragePool are supported, thanks @quic-sanirudh for the support

[quic-sanirudh]

@tmoreau89 Thanks for your help in reviewing the AveragePool op support quickly. Shall I pick up the com.microsoft.DynamicQuantizeMatMul op next to work on, if no one else has started working on that. If that's an urgently needed op, I can pick something else as well

[tmoreau89]

@quic-sanirudh please go ahead and pick com.microsoft.DynamicQuantizeMatMul given that no one else has claimed, it. I can mark you down for it.

[tmoreau89]

And thanks to @arangasa , #9028 has been merged! com.microsoft.QLinearSigmoid is now supported.

[abhikran-quic]

Hi @tmoreau89 , I've raised #9186 for com.microsoft.MatMulInteger16. Just FYI.

[tmoreau89]

Thanks @abhikran-quic , added

[tmoreau89]

QLinearMatMul support is now merged! Thank you @cconvey for completing this task

[tmoreau89]

@WenheLI and @gayatripk1, is support for MatMulInteger and com.microsoft.QLinearLeakyRelu still on your plates? Just checking in periodically.

[gayatripk1]

@WenheLI and @gayatripk1, is support for MatMulInteger and com.microsoft.QLinearLeakyRelu still on your plates? Just checking in periodically.

com.microsoft.QLinearLeakyRelu is already merged.

[tmoreau89]

Thank you I somehow failed to catch it, I'm adding #9063 to the table, thank you for your contributions.

[onkar-sima-ai]

Hi @tmoreau89 I would like to work on com.microsoft.MatMulIntegerToFloat , could you please assign it to my name?

[tmoreau89]

Thank you @onkar-sima-ai I entered you in the table.

[jshuhnow]

Hi @tmoreau89 , I'd like to work on com.microsoft.QAttention. Could you assign it to me, if possible?