Irregular numbers on IPEX 2.3.0 and 2.3.100 for Bert-Large and Distil-bert

[ramaseshan1]

Describe the issue

On Bert-large, & distil-bert, I am getting very low throughput numbers (BS 96), after i use ipex.optimize(model) followed by torch.compile(model, backend="ipex"). I see similar poor numbers on ipex.optimize followed by torchscript path as well.

In PT 2.2.x and PT 2.1.x, the numbers are fine.

Can you let me know if this is a known bug?

[jgong5]

Could you please share more details? What the perf numbers did you get? What script did you use?

[ramaseshan1]

benchmark.txt

Observing 5 qps for BS 60 in IPEX 2.3.100 while with IPEX 2.2.x, I am observing 41.78 qps.

I am noticed multiple models behaving in the same way for IPEX 2.3.x

[ramaseshan1]

Hey @jgong5,
Can you tell me if this is a known issue in IPEX 2.3.x? If yes, when can I expect the next patch or minor?

[jgong5]

Hey @jgong5, Can you tell me if this is a known issue in IPEX 2.3.x? If yes, when can I expect the next patch or minor?

@ramaseshan1 Thanks for the detailed info. I guess @huiyan2021 is trying to repro the issue and see where the problem is. @huiyan2021 is there any finding?

[ramaseshan1]

Hey @huiyan2021,
Can you please confirm if you were able to reproduce the issue on your end?

[huiyan2021]

Sincerely sorry that I missed this issue. Let me try to reproduce and come back asap.

[huiyan2021]

Hi @ramaseshan1 , I tried your benchmark code on a SPR 6438Y+ with the latest ipex 2.4.0

got below throughput

So could you try the latest ipex 2.4.0 and update your result here?