[compute/cker] Optimize BatchMatMul for x86

[tomdol]

This commit adds an optimized version of the BatchMatMul kernel. The optimization targets the x86 architecture, in all other cases the code is compiled with existing reference kernel.

The new kernel calls the optimized::Gemm function which uses Eigen internally.

Additionally to avoid code duplication a new BatchMatMulParams struct is introduced and reused in both reference and optimized kernels.

ONE-DCO-1.0-Signed-off-by: Tomasz Dolbniak [email protected]

[tomdol]

This PR is a follow-up of this draft #14238 and was submitted to (partially) solve this issue #12140

[glistening]

@tomdol Just for your information, our main target is arm, not x64. I guess you're already aware of it since you used (partially) solve. Also, for LLM, we will use GGML kernel which provides the quantized type kernel for lower than 8 bit.

[glistening]

In addition, this PR does not have test. How did you test this kernel?

[tomdol]

@glistening

How did you test this kernel?

There are existing tests for this kernel. I was thinking if I should add any but it seems that all use cases are covered.

[ RUN      ] GeneratedTests.batch_matmul_ex_dynamic_nnfw
[       OK ] GeneratedTests.batch_matmul_ex_dynamic_nnfw (1 ms)
[ RUN      ] GeneratedTests.batch_matmul_ex_float_simple
[       OK ] GeneratedTests.batch_matmul_ex_float_simple (1 ms)
[ RUN      ] GeneratedTests.batch_matmul_ex_float_adj_y
[       OK ] GeneratedTests.batch_matmul_ex_float_adj_y (1 ms)
[ RUN      ] GeneratedTests.batch_matmul_ex_float_adj_x
[       OK ] GeneratedTests.batch_matmul_ex_float_adj_x (1 ms)
[ RUN      ] GeneratedTests.batch_matmul_ex_float_batch2
[       OK ] GeneratedTests.batch_matmul_ex_float_batch2 (1 ms)
[ RUN      ] GeneratedTests.batch_matmul_ex_float_broadcast
[       OK ] GeneratedTests.batch_matmul_ex_float_broadcast (1 ms)
[ RUN      ] GeneratedTests.batch_matmul_ex_float_broadcast_adj_x
[       OK ] GeneratedTests.batch_matmul_ex_float_broadcast_adj_x (1 ms)
[ RUN      ] GeneratedTests.batch_matmul_ex_float_broadcast2_adj_xy
[       OK ] GeneratedTests.batch_matmul_ex_float_broadcast2_adj_xy (1 ms)

Regarding the GGML kernel - is someone already working on it? I was going to attempt to write an optimized version for ARM too in the next step, I would just like to know if I should proceed.

[glistening]

@tomdol Thank you for answer. Test was done via nnap tests.

Regarding the GGML kernel - is someone already working on it?

I checked out model. For our model, BatchMatMul f32 (both lhs, rhs) is necessary.

[glistening]

LGTM

[glistening]

@tomdol For arm optimized kernel, I am thinking of using ggml mul_mat, which is already in our repo. It supports multithread and use neon optimized code. What kernel are you thinking of?

[zetwhite]

LGTM 👍

[tomdol]

@tomdol For arm optimized kernel, I am thinking of using ggml mul_mat, which is already in our repo. It supports multithread and use neon optimized code. What kernel are you thinking of?

@glistening sorry about the delay in replying. I didn't think about any particular kernel yet except that there was a need for an ARM-targetting optimized version too. I was hoping to figure out more by discussing it in #12140

I would appreciate some guidelines and would like to offer to help with this part of the BatchMatMul optimization work. Unless of course someone is already taking care of it :)