Performance regression introduced by pr/36963

[tiancaiamao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

See http://www.zenlife.tk:18081/alloc

2. What did you expect to see? (Required)

No regression

3. What did you see instead (Required)

Manual test, before and after the change:

cd session;
go test -run XXX -bench BenchmarkTableScan -benchmem

BenchmarkTableScan-16    	    6127	    195089 ns/op	   60804 B/op	     792 allocs/op
BenchmarkTableScan-16    	    5616	    196387 ns/op	   62716 B/op	     824 allocs/op

4. What is your TiDB version? (Required)

master

[tiancaiamao]

Ref #25754
Ref #36963