ddl: don't reuse the chunk until underlying memory is not referenced

[tangenta]

What problem does this PR solve?

Issue Number: close #39308

Problem Summary:

The issue could be reproduced easily when the data set is large enough.

diff --git a/ddl/index.go b/ddl/index.go
index c3c3ffd37c..f5ed601d2b 100644
--- a/ddl/index.go
+++ b/ddl/index.go
@@ -1548,6 +1548,13 @@ func (w *addIndexWorker) BackfillDataInTxn(handleRange reorgBackfillTask) (taskC
                        } else { // The lightning environment is ready.
                                vars := w.sessCtx.GetSessionVars()
                                sCtx, writeBufs := vars.StmtCtx, vars.GetWriteStmtBufs()
+                               logutil.BgLogger().Info("write", zap.String("handle", idxRecord.handle.String()),
+                                       zap.String("idxVal", idxRecord.vals[0].GetString()),
+                                       zap.String("addr", fmt.Sprintf("%p", idxRecords)),
+                                       zap.String("idxAddr", fmt.Sprintf("%p", idxRecord)),
+                                       zap.String("idxDtAddr", fmt.Sprintf("%p", idxRecord.vals)),
+                                       zap.String("idxDt0Addr", fmt.Sprintf("%p", &idxRecord.vals[0])),
+                                       zap.String("d.b", fmt.Sprintf("%p", idxRecord.vals[0].GetBytes())))
                                key, distinct, err := w.index.GenIndexKey(sCtx, idxRecord.vals, idxRecord.handle, writeBufs.IndexKeyBuf)
@@ -291,10 +295,17 @@ func (c *copContext) fetchTableScanResult(ctx context.Context, result distsql.Se
                        return nil, false, errors.Trace(err)
                }
                rsData := tables.TryGetHandleRestoredDataWrapper(c.tblInfo, hdDt, nil, c.idxInfo)
-               buf = append(buf, &indexRecord{handle: handle, key: nil, vals: idxDt, rsData: rsData, skip: false})
+               idxRec := &indexRecord{handle: handle, key: nil, vals: idxDt, rsData: rsData, skip: false}
+               buf = append(buf, idxRec)
+               logutil.BgLogger().Info("rows", zap.String("val", row.ToString(c.fieldTps)),
+                       zap.String("handle", handle.String()),
+                       zap.String("addr", fmt.Sprintf("%p", buf)),
+                       zap.String("idxAddr", fmt.Sprintf("%p", idxRec)),
+                       zap.String("idxDtAddr", fmt.Sprintf("%p", idxRec.vals)),
+                       zap.String("idxDt0Addr", fmt.Sprintf("%p", &idxRec.vals[0])),
+                       zap.String("d.b", fmt.Sprintf("%p", idxRec.vals[0].GetBytes())))
        }

[2022/11/25 01:27:51.107 +08:00] [INFO] [index_cop.go:300] [rows] [val="32644160646-97332178411-20202217618-32930568818-43901782951-91030404202-17535639422-27973945537-69415721036-64437459094, 2561"] [handle=2561] [addr=0x1400182e000] [idxAddr=0x14003662c60] [idxDtAddr=0x14003b562d0] [idxDt0Addr=0x14003b562d0] [d.b=0x14001e60000]

[2022/11/25 01:27:51.107 +08:00] [INFO] [index.go:1551] [write] [handle=1] [idxVal=32644160646-97332178411-20202217618-32930568818-43901782951-91030404202-17535639422-27973945537-69415721036-64437459094] [addr=0x14000e57000] [idxAddr=0x1400069cde0] [idxDtAddr=0x140015381b0] [idxDt0Addr=0x140015381b0] [d.b=0x14001e60000]

The actual data underlying types.Datum is referenced by two different goroutines, because when we initialize the datum from the chunk, we use GetString() for varchar columns.

// GetString gets string value.
func (d *Datum) GetString() string {
	return string(hack.String(d.b))
}

The string is not copied here!

What is changed and how it works?

Similar to idxBufPool, this PR adds a chunk sync.Pool to manage & reuse the memory as soon as possible.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
  It is too hard to write a suitable unit test, because it needs a large amount of data and the concurrency. I test it with sysbench, no data inconsistency is found anymore.
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• Benjamin2037
• zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[Benjamin2037]

LGTM

[zimulala]

LGTM

[tangenta]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 009058d

[hawkingrei]

LGTM

FYI, We have the reuse-chunk feature by #38607. we maybe able to use it.

[hawkingrei]

/run-all-tests

[hawkingrei]

/run-all-tests

[Benjamin2037]

/run-all-tests

[tangenta]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 1399840

[hawkingrei]

@tangenta PTAL

        	Error Trace:	/home/jenkins/.tidb/tmp/63a9840cd0739f2c243bb46478607469/sandbox/linux-sandbox/6874/execroot/__main__/bazel-out/k8-fastbuild/bin/ddl/ddl_test_/ddl_test.runfiles/__main__/ddl/index_cop_test.go:48
        	            				/home/jenkins/.tidb/tmp/63a9840cd0739f2c243bb46478607469/sandbox/linux-sandbox/6874/execroot/__main__/bazel-out/k8-fastbuild/bin/ddl/ddl_test_/ddl_test.runfiles/__main__/ddl/index_cop_test.go:66
        	Error:      	Should be true
        	Test:       	TestAddIndexFetchRowsFromCoprocessor

[tangenta]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 2642946

[sre-bot]

TiDB MergeCI notify

✅ Well Done! New fixed [1] after this pr merged.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/mybatis-test	🔴 failed 1, success 0, total 1	11 min	Existing failure
idc-jenkins-ci-tidb/integration-ddl-test	✅ all 6 tests passed	4 min 48 sec	Fixed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	32 min	Existing passed
idc-jenkins-ci/integration-cdc-test	🟢 all 39 tests passed	21 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 11 tests passed	18 min	Existing passed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 17 tests passed	17 min	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	6 min 36 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	4 min 47 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	2 min 45 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed