merge master into cse region client (#939)

* client-go: add some key range info to error when PD returned no region (#862) Signed-off-by: Chao Wang <[email protected]> * *: refine non-global stale-read request retry logic (#863) Signed-off-by: crazycs520 <[email protected]> * Fix the issue that primary pessimistic lock may be left not cleared after GC (#866) * Fix the issue that primary pessimistic lock may be left not cleared after GC Signed-off-by: MyonKeminta <[email protected]> * Fix mysteriously shown up thing that makes compilation failed Signed-off-by: MyonKeminta <[email protected]> * Fix test effectiveness (forgot to set txn2 to pessimistic txn); add more strict checks Signed-off-by: MyonKeminta <[email protected]> * Address comments Signed-off-by: MyonKeminta <[email protected]> --------- Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> * add explicit request source type to label the external request like lightning/br (#868) Signed-off-by: nolouch <[email protected]> * use '%d' instead of '%q' for some int values in error message (#875) Signed-off-by: Chao Wang <[email protected]> * format key in error message in method `scanRegions` (#876) Signed-off-by: Chao Wang <[email protected]> * make cop request timeout a config paramter (#865) * update Signed-off-by: Spade A <[email protected]> * update Signed-off-by: Spade A <[email protected]> * update Signed-off-by: Spade A <[email protected]> * update Signed-off-by: Spade A <[email protected]> --------- Signed-off-by: Spade A <[email protected]> * region_cache: support check pending tiflash peer (#821) Signed-off-by: guo-shaoge <[email protected]> Co-authored-by: disksing <[email protected]> * *: add `SnapshotIterReverse` and make `iterReverse` supports `lowerBound` (#883) Signed-off-by: Jason Mo <[email protected]> * *: fix stale read ops metric (#878) (#889) Signed-off-by: crazycs520 <[email protected]> Co-authored-by: disksing <[email protected]> * add gc options (#828) Signed-off-by: weedge <[email protected]> Co-authored-by: disksing <[email protected]> * reload region cache when store is resolved from invalid status (#843) Signed-off-by: you06 <[email protected]> Co-authored-by: disksing <[email protected]> * ci: update setup-go action (#904) Signed-off-by: disksing <[email protected]> * fix unexpected slow query during GC running after stop 1 tikv-server (#899) (#909) * fix unexpected slow query during GC running after stop 1 tikv-server Signed-off-by: crazycs520 <[email protected]> * fix test Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * resource_manager: ignore ru metrics for background request (#872) Signed-off-by: husharp <[email protected]> Co-authored-by: disksing <[email protected]> * add more log for diagnose (#915) * add more log for diagnose Signed-off-by: crazycs520 <[email protected]> * fix Signed-off-by: crazycs520 <[email protected]> * add more log for diagnose Signed-off-by: crazycs520 <[email protected]> * add more log Signed-off-by: crazycs520 <[email protected]> * address comment Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * use context logger as much as possible (#908) * use context logger as much as possible Signed-off-by: crazycs520 <[email protected]> * refine Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * Resume max retry time check for stale read retry with leader option(#903) (#911) * Resume max retry time check for stale read retry with leader option Signed-off-by: cfzjywxk <[email protected]> * add cancel Signed-off-by: cfzjywxk <[email protected]> --------- Signed-off-by: cfzjywxk <[email protected]> * request_source: remove default label (#890) * request_source: remove default label Signed-off-by: nolouch <[email protected]> * add a function to set request source task type (#925) * add a function to set request source task type Signed-off-by: glorv <[email protected]> * ci: update go version (#936) * ci: update go version Signed-off-by: crazycs520 <[email protected]> * fix test Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * use tidb_kv_read_timeout as first kv request timeout (#919) * support tidb_kv_read_timeout as first round kv request timeout Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * update comment Signed-off-by: crazycs520 <[email protected]> * refine test Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * [pick] resource_control: bypass some internal urgent request (#938) * resource_control: bypass some internal urgent request (#884) Signed-off-by: nolouch <[email protected]> * resourcecontrol: fix nil pointer (#900) Signed-off-by: nolouch <[email protected]> --------- Signed-off-by: nolouch <[email protected]> --------- Signed-off-by: Chao Wang <[email protected]> Signed-off-by: crazycs520 <[email protected]> Signed-off-by: MyonKeminta <[email protected]> Signed-off-by: nolouch <[email protected]> Signed-off-by: Spade A <[email protected]> Signed-off-by: guo-shaoge <[email protected]> Signed-off-by: Jason Mo <[email protected]> Signed-off-by: weedge <[email protected]> Signed-off-by: you06 <[email protected]> Signed-off-by: disksing <[email protected]> Signed-off-by: husharp <[email protected]> Signed-off-by: cfzjywxk <[email protected]> Signed-off-by: glorv <[email protected]> Signed-off-by: iosmanthus <[email protected]> Co-authored-by: 王超 <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: ShuNing <[email protected]> Co-authored-by: Spade A <[email protected]> Co-authored-by: guo-shaoge <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Hangjie Mo <[email protected]> Co-authored-by: weedge <[email protected]> Co-authored-by: you06 <[email protected]> Co-authored-by: Hu# <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: glorv <[email protected]>
tikv · Aug 11, 2023 · b67add4 · b67add4
1 parent 2ad441f
commit b67add4
Show file tree

Hide file tree

Showing 40 changed files with 1,225 additions and 361 deletions.
diff --git a/.github/workflows/compatibility.yml b/.github/workflows/compatibility.yml
@@ -12,9 +12,9 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v4
       with:
-        go-version: "1.20.x"
+        go-version: "1.21.0"
 
     - name: Checkout Client-Go
       uses: actions/checkout@v2

diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
@@ -15,9 +15,9 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v4
         with:
-          go-version: 1.20.2
+          go-version: 1.21.0
 
       - name: Test
         run: go test ./...
@@ -30,9 +30,9 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v4
         with:
-          go-version: 1.20.2
+          go-version: 1.21.0
 
       - name: Test
         run: go test ./... -race
@@ -45,9 +45,9 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v4
         with:
-          go-version: 1.20.2
+          go-version: 1.21.0
 
       - name: Fetch PD
         uses: shrink/actions-docker-extract@v1
@@ -87,9 +87,9 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v4
         with:
-          go-version: 1.20.2
+          go-version: 1.21.0
 
       - name: Fetch PD
         uses: shrink/actions-docker-extract@v1

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -13,9 +13,9 @@ jobs:
     - uses: actions/checkout@v2
 
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v4
       with:
-        go-version: 1.20.2
+        go-version: 1.21.0
 
     - name: Test
       run: go test ./...
@@ -26,9 +26,9 @@ jobs:
     - uses: actions/checkout@v2
 
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v4
       with:
-        go-version: 1.20.2
+        go-version: 1.21.0
 
     - name: Test with race
       run: go test -race ./...
@@ -40,9 +40,9 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v4
         with:
-          go-version: 1.20.2
+          go-version: 1.21.0
 
       - name: Lint
         uses: golangci/golangci-lint-action@v3

diff --git a/config/client.go b/config/client.go
@@ -81,6 +81,9 @@ type TiKVClient struct {
 	// StoreLivenessTimeout is the timeout for store liveness check request.
 	StoreLivenessTimeout string           `toml:"store-liveness-timeout" json:"store-liveness-timeout"`
 	CoprCache            CoprocessorCache `toml:"copr-cache" json:"copr-cache"`
+	// CoprReqTimeout is the timeout for a single coprocessor request
+	// Note: this is a transitional modification, and it will be removed if it's dynamic configurable version is ready.
+	CoprReqTimeout time.Duration `toml:"copr-req-timeout" json:"copr-req-timeout"`
 	// TTLRefreshedTxnSize controls whether a transaction should update its TTL or not.
 	TTLRefreshedTxnSize      int64  `toml:"ttl-refreshed-txn-size" json:"ttl-refreshed-txn-size"`
 	ResolveLockLiteThreshold uint64 `toml:"resolve-lock-lite-threshold" json:"resolve-lock-lite-threshold"`
@@ -153,6 +156,7 @@ func DefaultTiKVClient() TiKVClient {
 			AdmissionMaxResultMB:  10,
 			AdmissionMinProcessMs: 5,
 		},
+		CoprReqTimeout: 60 * time.Second,
 
 		ResolveLockLiteThreshold: 16,
 	}

diff --git a/error/error.go b/error/error.go
@@ -246,7 +246,7 @@ type ErrAssertionFailed struct {
 	*kvrpcpb.AssertionFailed
 }
 
-// ErrLockOnlyIfExistsNoReturnValue is used when the flag `LockOnlyIfExists` of `LockCtx` is set, but `ReturnValues“ is not.
+// ErrLockOnlyIfExistsNoReturnValue is used when the flag `LockOnlyIfExists` of `LockCtx` is set, but `ReturnValues` is not.
 type ErrLockOnlyIfExistsNoReturnValue struct {
 	StartTS     uint64
 	ForUpdateTs uint64

diff --git a/examples/gcworker/gcworker.go b/examples/gcworker/gcworker.go
@@ -21,6 +21,7 @@ import (
 	"time"
 
 	"github.com/tikv/client-go/v2/oracle"
+	"github.com/tikv/client-go/v2/tikv"
 	"github.com/tikv/client-go/v2/txnkv"
 )
 
@@ -36,10 +37,14 @@ func main() {
 		panic(err)
 	}
 
-	sysSafepoint, err := client.GC(context.Background(), *safepoint)
+	sysSafepoint, err := client.GC(context.Background(), *safepoint, tikv.WithConcurrency(10))
 	if err != nil {
 		panic(err)
 	}
 	fmt.Printf("Finished GC, expect safepoint:%d(%+v),real safepoint:%d(+%v)\n", *safepoint, oracle.GetTimeFromTS(*safepoint), sysSafepoint, oracle.GetTimeFromTS(sysSafepoint))
 
+	err = client.Close()
+	if err != nil {
+		panic(err)
+	}
 }
diff --git a/go.mod b/go.mod
@@ -14,14 +14,14 @@ require (
 	github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c
 	github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c
 	github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989
-	github.com/pingcap/kvproto v0.0.0-20230713060620-89756bd21be1
+	github.com/pingcap/kvproto v0.0.0-20230720094213-a3b4a77b4333
 	github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.15.1
 	github.com/prometheus/client_model v0.4.0
 	github.com/stretchr/testify v1.8.2
 	github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a
-	github.com/tikv/pd/client v0.0.0-20230717031845-58eb48b840e7
+	github.com/tikv/pd/client v0.0.0-20230724080549-de985b8e0afc
 	github.com/twmb/murmur3 v1.1.3
 	go.etcd.io/etcd/api/v3 v3.5.2
 	go.etcd.io/etcd/client/v3 v3.5.2
@@ -51,6 +51,7 @@ require (
 	github.com/rogpeppe/go-internal v1.10.0 // indirect
 	go.etcd.io/etcd/client/pkg/v3 v3.5.2 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
+	golang.org/x/exp v0.0.0-20230711005742-c3f37128e5a4 // indirect
 	golang.org/x/net v0.10.0 // indirect
 	golang.org/x/sys v0.8.0 // indirect
 	golang.org/x/text v0.9.0 // indirect

diff --git a/go.sum b/go.sum
@@ -135,8 +135,8 @@ github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c h1:CgbKAHto5CQgW
 github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c/go.mod h1:4qGtCB0QK0wBzKtFEGDhxXnSnbQApw1gc9siScUl8ew=
 github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 h1:surzm05a8C9dN8dIUmo4Be2+pMRb6f55i+UIYrluu2E=
 github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989/go.mod h1:O17XtbryoCJhkKGbT62+L2OlrniwqiGLSqrmdHCMzZw=
-github.com/pingcap/kvproto v0.0.0-20230713060620-89756bd21be1 h1:sC3XRNNBQNjFJGRtSzJRvqi2aDLFOsQoCHItr9rbbY8=
-github.com/pingcap/kvproto v0.0.0-20230713060620-89756bd21be1/go.mod h1:r0q/CFcwvyeRhKtoqzmWMBebrtpIziQQ9vR+JKh1knc=
+github.com/pingcap/kvproto v0.0.0-20230720094213-a3b4a77b4333 h1:A6Wqgq0uMw51UiRAH27TVN0QlzVR5CVtV6fTQSAmvKM=
+github.com/pingcap/kvproto v0.0.0-20230720094213-a3b4a77b4333/go.mod h1:r0q/CFcwvyeRhKtoqzmWMBebrtpIziQQ9vR+JKh1knc=
 github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw=
 github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -192,8 +192,8 @@ github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ
 github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
 github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a h1:J/YdBZ46WKpXsxsW93SG+q0F8KI+yFrcIDT4c/RNoc4=
 github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a/go.mod h1:h4xBhSNtOeEosLJ4P7JyKXX7Cabg7AVkWCK5gV2vOrM=
-github.com/tikv/pd/client v0.0.0-20230717031845-58eb48b840e7 h1:N1m2YFtQAZEkk5ilmsMqauELh29TJQrX2JbcNhdnm7k=
-github.com/tikv/pd/client v0.0.0-20230717031845-58eb48b840e7/go.mod h1:YAB/lbBQtURR2A1tyqGQYEkBVCKHVqyZFefN7CPIZi4=
+github.com/tikv/pd/client v0.0.0-20230724080549-de985b8e0afc h1:IUg0j2nWoGYj3FQ3vA3vg97fPSpJEZQrDpgF8RkMLEU=
+github.com/tikv/pd/client v0.0.0-20230724080549-de985b8e0afc/go.mod h1:wfHRc4iYaqJiOQZCHcrF+r4hYnkGDaYWDfcicee//pc=
 github.com/twmb/murmur3 v1.1.3 h1:D83U0XYKcHRYwYIpBKf3Pks91Z0Byda/9SJ8B6EMRcA=
 github.com/twmb/murmur3 v1.1.3/go.mod h1:Qq/R7NUyOfr65zD+6Q5IHKsJLwP7exErjN6lyyq3OSQ=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -229,6 +229,8 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/exp v0.0.0-20230711005742-c3f37128e5a4 h1:QLureRX3moex6NVu/Lr4MGakp9FdA7sBHGBmvRW7NaM=
+golang.org/x/exp v0.0.0-20230711005742-c3f37128e5a4/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc=
 golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
 golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
 golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=

diff --git a/integration_tests/go.mod b/integration_tests/go.mod
@@ -6,13 +6,13 @@ require (
 	github.com/ninedraft/israce v0.0.3
 	github.com/pingcap/errors v0.11.5-0.20221009092201-b66cddb77c32
 	github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c
-	github.com/pingcap/kvproto v0.0.0-20230713060620-89756bd21be1
+	github.com/pingcap/kvproto v0.0.0-20230720094213-a3b4a77b4333
 	github.com/pingcap/tidb v1.1.0-beta.0.20230619015310-8b1006f1af04
 	github.com/pkg/errors v0.9.1
 	github.com/stretchr/testify v1.8.4
 	github.com/tidwall/gjson v1.14.1
-	github.com/tikv/client-go/v2 v2.0.8-0.20230615161845-b32f340d0609
-	github.com/tikv/pd/client v0.0.0-20230717031845-58eb48b840e7
+	github.com/tikv/client-go/v2 v2.0.8-0.20230714052714-85fc8f337565
+	github.com/tikv/pd/client v0.0.0-20230724080549-de985b8e0afc
 	go.uber.org/goleak v1.2.1
 )
 
@@ -66,13 +66,13 @@ require (
 	github.com/pingcap/log v1.1.1-0.20230317032135-a0d097d16e22 // indirect
 	github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21 // indirect
 	github.com/pingcap/tidb/parser v0.0.0-20230619015310-8b1006f1af04 // indirect
-	github.com/pingcap/tipb v0.0.0-20230602100112-acb7942db1ca // indirect
+	github.com/pingcap/tipb v0.0.0-20230607071926-bda24015c2d6 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/power-devops/perfstat v0.0.0-20221212215047-62379fc7944b // indirect
-	github.com/prometheus/client_golang v1.15.1 // indirect
+	github.com/prometheus/client_golang v1.16.0 // indirect
 	github.com/prometheus/client_model v0.4.0 // indirect
 	github.com/prometheus/common v0.44.0 // indirect
-	github.com/prometheus/procfs v0.10.1 // indirect
+	github.com/prometheus/procfs v0.11.0 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/rogpeppe/go-internal v1.10.0 // indirect
 	github.com/sasha-s/go-deadlock v0.2.0 // indirect
@@ -94,14 +94,14 @@ require (
 	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.24.0 // indirect
-	golang.org/x/crypto v0.9.0 // indirect
-	golang.org/x/exp v0.0.0-20230519143937-03e91628a987 // indirect
-	golang.org/x/net v0.10.0 // indirect
-	golang.org/x/sync v0.2.0 // indirect
-	golang.org/x/sys v0.8.0 // indirect
-	golang.org/x/text v0.9.0 // indirect
+	golang.org/x/crypto v0.11.0 // indirect
+	golang.org/x/exp v0.0.0-20230711005742-c3f37128e5a4 // indirect
+	golang.org/x/net v0.11.0 // indirect
+	golang.org/x/sync v0.3.0 // indirect
+	golang.org/x/sys v0.10.0 // indirect
+	golang.org/x/text v0.11.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
-	golang.org/x/tools v0.9.3 // indirect
+	golang.org/x/tools v0.10.0 // indirect
 	google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
 	google.golang.org/grpc v1.54.0 // indirect
 	google.golang.org/protobuf v1.30.0 // indirect
@@ -112,5 +112,7 @@ require (
 
 replace (
 	github.com/go-ldap/ldap/v3 => github.com/YangKeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117
+	github.com/pingcap/tidb => github.com/HuSharp/tidb v1.1.0-beta.0.20230726045237-a2b0085ad7c5
+	github.com/pingcap/tidb/parser => github.com/HuSharp/tidb/parser v0.0.0-20230726045237-a2b0085ad7c5
 	github.com/tikv/client-go/v2 => ../
 )