From dd7f7533f979f80449ba4fd0b8568e3514590b6c Mon Sep 17 00:00:00 2001 From: lhy1024 Date: Fri, 19 May 2023 10:04:02 +0800 Subject: [PATCH] mcs: support multi keyspace group in tso service (#88) * client: refine serviceModeKeeper code (#6201) ref tikv/pd#5895 Some code refinements for `serviceModeKeeper`. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * *: use revision for watch test (#6205) ref tikv/pd#6071 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * *: remove unnecessary rand init (#6207) close tikv/pd#6134 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * Refactor TSO forward/dispatcher to be shared by both PD and TSO (#6175) ref tikv/pd#5895 Add general tso forward/dispatcher for independent pd(tso)/tso services and cross cluster forwarding. Signed-off-by: Bin Shi Signed-off-by: lhy1024 * Add basic multi-keyspace-group management (#6214) ref tikv/pd#5895 Support basic functions of multi-keyspace-group management Signed-off-by: Bin Shi Signed-off-by: lhy1024 * *: support keyspace group RESTful API (#6229) ref tikv/pd#6231 Signed-off-by: Ryan Leung Signed-off-by: lhy1024 * mcs: add more tso tests (#6184) ref tikv/pd#5836 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * client: fix compatibility problem of pd client (#6244) close tikv/pd#6243 Signed-off-by: Ryan Leung * *: unify the key prefix (#6248) ref tikv/pd#5836 Signed-off-by: Ryan Leung Signed-off-by: lhy1024 * *: remove cluster dependency from keyspace (#6249) ref tikv/pd#6231 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * *: make code clear by rename `isServing` to `isRunning` (#6258) ref tikv/pd#4399 Signed-off-by: lhy1024 Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * cgroup: fix the path problem due to special container name (#6267) close tikv/pd#6266 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * server: fix watch keyspace revision (#6251) ref tikv/pd#5895 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * tso, server: refine the TSO allocator manager parameters (#6269) ref tikv/pd#5895 - Refine the TSO allocator manager parameters. - Always run `tsoAllocatorLoop` to advance the Global TSO. Signed-off-by: JmPotato Signed-off-by: lhy1024 * tso: unify the TSO ServiceConfig and ConfigProvider interfaces (#6272) ref tikv/pd#5895 Unify the TSO `ServiceConfig` and `ConfigProvider` interfaces. Signed-off-by: JmPotato * Load initial assignment and dynamically watch/apply keyspace groups' membership/distribution change (#6247) ref tikv/pd#6232 Load initial keyspace group assignment. Dynamically watch/apply keyspace groups' membership/distribution change. Signed-off-by: Bin Shi * *: define user kind for keyspace group (#6241) ref tikv/pd#6231 Signed-off-by: Ryan Leung Signed-off-by: lhy1024 * Add more failure tests when tso service loading initial keyspace groups assignment (#6280) ref tikv/pd#6232 Add more failure tests when tso service loading initial keyspace groups assignment Signed-off-by: Bin Shi * Apply multi-keyspace-group membership to tso service and handle inconsistency issue (#6282) ref tikv/pd#6232 Apply multi-keyspace-group membership to tso service and handle inconsistency issue. 1. Add KeyspaceLookupTable to endpoint.KeyspaceGroup type KeyspaceGroup struct { ... // KeyspaceLookupTable is for fast lookup if a given keyspace belongs to this keyspace group. // It's not persisted and will be built when loading from storage. KeyspaceLookupTable map[uint32]struct{} `json:"-"` } 2. After loading keyspace groups, the Keyspace Group Manager builds KeyspaceLookupTable for every keyspace groups. 3. When Keyspace Group Manager handles tso requests, it uses the keyspaceLookupTable to check if the required keypsace still belongs to the required keyspace group. If not, returns the current keyspace group id in the tso response header. Signed-off-by: Bin Shi * *: auto assign keyspace group (#6268) close tikv/pd#6231 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * keyspace, api: support the keyspace group split (#6293) ref tikv/pd#6232 Support the keyspace group split and add related tests. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * Improve lock mechanism in tso.KeyspaceGroupManager (#6305) ref tikv/pd#6232 Use the RWMutex instead of individual atomic types to better protect the state of the keyspace group manager Signed-off-by: Bin Shi * keyspace: add split-from field for endpoint.KeyspaceGroup (#6309) ref tikv/pd#6232 Add `split-from` field for `endpoint.KeyspaceGroup`. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * Add read lock at one place for protection and better structure (#6310) ref tikv/pd#6232, ref tikv/pd#6305 follow-up #6305 Add read lock at one place for protection and better structure Signed-off-by: Bin Shi Co-authored-by: Ti Chi Robot * tso: optimize function signatures to reduce parameter passing (#6315) ref tikv/pd#6232 Optimize function signatures to reduce parameter passing. Signed-off-by: JmPotato * *: bootstrap keyspace group when server is in API mode (#6308) ref tikv/pd#6231 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * keyspace: avoid keyspace being updated during the split (#6316) ref tikv/pd#6232 Prevent keyspace from being updated during the split. Signed-off-by: JmPotato * *: fix `TestConcurrentlyReset` (#6318) close tikv/pd#6275 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot * bootstrap default keyspace group in the tso service (#6306) ref tikv/pd#6232 Changes: 1. Introduce the initialization logic of the default keyspace group. - If the default keyspace group isn't configured in the etcd, every tso node/pod should initialize it and join the election for the primary of this group. - If the default keyspace group is configured in the etcd, the tso nodes/pods which are assigned with this group will initialize it and join the election for the primary of this group. 2. Introduce the keyspace group membership restriction -- default keyspace always belongs to default keyspace group. Signed-off-by: Bin Shi * *: change the log level (#6324) ref tikv/pd#6232 Signed-off-by: Ryan Leung * *: fix the missing log panic (#6325) close tikv/pd#6257 Signed-off-by: Ryan Leung Signed-off-by: lhy1024 * mcs: fix watch primary address revision and update cache when meets not leader (#6279) ref tikv/pd#5895 Signed-off-by: lhy1024 Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * tso, member: support TSO split based on keyspace group split (#6313) ref tikv/pd#6232 Support TSO split based on keyspace group split. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * mcs: support metrics HTTP interface for tso/resource manager server (#6329) ref tikv/pd#5895 Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * tso: put finishSplitKeyspaceGroup into the critical section (#6331) ref tikv/pd#6232 Put `finishSplitKeyspaceGroup` into the critical section. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * *: make `TestServerRegister` stable (#6337) close tikv/pd#6334 Signed-off-by: Ryan Leung * tests: divide all tests into the CI chunks including submodule tests (#6198) ref tikv/pd#6181, ref tikv/pd#6183 Divide all tests into the CI chunks including submodule tests. Signed-off-by: JmPotato Signed-off-by: lhy1024 * tests: introduce TSO TestCluster in the test (#6333) ref tikv/pd#6232 Introduce TSO `TestCluster` in the test. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * mcs: add balancer for keyspace group (#6274) close tikv/pd#6233 Signed-off-by: lhy1024 Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * Fixed bugs in tso service registry watching loop. (#6346) ref tikv/pd#6343 Fixed the following two bugs: 1. When re-watch a range, to continue from what left by the last watch, the revision is wresp.Header.Revision + 1 instead of wresp.Header.Revision, where wresp.Header.Revision is the revision indicated in the response of the last watch. Because of this bug, it was processing the same event endless as you can see from the log below. 2. In tso service watch loop in /Users/binshi/code/pingcap/my-pd/pkg/keyspace/tso_keyspace_group.go, If this is delete event, the json.Unmarshal(event.Kv.Value, s) will fail with the error "unexpected end of JSON input", so there is no way to get s.serviceAddr from the result of json.Unmarshal. Signed-off-by: Bin Shi * mcs: fix double compression of prom handler (#6339) ref prometheus/client_golang#622, ref tikv/pd#5895 Signed-off-by: Ryan Leung * tests, tso: add more TSO split tests (#6338) ref tikv/pd#6232 Add more TSO split tests. Signed-off-by: JmPotato Co-authored-by: Ti Chi Robot * keyspace, tso: fix next revision to watch after watch/Get/RangeScan (#6353) ref tikv/pd#6232 The next revision to watch should always be Header.Revision + 1 where header is response header of watch/Get/RangeScan Signed-off-by: Bin Shi * mcs, tests: use TSO cluster to do the failover test (#6356) ref tikv/pd#5895 Use TSO cluster to do the failover test. Signed-off-by: JmPotato * fix startWatchLoop leak (#6352) Signed-off-by: Ryan Leung Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * mcs: update client when meet transport is closing (#6341) * mcs: update client when meet transport is closing Signed-off-by: lhy1024 * address comments Signed-off-by: lhy1024 * add retry Signed-off-by: lhy1024 --------- Signed-off-by: lhy1024 Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 * add bootstrap test (#6347) Signed-off-by: Ryan Leung Co-authored-by: Ti Chi Robot Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 * Fix flaky TestLoadKeyspaceGroupsAssignment test (#6365) Reduce the count of keyspace groups from 4096 to 512 in TestKeyspaceGroupManagerTestSuite/TestLoadKeyspaceGroupsAssignment to avoid timeout when test running slow. Signed-off-by: Bin Shi * mcs, tso: fix ts fallback caused by multi-primary of the same keyspace group (#6362) * Change participant election-prifix from listen-addr to advertise-listen-addr to gurantee uniqueness. Signed-off-by: Bin Shi * Add TestPariticipantStartWithAdvertiseListenAddr Signed-off-by: Bin Shi * Add comments to fix go fmt errors Signed-off-by: Bin Shi --------- Signed-off-by: Bin Shi Co-authored-by: Ryan Leung Signed-off-by: lhy1024 * fix log output (#6364) Signed-off-by: Ryan Leung Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * mcs: fix duplicate start of RaftCluster. (#6358) * Using double-checked locking to avoid duplicate start of RaftCluster. Signed-off-by: Bin Shi * Handle feedback Signed-off-by: Bin Shi * improve locking Signed-off-by: Bin Shi * handle feedback Signed-off-by: Bin Shi --------- Signed-off-by: Bin Shi Co-authored-by: Ryan Leung * Add retry mechanism for updating keyspace group (#6372) Signed-off-by: JmPotato * mcs: add set handler for balancer and alloc node for default keyspace group (#6342) ref tikv/pd#6233 Signed-off-by: lhy1024 Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * mcs, tso: fix Nil pointer deference when (*AllocatorManager).GetMember (#6383) close tikv/pd#6381 If the desired keyspace group fall back to the default keyspace group and the AM isn't initialized, return not served error. Signed-off-by: Bin Shi * mcs, tso: support multi-keyspace-group and its service discovery in E2E path (#6321) ref tikv/pd#6232 Support multi-keyspace-group in PD(TSO) client Signed-off-by: Bin Shi Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 * client: add `NewClientWithKeyspaceName` for client (#6380) ref tikv/pd#5895 Signed-off-by: Ryan Leung * keyspace, tso: check the replica count before the split (#6382) ref tikv/pd#6233 Check the replica count before the split. Signed-off-by: JmPotato Co-authored-by: lhy1024 Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * tso: fix bugs to make split test case to pass (#6389) ref tikv/pd#6232 fix bugs to make split test case to pass Signed-off-by: Bin Shi Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * keyspace: patrol keyspace assignment before the first split (#6388) ref tikv/pd#6232 Patrol the keyspace assignment before the first split to make sure every keyspace has its group assignment. Signed-off-by: JmPotato Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * tests: fix Flaky TestMicroserviceTSOServer/TestConcurrentlyReset (#6396) close tikv/pd#6385 Get a copy of now then call base.add, because now is shared by all goroutines and now.add() will add to itself which isn't atomic and multi-goroutine safe. Signed-off-by: Bin Shi * keyspace, slice: improve code efficiency in membership ops (#6392) ref tikv/pd#6231 Improve code efficiency in membership ops Signed-off-by: Bin Shi * tests: enable TestTSOKeyspaceGroupSplitClient (#6398) ref tikv/pd#6232 Enable `TestTSOKeyspaceGroupSplitClient`. Signed-off-by: JmPotato Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * tests: add more tests for multiple keyspace groups (#6395) ref tikv/pd#5895 Add CheckMultiKeyspacesTSO() and WaitForMultiKeyspacesTSOAvailable in test utility. Add TestTSOKeyspaceGroupManager/TestKeyspacesServedByNonDefaultKeyspaceGroup. Cover TestGetTS, TestGetTSAsync, TestUpdateAfterResetTSO in TestMicroserviceTSOClient for multiple keyspace groups. Signed-off-by: Bin Shi * tests: fix failpoint disable (#6401) ref tikv/pd#4399 Signed-off-by: lhy1024 Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 * client: retry load keyspace meta when creating a new client (#6402) ref tikv/pd#5895 Signed-off-by: Ryan Leung Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * Fix test issue in TestRandomResignLeader. (#6410) close tikv/pd#6404 We need to make sure the selected keyspaces are from different keyspace groups, otherwise multiple goroutines below could try to resign the primary of the same keyspace group and cause race condition. Signed-off-by: Bin Shi * keyspace, api2: fix the keyspace assignment patrol consistency (#6397) ref tikv/pd#6232 Fix the keyspace assignment patrol consistency. Signed-off-by: JmPotato Co-authored-by: Ryan Leung Signed-off-by: lhy1024 * election, tso: fix data race in lease.go (#6379) close tikv/pd#6378 fix data race in lease.go Signed-off-by: Bin Shi Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * mcs: fix forward test with pd mode client (#6290) ref tikv/pd#5895, ref tikv/pd#6279, close tikv/pd#6289 Signed-off-by: lhy1024 * keyspace: patrol the keyspace assignment in batch (#6411) ref tikv/pd#6232 Patrol the keyspace assignment in batch. Signed-off-by: JmPotato * etcdutil: add watch loop (#6390) close tikv/pd#6391 Signed-off-by: lhy1024 * mcs, tso: add API interface to obtain the TSO keyspace group member info (#6373) ref tikv/pd#6232 Add API interface to obtain the TSO keyspace group member info. Signed-off-by: JmPotato Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * pkg: move operator_check out from test_util #6162 Signed-off-by: lhy1024 * keysapce: wait region split when creating keyspace (#6414) ref tikv/pd#6231 Signed-off-by: zeminzhou Signed-off-by: lhy1024 Co-authored-by: zzm Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 * mcs: use getClusterInfo to check whether api service is ready (#6422) ref tikv/pd#5836 Signed-off-by: lhy1024 Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 * fix data race by replace clone (#6242) close tikv/pd#6230 Signed-off-by: bufferflies <1045931706@qq.com> Co-authored-by: Ti Chi Robot Signed-off-by: lhy1024 * fix test and git mod tidy Signed-off-by: lhy1024 * revert makefile Signed-off-by: lhy1024 * fix ctx in watch loop Signed-off-by: lhy1024 * delete pd-tests.yaml Signed-off-by: lhy1024 * pd-ctl, tests: add the keyspace group commands (#6423) ref tikv/pd#6232 Add the keyspace group commands to show and split keyspace groups. Signed-off-by: JmPotato Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * Handle compatibility issue in GetClusterInfo RPC (#6434) ref tikv/pd#5895, close tikv/pd#6448 Handle the compatibility issue in the GetClusterInfo RPC Signed-off-by: Bin Shi Signed-off-by: lhy1024 * Provide GetMinTS API to solve the compatibility issue brought by multi-timeline tso (#6421) ref tikv/pd#6142 1. Import kvproto change to introduce GetMinTS rpc in the TSO service. 6. Add server side implementation for GetMinTS rpc. 7. Add client side implementation for GetMinTS rpc. 8. Add unit test Signed-off-by: Bin Shi Signed-off-by: lhy1024 * tso: use less interval when waiting api service (#6451) close tikv/pd#6449 Signed-off-by: lhy1024 * etcdutil: fix ctx in watch loop (#6445) close tikv/pd#6439 Signed-off-by: lhy1024 Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 * Fix "non-default keyspace groups use the same timestamp path by mistake" (#6457) close tikv/pd#6453, close tikv/pd#6465 The tso servers are loading keyspace groups asynchronously. Make sure all keyspace groups are available for serving tso requests from corresponding keyspaces by querying IsKeyspaceServing(keyspaceID, the Desired KeyspaceGroupID). if use default keyspace group id in the query, it will always return true as the keyspace will be served by default keyspace group before the keyspace groups are loaded. Signed-off-by: Bin Shi Signed-off-by: lhy1024 * TSO microservice discovery fallback path shouldn't call FindGroupByKeyspaceID (#6473) close tikv/pd#6472 TSO microservice discovery fallback path shouldn't call FindGroupByKeyspaceID Signed-off-by: Bin Shi * Revert "cgroup: fix the path problem due to special container name (#6267)" This reverts commit 0c4cf7f947799e5c45d6e37448475b921044bdde. * *: rm debug file (#6458) ref tikv/pd#4399 Signed-off-by: lhy1024 Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * Revert "*: remove unnecessary rand init (#6207)" This reverts commit 7383ded7581c417a3866da271eb2ec0a27b5a6c8. * mcs, tso: handle null keyspace (#6476) ref tikv/pd#5895 For API V1 and legacy path (NewClientWithContext w/o keyspace id/name), using Null Keypsace ID (uint32max) instead of default keyspace id and make sure it can be served by the default keyspace group's timeline. Modifying test accordingly. Signed-off-by: Bin Shi * mcs, tso: print TSO service discovery fallback log just once (#6478) ref tikv/pd#5895 Print TSO service discovery fallback log just once Signed-off-by: Bin Shi * client: return error if the keyspace meta cannot be found (#6479) ref tikv/pd#6142 Signed-off-by: Ryan Leung Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * client: support use API context to create client (#6482) ref tikv/pd#6142 Signed-off-by: Ryan Leung --------- Signed-off-by: Bin Shi Signed-off-by: lhy1024 Signed-off-by: Ryan Leung Signed-off-by: JmPotato Co-authored-by: JmPotato Co-authored-by: Ti Chi Robot Co-authored-by: Ryan Leung Co-authored-by: Bin Shi <39923490+binshi-bing@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: zzm Co-authored-by: buffer <1045931706@qq.com> --- client/client.go | 355 +++++-- client/client_test.go | 19 +- client/errs/errno.go | 40 +- client/go.mod | 8 +- client/go.sum | 16 +- client/grpcutil/grpcutil.go | 9 +- client/pd_service_discovery.go | 198 ++-- client/tso_batch_controller.go | 8 +- client/tso_client.go | 141 ++- client/tso_dispatcher.go | 68 +- client/tso_service_discovery.go | 449 +++++++-- client/tso_stream.go | 31 +- client/tsoutil/tsoutil.go | 46 + errors.toml | 47 +- go.mod | 33 +- go.sum | 105 +- pd.code-workspace | 9 +- pkg/balancer/balancer.go | 61 ++ pkg/balancer/balancer_test.go | 102 ++ pkg/balancer/round_robin.go | 87 ++ pkg/core/storelimit/limit_test.go | 3 +- pkg/core/storelimit/store_limit.go | 8 + pkg/election/leadership.go | 2 +- pkg/election/lease.go | 16 +- pkg/errs/errno.go | 39 +- {server => pkg}/keyspace/keyspace.go | 311 ++++-- {server => pkg}/keyspace/keyspace_test.go | 120 ++- pkg/keyspace/tso_keyspace_group.go | 712 ++++++++++++++ pkg/keyspace/tso_keyspace_group_test.go | 322 +++++++ {server => pkg}/keyspace/util.go | 124 ++- {server => pkg}/keyspace/util_test.go | 5 +- pkg/mcs/discovery/discover.go | 4 +- pkg/mcs/discovery/discover_test.go | 16 +- pkg/mcs/discovery/key_path.go | 20 +- pkg/mcs/discovery/register.go | 4 +- pkg/mcs/discovery/register_test.go | 8 +- pkg/mcs/resource_manager/server/server.go | 12 +- pkg/mcs/tso/server/apis/v1/api.go | 70 +- pkg/mcs/tso/server/config.go | 33 + pkg/mcs/tso/server/config_test.go | 99 ++ pkg/mcs/tso/server/grpc_service.go | 301 +++--- pkg/mcs/tso/server/handler.go | 16 +- pkg/mcs/tso/server/server.go | 359 +++---- pkg/mcs/tso/server/testutil.go | 28 - pkg/mcs/utils/constant.go | 30 +- pkg/member/election_leader.go | 90 ++ pkg/member/member.go | 60 +- pkg/member/participant.go | 88 +- pkg/slice/slice.go | 12 + pkg/slice/slice_test.go | 51 +- pkg/storage/endpoint/key_path.go | 39 +- pkg/storage/endpoint/key_path_test.go | 48 + pkg/storage/endpoint/keyspace.go | 6 +- pkg/storage/endpoint/tso_keyspace_group.go | 186 ++++ pkg/storage/keyspace_test.go | 30 +- pkg/storage/storage.go | 1 + pkg/tso/admin.go | 4 +- pkg/tso/allocator_manager.go | 315 ++++-- pkg/tso/config.go | 51 + pkg/tso/global_allocator.go | 225 ++++- pkg/tso/keyspace_group_manager.go | 827 ++++++++++++++++ pkg/tso/keyspace_group_manager_test.go | 899 ++++++++++++++++++ pkg/tso/local_allocator.go | 31 +- pkg/tso/testutil.go | 112 +++ pkg/tso/tso.go | 38 +- .../apiutil/multiservicesapi/middleware.go | 9 +- pkg/utils/etcdutil/etcdutil.go | 285 ++++++ pkg/utils/etcdutil/etcdutil_test.go | 339 ++++++- pkg/utils/grpcutil/grpcutil_test.go | 6 +- .../operator_check.go | 2 +- pkg/utils/testutil/testutil.go | 3 + pkg/utils/tsoutil/tso_dispatcher.go | 243 +++++ pkg/utils/tsoutil/tso_proto_factory.go | 107 +++ pkg/utils/tsoutil/tso_request.go | 165 ++++ pkg/utils/tsoutil/{tso.go => tsoutil.go} | 0 scripts/ci-subtask.sh | 63 +- server/api/region.go | 2 +- server/api/region_label_test.go | 4 +- server/apiv2/handlers/keyspace.go | 62 +- server/apiv2/handlers/tso_keyspace_group.go | 351 +++++++ server/apiv2/middlewares/bootstrap_checker.go | 5 +- server/apiv2/middlewares/redirector.go | 2 +- server/apiv2/router.go | 3 +- server/cluster/cluster.go | 44 +- server/cluster/cluster_test.go | 2 +- server/cluster/coordinator_test.go | 9 +- server/config/config.go | 27 +- server/grpc_service.go | 327 +------ server/handler.go | 2 +- server/keyspace_service.go | 98 +- server/schedule/checker/merge_checker_test.go | 25 +- .../schedule/checker/replica_checker_test.go | 78 +- server/schedule/checker/rule_checker_test.go | 10 +- server/schedule/operator_controller.go | 9 +- server/schedule/operator_controller_test.go | 3 +- server/schedulers/balance_test.go | 62 +- server/schedulers/evict_leader_test.go | 8 +- server/schedulers/evict_slow_store_test.go | 6 +- server/schedulers/evict_slow_trend_test.go | 6 +- server/schedulers/hot_region_test.go | 96 +- server/schedulers/hot_region_v2_test.go | 38 +- server/schedulers/scheduler_test.go | 18 +- .../transfer_witness_leader_test.go | 8 +- server/server.go | 256 ++--- server/server_test.go | 1 + tests/cluster.go | 16 +- tests/{ => integrations}/client/Makefile | 28 +- .../client/cert-expired/ca-config.json | 0 .../client/cert-expired/ca-csr.json | 0 .../client/cert-expired/ca-key.pem | 0 .../client/cert-expired/ca.csr | 0 .../client/cert-expired/ca.pem | 0 .../client/cert-expired/client-key.pem | 0 .../client/cert-expired/client.csr | 0 .../client/cert-expired/client.pem | 0 .../client/cert-expired/gencerts.sh | 0 .../client/cert-expired/pd-server-key.pem | 0 .../client/cert-expired/pd-server.csr | 0 .../client/cert-expired/pd-server.pem | 0 .../client/cert/ca-config.json | 0 .../client/cert/ca-csr.json | 0 .../{ => integrations}/client/cert/ca-key.pem | 0 tests/{ => integrations}/client/cert/ca.csr | 0 tests/{ => integrations}/client/cert/ca.pem | 0 .../client/cert/client-key.pem | 0 .../{ => integrations}/client/cert/client.csr | 0 .../{ => integrations}/client/cert/client.pem | 0 .../client/cert/gencerts.sh | 0 .../client/cert/pd-server-key.pem | 0 .../client/cert/pd-server.csr | 0 .../client/cert/pd-server.pem | 0 .../{ => integrations}/client/client_test.go | 9 +- .../client/client_tls_test.go | 0 .../client/global_config_test.go | 0 tests/{ => integrations}/client/go.mod | 38 +- tests/{ => integrations}/client/go.sum | 92 +- .../client/keyspace_test.go | 23 +- tests/{ => integrations}/mcs/Makefile | 25 +- tests/integrations/mcs/cluster.go | 158 +++ .../mcs/discovery/register_test.go | 17 +- tests/{ => integrations}/mcs/go.mod | 46 +- tests/{ => integrations}/mcs/go.sum | 113 ++- .../mcs/keyspace/tso_keyspace_group_test.go | 360 +++++++ .../resource_manager/resource_manager_test.go | 0 .../mcs/resource_manager/server_test.go | 23 +- tests/integrations/mcs/testutil.go | 218 +++++ tests/integrations/mcs/tso/api_test.go | 106 +++ .../mcs/tso/keyspace_group_manager_test.go | 427 +++++++++ .../{ => integrations}/mcs/tso/server_test.go | 414 +++++--- tests/{ => integrations}/tso/Makefile | 25 +- tests/integrations/tso/client_test.go | 480 ++++++++++ .../tso/consistency_test.go | 2 +- tests/{ => integrations}/tso/go.mod | 32 +- tests/{ => integrations}/tso/go.sum | 110 ++- tests/{ => integrations}/tso/server_test.go | 11 +- tests/{ => integrations}/tso/testutil.go | 2 +- tests/mcs/testutil.go | 114 --- tests/pdctl/keyspace/keyspace_group_test.go | 84 ++ tests/server/apiv2/handlers/keyspace_test.go | 120 +-- tests/server/apiv2/handlers/testutil.go | 233 +++++ .../apiv2/handlers/tso_keyspace_group_test.go | 174 ++++ tests/server/cluster/cluster_test.go | 6 +- tests/server/keyspace/keyspace_test.go | 7 +- tests/server/tso/common_test.go | 2 +- tests/server/tso/tso_test.go | 2 +- tests/tso/client_test.go | 170 ---- tools/pd-ctl/pdctl/command/cluster_command.go | 6 +- .../pdctl/command/keyspace_group_command.go | 83 ++ tools/pd-ctl/pdctl/command/tso_command.go | 2 +- tools/pd-ctl/pdctl/ctl.go | 1 + tools/pd-tso-bench/go.sum | 21 +- 171 files changed, 11439 insertions(+), 2752 deletions(-) create mode 100644 client/tsoutil/tsoutil.go create mode 100644 pkg/balancer/balancer.go create mode 100644 pkg/balancer/balancer_test.go create mode 100644 pkg/balancer/round_robin.go rename {server => pkg}/keyspace/keyspace.go (61%) rename {server => pkg}/keyspace/keyspace_test.go (71%) create mode 100644 pkg/keyspace/tso_keyspace_group.go create mode 100644 pkg/keyspace/tso_keyspace_group_test.go rename {server => pkg}/keyspace/util.go (63%) rename {server => pkg}/keyspace/util_test.go (94%) create mode 100644 pkg/mcs/tso/server/config_test.go create mode 100644 pkg/member/election_leader.go create mode 100644 pkg/storage/endpoint/tso_keyspace_group.go create mode 100644 pkg/tso/config.go create mode 100644 pkg/tso/keyspace_group_manager.go create mode 100644 pkg/tso/keyspace_group_manager_test.go create mode 100644 pkg/tso/testutil.go rename pkg/utils/{testutil => operatorutil}/operator_check.go (99%) create mode 100644 pkg/utils/tsoutil/tso_dispatcher.go create mode 100644 pkg/utils/tsoutil/tso_proto_factory.go create mode 100644 pkg/utils/tsoutil/tso_request.go rename pkg/utils/tsoutil/{tso.go => tsoutil.go} (100%) create mode 100644 server/apiv2/handlers/tso_keyspace_group.go rename tests/{ => integrations}/client/Makefile (68%) rename tests/{ => integrations}/client/cert-expired/ca-config.json (100%) rename tests/{ => integrations}/client/cert-expired/ca-csr.json (100%) rename tests/{ => integrations}/client/cert-expired/ca-key.pem (100%) rename tests/{ => integrations}/client/cert-expired/ca.csr (100%) rename tests/{ => integrations}/client/cert-expired/ca.pem (100%) rename tests/{ => integrations}/client/cert-expired/client-key.pem (100%) rename tests/{ => integrations}/client/cert-expired/client.csr (100%) rename tests/{ => integrations}/client/cert-expired/client.pem (100%) rename tests/{ => integrations}/client/cert-expired/gencerts.sh (100%) rename tests/{ => integrations}/client/cert-expired/pd-server-key.pem (100%) rename tests/{ => integrations}/client/cert-expired/pd-server.csr (100%) rename tests/{ => integrations}/client/cert-expired/pd-server.pem (100%) rename tests/{ => integrations}/client/cert/ca-config.json (100%) rename tests/{ => integrations}/client/cert/ca-csr.json (100%) rename tests/{ => integrations}/client/cert/ca-key.pem (100%) rename tests/{ => integrations}/client/cert/ca.csr (100%) rename tests/{ => integrations}/client/cert/ca.pem (100%) rename tests/{ => integrations}/client/cert/client-key.pem (100%) rename tests/{ => integrations}/client/cert/client.csr (100%) rename tests/{ => integrations}/client/cert/client.pem (100%) rename tests/{ => integrations}/client/cert/gencerts.sh (100%) rename tests/{ => integrations}/client/cert/pd-server-key.pem (100%) rename tests/{ => integrations}/client/cert/pd-server.csr (100%) rename tests/{ => integrations}/client/cert/pd-server.pem (100%) rename tests/{ => integrations}/client/client_test.go (99%) rename tests/{ => integrations}/client/client_tls_test.go (100%) rename tests/{ => integrations}/client/global_config_test.go (100%) rename tests/{ => integrations}/client/go.mod (92%) rename tests/{ => integrations}/client/go.sum (94%) rename tests/{ => integrations}/client/keyspace_test.go (91%) rename tests/{ => integrations}/mcs/Makefile (64%) create mode 100644 tests/integrations/mcs/cluster.go rename tests/{ => integrations}/mcs/discovery/register_test.go (89%) rename tests/{ => integrations}/mcs/go.mod (89%) rename tests/{ => integrations}/mcs/go.sum (92%) create mode 100644 tests/integrations/mcs/keyspace/tso_keyspace_group_test.go rename tests/{ => integrations}/mcs/resource_manager/resource_manager_test.go (100%) rename tests/{ => integrations}/mcs/resource_manager/server_test.go (86%) create mode 100644 tests/integrations/mcs/testutil.go create mode 100644 tests/integrations/mcs/tso/api_test.go create mode 100644 tests/integrations/mcs/tso/keyspace_group_manager_test.go rename tests/{ => integrations}/mcs/tso/server_test.go (51%) rename tests/{ => integrations}/tso/Makefile (60%) create mode 100644 tests/integrations/tso/client_test.go rename tests/{ => integrations}/tso/consistency_test.go (99%) rename tests/{ => integrations}/tso/go.mod (89%) rename tests/{ => integrations}/tso/go.sum (92%) rename tests/{ => integrations}/tso/server_test.go (92%) rename tests/{ => integrations}/tso/testutil.go (97%) delete mode 100644 tests/mcs/testutil.go create mode 100644 tests/pdctl/keyspace/keyspace_group_test.go create mode 100644 tests/server/apiv2/handlers/testutil.go create mode 100644 tests/server/apiv2/handlers/tso_keyspace_group_test.go delete mode 100644 tests/tso/client_test.go create mode 100644 tools/pd-ctl/pdctl/command/keyspace_group_command.go diff --git a/client/client.go b/client/client.go index b30f9534caf..aca6ee851af 100644 --- a/client/client.go +++ b/client/client.go @@ -43,8 +43,16 @@ import ( const ( // defaultKeyspaceID is the default key space id. // Valid keyspace id range is [0, 0xFFFFFF](uint24max, or 16777215) - // ​0 is reserved for default keyspace with the name "DEFAULT", It's initialized when PD bootstrap and reserved for users who haven't been assigned keyspace. + // ​0 is reserved for default keyspace with the name "DEFAULT", It's initialized + // when PD bootstrap and reserved for users who haven't been assigned keyspace. defaultKeyspaceID = uint32(0) + maxKeyspaceID = uint32(0xFFFFFF) + // nullKeyspaceID is used for api v1 or legacy path where is keyspace agnostic. + nullKeyspaceID = uint32(0xFFFFFFFF) + // defaultKeySpaceGroupID is the default key space group id. + // We also reserved 0 for the keyspace group for the same purpose. + defaultKeySpaceGroupID = uint32(0) + defaultKeyspaceName = "DEFAULT" ) // Region contains information of a region's meta and its peers. @@ -205,6 +213,8 @@ var ( errClosing = errors.New("[pd] closing") // errTSOLength is returned when the number of response timestamps is inconsistent with request. errTSOLength = errors.New("[pd] tso length in rpc response is incorrect") + // errInvalidRespHeader is returned when the response doesn't contain service mode info unexpectedly. + errNoServiceModeReturned = errors.New("[pd] no service mode returned") ) // ClientOption configures client. @@ -240,19 +250,38 @@ func WithMaxErrorRetry(count int) ClientOption { var _ Client = (*client)(nil) -// serviceModeKeeper is for service mode switching +// serviceModeKeeper is for service mode switching. type serviceModeKeeper struct { - svcModeMutex sync.RWMutex + // RMutex here is for the future usage that there might be multiple goroutines + // triggering service mode switching concurrently. + sync.RWMutex serviceMode pdpb.ServiceMode - tsoClient atomic.Value + tsoClient *tsoClient tsoSvcDiscovery ServiceDiscovery } +func (k *serviceModeKeeper) close() { + k.Lock() + defer k.Unlock() + switch k.serviceMode { + case pdpb.ServiceMode_API_SVC_MODE: + k.tsoSvcDiscovery.Close() + fallthrough + case pdpb.ServiceMode_PD_SVC_MODE: + if k.tsoClient != nil { + k.tsoClient.Close() + } + case pdpb.ServiceMode_UNKNOWN_SVC_MODE: + } +} + type client struct { keyspaceID uint32 svrUrls []string pdSvcDiscovery ServiceDiscovery tokenDispatcher *tokenDispatcher + + // For service mode switching. serviceModeKeeper // For internal usage. @@ -278,19 +307,38 @@ type SecurityOption struct { } // NewClient creates a PD client. -func NewClient(svrAddrs []string, security SecurityOption, opts ...ClientOption) (Client, error) { +func NewClient( + svrAddrs []string, security SecurityOption, opts ...ClientOption, +) (Client, error) { return NewClientWithContext(context.Background(), svrAddrs, security, opts...) } // NewClientWithContext creates a PD client with context. This API uses the default keyspace id 0. -func NewClientWithContext(ctx context.Context, svrAddrs []string, security SecurityOption, opts ...ClientOption) (Client, error) { - return NewClientWithKeyspace(ctx, defaultKeyspaceID, svrAddrs, security, opts...) +func NewClientWithContext( + ctx context.Context, svrAddrs []string, + security SecurityOption, opts ...ClientOption, +) (Client, error) { + return createClientWithKeyspace(ctx, nullKeyspaceID, svrAddrs, security, opts...) } // NewClientWithKeyspace creates a client with context and the specified keyspace id. -func NewClientWithKeyspace(ctx context.Context, keyspaceID uint32, svrAddrs []string, security SecurityOption, opts ...ClientOption) (Client, error) { - log.Info("[pd] create pd client with endpoints and keyspace", zap.Strings("pd-address", svrAddrs), zap.Uint32("keyspace-id", keyspaceID)) - +// And now, it's only for test purpose. +func NewClientWithKeyspace( + ctx context.Context, keyspaceID uint32, svrAddrs []string, + security SecurityOption, opts ...ClientOption, +) (Client, error) { + if keyspaceID < defaultKeyspaceID || keyspaceID > maxKeyspaceID { + return nil, errors.Errorf("invalid keyspace id %d. It must be in the range of [%d, %d]", + keyspaceID, defaultKeyspaceID, maxKeyspaceID) + } + return createClientWithKeyspace(ctx, keyspaceID, svrAddrs, security, opts...) +} + +// createClientWithKeyspace creates a client with context and the specified keyspace id. +func createClientWithKeyspace( + ctx context.Context, keyspaceID uint32, svrAddrs []string, + security SecurityOption, opts ...ClientOption, +) (Client, error) { tlsCfg := &tlsutil.TLSConfig{ CAPath: security.CAPath, CertPath: security.CertPath, @@ -317,15 +365,163 @@ func NewClientWithKeyspace(ctx context.Context, keyspaceID uint32, svrAddrs []st opt(c) } - c.pdSvcDiscovery = newPDServiceDiscovery(clientCtx, clientCancel, &c.wg, c.setServiceMode, c.svrUrls, c.tlsCfg, c.option) + c.pdSvcDiscovery = newPDServiceDiscovery( + clientCtx, clientCancel, &c.wg, c.setServiceMode, + keyspaceID, c.svrUrls, c.tlsCfg, c.option) + if err := c.setup(); err != nil { + c.cancel() + return nil, err + } + + return c, nil +} + +// APIVersion is the API version the server and the client is using. +// See more details in https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md#kvproto +type APIVersion int + +// The API versions the client supports. +// As for V1TTL, client won't use it and we just remove it. +const ( + V1 APIVersion = iota + _ + V2 +) + +// APIContext is the context for API version. +type APIContext interface { + GetAPIVersion() (apiVersion APIVersion) + GetKeyspaceName() (keyspaceName string) +} + +type apiContextV1 struct{} + +// NewAPIContextV1 creates a API context for V1. +func NewAPIContextV1() APIContext { + return &apiContextV1{} +} + +// GetAPIVersion returns the API version. +func (apiCtx *apiContextV1) GetAPIVersion() (version APIVersion) { + return V1 +} + +// GetKeyspaceName returns the keyspace name. +func (apiCtx *apiContextV1) GetKeyspaceName() (keyspaceName string) { + return "" +} + +type apiContextV2 struct { + keyspaceName string +} + +// NewAPIContextV2 creates a API context with the specified keyspace name for V2. +func NewAPIContextV2(keyspaceName string) APIContext { + if len(keyspaceName) == 0 { + keyspaceName = defaultKeyspaceName + } + return &apiContextV2{keyspaceName: keyspaceName} +} + +// GetAPIVersion returns the API version. +func (apiCtx *apiContextV2) GetAPIVersion() (version APIVersion) { + return V2 +} + +// GetKeyspaceName returns the keyspace name. +func (apiCtx *apiContextV2) GetKeyspaceName() (keyspaceName string) { + return apiCtx.keyspaceName +} + +// NewClientWithAPIContext creates a client according to the API context. +func NewClientWithAPIContext( + ctx context.Context, apiCtx APIContext, svrAddrs []string, + security SecurityOption, opts ...ClientOption, +) (Client, error) { + apiVersion, keyspaceName := apiCtx.GetAPIVersion(), apiCtx.GetKeyspaceName() + switch apiVersion { + case V1: + return NewClientWithContext(ctx, svrAddrs, security, opts...) + case V2: + return newClientWithKeyspaceName(ctx, keyspaceName, svrAddrs, security, opts...) + default: + return nil, errors.Errorf("[pd] invalid API version %d", apiVersion) + } +} + +// newClientWithKeyspaceName creates a client with context and the specified keyspace name. +func newClientWithKeyspaceName( + ctx context.Context, keyspaceName string, svrAddrs []string, + security SecurityOption, opts ...ClientOption, +) (Client, error) { + log.Info("[pd] create pd client with endpoints and keyspace", + zap.Strings("pd-address", svrAddrs), zap.String("keyspace-name", keyspaceName)) + + tlsCfg := &tlsutil.TLSConfig{ + CAPath: security.CAPath, + CertPath: security.CertPath, + KeyPath: security.KeyPath, + + SSLCABytes: security.SSLCABytes, + SSLCertBytes: security.SSLCertBytes, + SSLKEYBytes: security.SSLKEYBytes, + } + + clientCtx, clientCancel := context.WithCancel(ctx) + c := &client{ + updateTokenConnectionCh: make(chan struct{}, 1), + ctx: clientCtx, + cancel: clientCancel, + svrUrls: addrsToUrls(svrAddrs), + tlsCfg: tlsCfg, + option: newOption(), + } + + // Inject the client options. + for _, opt := range opts { + opt(c) + } + + // Create a PD service discovery with null keyspace id, then query the real id wth the keyspace name, + // finally update the keyspace id to the PD service discovery for the following interactions. + c.pdSvcDiscovery = newPDServiceDiscovery( + clientCtx, clientCancel, &c.wg, c.setServiceMode, nullKeyspaceID, c.svrUrls, c.tlsCfg, c.option) if err := c.setup(); err != nil { c.cancel() return nil, err } + if err := c.initRetry(c.loadKeyspaceMeta, keyspaceName); err != nil { + return nil, err + } + c.pdSvcDiscovery.SetKeyspaceID(c.keyspaceID) return c, nil } +func (c *client) initRetry(f func(s string) error, str string) error { + var err error + for i := 0; i < c.option.maxRetryTimes; i++ { + if err = f(str); err == nil { + return nil + } + select { + case <-c.ctx.Done(): + return err + case <-time.After(time.Second): + } + } + return errors.WithStack(err) +} + +func (c *client) loadKeyspaceMeta(keyspace string) error { + keyspaceMeta, err := c.LoadKeyspace(context.TODO(), keyspace) + if err != nil { + return err + } + c.keyspaceID = keyspaceMeta.GetId() + return nil +} + func (c *client) setup() error { // Init the client base. if err := c.pdSvcDiscovery.Init(); err != nil { @@ -348,12 +544,7 @@ func (c *client) Close() { c.cancel() c.wg.Wait() - if tsoClient := c.getTSOClient(); tsoClient != nil { - tsoClient.Close() - } - if c.tsoSvcDiscovery != nil { - c.tsoSvcDiscovery.Close() - } + c.serviceModeKeeper.close() c.pdSvcDiscovery.Close() if c.tokenDispatcher != nil { @@ -364,64 +555,69 @@ func (c *client) Close() { } func (c *client) setServiceMode(newMode pdpb.ServiceMode) { - c.svcModeMutex.Lock() - defer c.svcModeMutex.Unlock() + c.Lock() + defer c.Unlock() if newMode == c.serviceMode { return } - - log.Info("changing service mode", zap.String("old-mode", pdpb.ServiceMode_name[int32(c.serviceMode)]), - zap.String("new-mode", pdpb.ServiceMode_name[int32(newMode)])) - - if newMode == pdpb.ServiceMode_UNKNOWN_SVC_MODE { - log.Warn("intend to switch to unknown service mode. do nothing") - return - } - - var newTSOCli *tsoClient - tsoSvcDiscovery := c.tsoSvcDiscovery - ctx, cancel := context.WithCancel(c.ctx) - - if newMode == pdpb.ServiceMode_PD_SVC_MODE { - newTSOCli = newTSOClient(ctx, cancel, c.option, c.keyspaceID, - c.pdSvcDiscovery, c.pdSvcDiscovery.(tsoAllocatorEventSource), &pdTSOStreamBuilderFactory{}) - newTSOCli.Setup() - } else { - tsoSvcDiscovery = newTSOServiceDiscovery(ctx, cancel, MetaStorageClient(c), - c.GetClusterID(c.ctx), c.keyspaceID, c.svrUrls, c.tlsCfg, c.option) - newTSOCli = newTSOClient(ctx, cancel, c.option, c.keyspaceID, - tsoSvcDiscovery, tsoSvcDiscovery.(tsoAllocatorEventSource), &tsoTSOStreamBuilderFactory{}) - if err := tsoSvcDiscovery.Init(); err != nil { - cancel() - log.Error("failed to initialize tso service discovery. keep the current service mode", - zap.Strings("svr-urls", c.svrUrls), zap.String("current-mode", pdpb.ServiceMode_name[int32(c.serviceMode)]), zap.Error(err)) + log.Info("[pd] changing service mode", + zap.String("old-mode", c.serviceMode.String()), + zap.String("new-mode", newMode.String())) + // Re-create a new TSO client. + var ( + newTSOCli *tsoClient + newTSOSvcDiscovery ServiceDiscovery + ) + switch newMode { + case pdpb.ServiceMode_PD_SVC_MODE: + newTSOCli = newTSOClient(c.ctx, c.option, c.keyspaceID, + c.pdSvcDiscovery, &pdTSOStreamBuilderFactory{}) + case pdpb.ServiceMode_API_SVC_MODE: + newTSOSvcDiscovery = newTSOServiceDiscovery( + c.ctx, MetaStorageClient(c), c.pdSvcDiscovery, + c.GetClusterID(c.ctx), c.keyspaceID, c.tlsCfg, c.option) + // At this point, the keyspace group isn't known yet. Starts from the default keyspace group, + // and will be updated later. + newTSOCli = newTSOClient(c.ctx, c.option, c.keyspaceID, + newTSOSvcDiscovery, &tsoTSOStreamBuilderFactory{}) + if err := newTSOSvcDiscovery.Init(); err != nil { + log.Error("[pd] failed to initialize tso service discovery. keep the current service mode", + zap.Strings("svr-urls", c.svrUrls), + zap.String("current-mode", c.serviceMode.String()), + zap.Error(err)) return } - newTSOCli.Setup() - } - - // cleanup the old tso client - if oldTSOCli := c.getTSOClient(); oldTSOCli != nil { - oldTSOCli.Close() + case pdpb.ServiceMode_UNKNOWN_SVC_MODE: + log.Warn("[pd] intend to switch to unknown service mode, just return") + return } - if c.serviceMode == pdpb.ServiceMode_API_SVC_MODE { - c.tsoSvcDiscovery.Close() + newTSOCli.Setup() + // Replace the old TSO client. + oldTSOClient := c.tsoClient + c.tsoClient = newTSOCli + oldTSOClient.Close() + // Replace the old TSO service discovery if needed. + oldTSOSvcDiscovery := c.tsoSvcDiscovery + if newTSOSvcDiscovery != nil { + c.tsoSvcDiscovery = newTSOSvcDiscovery + // Close the old TSO service discovery safely after both the old client + // and service discovery are replaced. + if oldTSOSvcDiscovery != nil { + oldTSOSvcDiscovery.Close() + } } - - c.tsoSvcDiscovery = tsoSvcDiscovery - c.tsoClient.Store(newTSOCli) - - log.Info("service mode changed", zap.String("old-mode", pdpb.ServiceMode_name[int32(c.serviceMode)]), - zap.String("new-mode", pdpb.ServiceMode_name[int32(newMode)])) + oldMode := c.serviceMode c.serviceMode = newMode + log.Info("[pd] service mode changed", + zap.String("old-mode", oldMode.String()), + zap.String("new-mode", newMode.String())) } -func (c *client) getTSOClient() *tsoClient { - if tsoCli := c.tsoClient.Load(); tsoCli != nil { - return tsoCli.(*tsoClient) - } - return nil +func (c *client) getServiceClientProxy() (*tsoClient, pdpb.ServiceMode) { + c.RLock() + defer c.RUnlock() + return c.tsoClient, c.serviceMode } func (c *client) scheduleUpdateTokenConnection() { @@ -586,13 +782,12 @@ func (c *client) GetLocalTSAsync(ctx context.Context, dcLocation string) TSFutur req := tsoReqPool.Get().(*tsoRequest) req.requestCtx = ctx req.clientCtx = c.ctx - tsoClient := c.getTSOClient() + tsoClient, _ := c.getServiceClientProxy() req.start = time.Now() - req.keyspaceID = c.keyspaceID req.dcLocation = dcLocation if tsoClient == nil { - req.done <- errs.ErrClientGetTSO + req.done <- errs.ErrClientGetTSO.FastGenByArgs("tso client is nil") return req } @@ -616,6 +811,26 @@ func (c *client) GetLocalTS(ctx context.Context, dcLocation string) (physical in return resp.Wait() } +func (c *client) GetMinTS(ctx context.Context) (physical int64, logical int64, err error) { + tsoClient, serviceMode := c.getServiceClientProxy() + if tsoClient == nil { + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs("tso client is nil") + } + + switch serviceMode { + case pdpb.ServiceMode_UNKNOWN_SVC_MODE: + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs("unknown service mode") + case pdpb.ServiceMode_PD_SVC_MODE: + // If the service mode is switched to API during GetTS() call, which happens during migration, + // returning the default timeline should be fine. + return c.GetTS(ctx) + case pdpb.ServiceMode_API_SVC_MODE: + return tsoClient.getMinTS(ctx) + default: + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs("undefined service mode") + } +} + func handleRegionResponse(res *pdpb.GetRegionResponse) *Region { if res.Region == nil { return nil @@ -1154,7 +1369,9 @@ func IsLeaderChange(err error) bool { return true } errMsg := err.Error() - return strings.Contains(errMsg, errs.NotLeaderErr) || strings.Contains(errMsg, errs.MismatchLeaderErr) + return strings.Contains(errMsg, errs.NotLeaderErr) || + strings.Contains(errMsg, errs.MismatchLeaderErr) || + strings.Contains(errMsg, errs.NotServedErr) } func trimHTTPPrefix(str string) string { @@ -1305,7 +1522,7 @@ func (c *client) respForErr(observer prometheus.Observer, start time.Time, err e // GetTSOAllocators returns {dc-location -> TSO allocator leader URL} connection map // For test only. func (c *client) GetTSOAllocators() *sync.Map { - tsoClient := c.getTSOClient() + tsoClient, _ := c.getServiceClientProxy() if tsoClient == nil { return nil } diff --git a/client/client_test.go b/client/client_test.go index 43c0cc5c308..e82fe861a0e 100644 --- a/client/client_test.go +++ b/client/client_test.go @@ -24,6 +24,7 @@ import ( "github.com/stretchr/testify/require" "github.com/tikv/pd/client/testutil" "github.com/tikv/pd/client/tlsutil" + "github.com/tikv/pd/client/tsoutil" "go.uber.org/goleak" "google.golang.org/grpc" ) @@ -32,13 +33,13 @@ func TestMain(m *testing.M) { goleak.VerifyTestMain(m, testutil.LeakOptions...) } -func TestTsLessEqual(t *testing.T) { +func TestTSLessEqual(t *testing.T) { re := require.New(t) - re.True(tsLessEqual(9, 9, 9, 9)) - re.True(tsLessEqual(8, 9, 9, 8)) - re.False(tsLessEqual(9, 8, 8, 9)) - re.False(tsLessEqual(9, 8, 9, 6)) - re.True(tsLessEqual(9, 6, 9, 8)) + re.True(tsoutil.TSLessEqual(9, 9, 9, 9)) + re.True(tsoutil.TSLessEqual(8, 9, 9, 8)) + re.False(tsoutil.TSLessEqual(9, 8, 8, 9)) + re.False(tsoutil.TSLessEqual(9, 8, 9, 6)) + re.True(tsoutil.TSLessEqual(9, 6, 9, 8)) } func TestUpdateURLs(t *testing.T) { @@ -58,11 +59,11 @@ func TestUpdateURLs(t *testing.T) { cli := &pdServiceDiscovery{option: newOption()} cli.urls.Store([]string{}) cli.updateURLs(members[1:]) - re.Equal(getURLs([]*pdpb.Member{members[1], members[3], members[2]}), cli.GetURLs()) + re.Equal(getURLs([]*pdpb.Member{members[1], members[3], members[2]}), cli.GetServiceURLs()) cli.updateURLs(members[1:]) - re.Equal(getURLs([]*pdpb.Member{members[1], members[3], members[2]}), cli.GetURLs()) + re.Equal(getURLs([]*pdpb.Member{members[1], members[3], members[2]}), cli.GetServiceURLs()) cli.updateURLs(members) - re.Equal(getURLs([]*pdpb.Member{members[1], members[3], members[2], members[0]}), cli.GetURLs()) + re.Equal(getURLs([]*pdpb.Member{members[1], members[3], members[2], members[0]}), cli.GetServiceURLs()) } const testClientURL = "tmp://test.url:5255" diff --git a/client/errs/errno.go b/client/errs/errno.go index 43c08396f1e..73bbd41541e 100644 --- a/client/errs/errno.go +++ b/client/errs/errno.go @@ -21,27 +21,37 @@ import ( ) const ( - // NotLeaderErr indicates the the non-leader member received the requests which should be received by leader. + // NotLeaderErr indicates the non-leader member received the requests which should be received by leader. + // Note: keep the same as the ones defined on the server side, because the client side checks if an error message + // contains this string to judge whether the leader is changed. NotLeaderErr = "is not leader" - // MismatchLeaderErr indicates the the non-leader member received the requests which should be received by leader. + // MismatchLeaderErr indicates the non-leader member received the requests which should be received by leader. + // Note: keep the same as the ones defined on the server side, because the client side checks if an error message + // contains this string to judge whether the leader is changed. MismatchLeaderErr = "mismatch leader id" - RetryTimeoutErr = "retry timeout" + // NotServedErr indicates an tso node/pod received the requests for the keyspace groups which are not served by it. + // Note: keep the same as the ones defined on the server side, because the client side checks if an error message + // contains this string to judge whether the leader is changed. + NotServedErr = "is not served" + RetryTimeoutErr = "retry timeout" ) // client errors var ( - ErrClientGetProtoClient = errors.Normalize("failed to get proto client", errors.RFCCodeText("PD:client:ErrClientGetProtoClient")) - ErrClientCreateTSOStream = errors.Normalize("create TSO stream failed, %s", errors.RFCCodeText("PD:client:ErrClientCreateTSOStream")) - ErrClientTSOStreamClosed = errors.Normalize("encountered TSO stream being closed unexpectedly", errors.RFCCodeText("PD:client:ErrClientTSOStreamClosed")) - ErrClientGetTSOTimeout = errors.Normalize("get TSO timeout", errors.RFCCodeText("PD:client:ErrClientGetTSOTimeout")) - ErrClientGetTSO = errors.Normalize("get TSO failed, %v", errors.RFCCodeText("PD:client:ErrClientGetTSO")) - ErrClientGetLeader = errors.Normalize("get leader from %v error", errors.RFCCodeText("PD:client:ErrClientGetLeader")) - ErrClientGetMember = errors.Normalize("get member failed", errors.RFCCodeText("PD:client:ErrClientGetMember")) - ErrClientGetClusterInfo = errors.Normalize("get cluster info failed", errors.RFCCodeText("PD:client:ErrClientGetClusterInfo")) - ErrClientUpdateMember = errors.Normalize("update member failed, %v", errors.RFCCodeText("PD:client:ErrUpdateMember")) - ErrClientProtoUnmarshal = errors.Normalize("failed to unmarshal proto", errors.RFCCodeText("PD:proto:ErrClientProtoUnmarshal")) - ErrClientGetMultiResponse = errors.Normalize("get invalid value response %v, must only one", errors.RFCCodeText("PD:client:ErrClientGetMultiResponse")) - ErrClientGetServingEndpoint = errors.Normalize("get serving endpoint failed", errors.RFCCodeText("PD:client:ErrClientGetServingEndpoint")) + ErrClientGetProtoClient = errors.Normalize("failed to get proto client", errors.RFCCodeText("PD:client:ErrClientGetProtoClient")) + ErrClientCreateTSOStream = errors.Normalize("create TSO stream failed, %s", errors.RFCCodeText("PD:client:ErrClientCreateTSOStream")) + ErrClientTSOStreamClosed = errors.Normalize("encountered TSO stream being closed unexpectedly", errors.RFCCodeText("PD:client:ErrClientTSOStreamClosed")) + ErrClientGetTSOTimeout = errors.Normalize("get TSO timeout", errors.RFCCodeText("PD:client:ErrClientGetTSOTimeout")) + ErrClientGetTSO = errors.Normalize("get TSO failed, %v", errors.RFCCodeText("PD:client:ErrClientGetTSO")) + ErrClientGetMinTSO = errors.Normalize("get min TSO failed, %v", errors.RFCCodeText("PD:client:ErrClientGetMinTSO")) + ErrClientGetLeader = errors.Normalize("get leader failed, %v", errors.RFCCodeText("PD:client:ErrClientGetLeader")) + ErrClientGetMember = errors.Normalize("get member failed", errors.RFCCodeText("PD:client:ErrClientGetMember")) + ErrClientGetClusterInfo = errors.Normalize("get cluster info failed", errors.RFCCodeText("PD:client:ErrClientGetClusterInfo")) + ErrClientUpdateMember = errors.Normalize("update member failed, %v", errors.RFCCodeText("PD:client:ErrUpdateMember")) + ErrClientProtoUnmarshal = errors.Normalize("failed to unmarshal proto", errors.RFCCodeText("PD:proto:ErrClientProtoUnmarshal")) + ErrClientGetMultiResponse = errors.Normalize("get invalid value response %v, must only one", errors.RFCCodeText("PD:client:ErrClientGetMultiResponse")) + ErrClientGetServingEndpoint = errors.Normalize("get serving endpoint failed", errors.RFCCodeText("PD:client:ErrClientGetServingEndpoint")) + ErrClientFindGroupByKeyspaceID = errors.Normalize("can't find keyspace group by keyspace id", errors.RFCCodeText("PD:client:ErrClientFindGroupByKeyspaceID")) ) // grpcutil errors diff --git a/client/go.mod b/client/go.mod index 23d3d20de57..735d2ad30af 100644 --- a/client/go.mod +++ b/client/go.mod @@ -8,7 +8,7 @@ require ( github.com/opentracing/opentracing-go v1.2.0 github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 - github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba + github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 github.com/prometheus/client_golang v1.11.0 github.com/stretchr/testify v1.8.1 @@ -31,9 +31,9 @@ require ( github.com/prometheus/procfs v0.6.0 // indirect go.uber.org/atomic v1.9.0 // indirect go.uber.org/multierr v1.7.0 // indirect - golang.org/x/net v0.2.0 // indirect - golang.org/x/sys v0.2.0 // indirect - golang.org/x/text v0.4.0 // indirect + golang.org/x/net v0.7.0 // indirect + golang.org/x/sys v0.5.0 // indirect + golang.org/x/text v0.7.0 // indirect google.golang.org/genproto v0.0.0-20221202195650-67e5cbc046fd // indirect google.golang.org/protobuf v1.28.1 // indirect gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect diff --git a/client/go.sum b/client/go.sum index cb4df3c0def..fa6c4df1121 100644 --- a/client/go.sum +++ b/client/go.sum @@ -81,8 +81,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c h1:xpW9bvK+HuuTm github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ueLEUZgtx2cIogM0v4Zj5uvvzhuuiu7Pn8HzMPg= github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 h1:C3N3itkduZXDZFh4N3vQ5HEtld3S+Y+StULhWVvumU0= github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00/go.mod h1:4qGtCB0QK0wBzKtFEGDhxXnSnbQApw1gc9siScUl8ew= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba h1:7g2yM0llENlRqtjboBKFBJ8N9SE01hPDpKuTwxBLpLM= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba/go.mod h1:RjuuhxITxwATlt5adgTedg3ehKk01M03L1U4jNHdeeQ= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 h1:VXQ6Du/nKZ9IQnI9NWMzKbftWu8NV5pQkSLKIRzzGN4= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1/go.mod h1:guCyM5N+o+ru0TsoZ1hi9lDjUMs2sIBjW3ARTEpVbnk= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= @@ -161,8 +161,8 @@ golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLL golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= -golang.org/x/net v0.2.0 h1:sZfSu1wtKLGlWI4ZZayP0ck9Y73K1ynO6gqzTdBVdPU= -golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= +golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= +golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -187,14 +187,14 @@ golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.2.0 h1:ljd4t30dBnAvMZaQCevtY0xLLD0A+bRZXbgLMLU1F/A= -golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg= -golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= diff --git a/client/grpcutil/grpcutil.go b/client/grpcutil/grpcutil.go index 7f71468be1c..2a589dda80d 100644 --- a/client/grpcutil/grpcutil.go +++ b/client/grpcutil/grpcutil.go @@ -90,12 +90,13 @@ func GetOrCreateGRPCConn(ctx context.Context, clientConns *sync.Map, addr string if err != nil { return nil, err } - old, ok := clientConns.LoadOrStore(addr, cc) - if !ok { + conn, loaded := clientConns.LoadOrStore(addr, cc) + if !loaded { // Successfully stored the connection. return cc, nil } cc.Close() - log.Debug("use old connection", zap.String("target", cc.Target()), zap.String("state", cc.GetState().String())) - return old.(*grpc.ClientConn), nil + cc = conn.(*grpc.ClientConn) + log.Debug("use existing connection", zap.String("target", cc.Target()), zap.String("state", cc.GetState().String())) + return cc, nil } diff --git a/client/pd_service_discovery.go b/client/pd_service_discovery.go index f17688d43f0..9966b8c75cb 100644 --- a/client/pd_service_discovery.go +++ b/client/pd_service_discovery.go @@ -18,6 +18,7 @@ import ( "context" "reflect" "sort" + "strings" "sync" "sync/atomic" "time" @@ -40,6 +41,13 @@ const ( updateMemberTimeout = time.Second // Use a shorter timeout to recover faster from network isolation. ) +type serviceType int + +const ( + apiService serviceType = iota + tsoService +) + // ServiceDiscovery defines the general interface for service discovery on a quorum-based cluster // or a primary/secondary configured cluster. type ServiceDiscovery interface { @@ -49,8 +57,16 @@ type ServiceDiscovery interface { Close() // GetClusterID returns the ID of the cluster GetClusterID() uint64 - // GetURLs returns the URLs of the servers. - GetURLs() []string + // GetKeyspaceID returns the ID of the keyspace + GetKeyspaceID() uint32 + // SetKeyspaceID sets the ID of the keyspace + SetKeyspaceID(keyspaceID uint32) + // GetKeyspaceGroupID returns the ID of the keyspace group + GetKeyspaceGroupID() uint32 + // DiscoverServiceURLs discovers the microservice with the specified type and returns the server urls. + DiscoverMicroservice(svcType serviceType) ([]string, error) + // GetServiceURLs returns the URLs of the servers providing the service + GetServiceURLs() []string // GetServingEndpointClientConn returns the grpc client connection of the serving endpoint // which is the leader in a quorum-based cluster or the primary in a primary/secondary // configured cluster. @@ -114,13 +130,13 @@ type pdServiceDiscovery struct { // serviceModeUpdateCb will be called when the service mode gets updated serviceModeUpdateCb func(pdpb.ServiceMode) - // leaderSwitchedCbs will be called after the leader swichted + // leaderSwitchedCbs will be called after the leader switched leaderSwitchedCbs []func() // membersChangedCbs will be called after there is any membership change in the // leader and followers membersChangedCbs []func() // tsoLocalAllocLeadersUpdatedCb will be called when the local tso allocator - // leader list is updated. The input is a map {DC Localtion -> Leader Addr} + // leader list is updated. The input is a map {DC Location -> Leader Addr} tsoLocalAllocLeadersUpdatedCb tsoLocalServAddrsUpdatedFunc // tsoGlobalAllocLeaderUpdatedCb will be called when the global tso allocator // leader is updated. @@ -133,20 +149,27 @@ type pdServiceDiscovery struct { cancel context.CancelFunc closeOnce sync.Once - tlsCfg *tlsutil.TLSConfig + keyspaceID uint32 + tlsCfg *tlsutil.TLSConfig // Client option. option *option } -// newPDServiceDiscovery returns a new baseClient. -func newPDServiceDiscovery(ctx context.Context, cancel context.CancelFunc, wg *sync.WaitGroup, - serviceModeUpdateCb func(pdpb.ServiceMode), urls []string, tlsCfg *tlsutil.TLSConfig, option *option) *pdServiceDiscovery { +// newPDServiceDiscovery returns a new PD service discovery-based client. +func newPDServiceDiscovery( + ctx context.Context, cancel context.CancelFunc, + wg *sync.WaitGroup, + serviceModeUpdateCb func(pdpb.ServiceMode), + keyspaceID uint32, + urls []string, tlsCfg *tlsutil.TLSConfig, option *option, +) *pdServiceDiscovery { pdsd := &pdServiceDiscovery{ checkMembershipCh: make(chan struct{}, 1), ctx: ctx, cancel: cancel, wg: wg, serviceModeUpdateCb: serviceModeUpdateCb, + keyspaceID: keyspaceID, tlsCfg: tlsCfg, option: option, } @@ -155,26 +178,29 @@ func newPDServiceDiscovery(ctx context.Context, cancel context.CancelFunc, wg *s } func (c *pdServiceDiscovery) Init() error { - if !c.isInitialized { - if err := c.initRetry(c.initClusterID); err != nil { - c.cancel() - return err - } - if err := c.initRetry(c.updateMember); err != nil { - c.cancel() - return err - } - log.Info("[pd] init cluster id", zap.Uint64("cluster-id", c.clusterID)) - - c.updateServiceMode() + if c.isInitialized { + return nil + } - c.wg.Add(2) - go c.updateMemberLoop() - go c.updateServiceModeLoop() + if err := c.initRetry(c.initClusterID); err != nil { + c.cancel() + return err + } + if err := c.initRetry(c.updateMember); err != nil { + c.cancel() + return err + } + log.Info("[pd] init cluster id", zap.Uint64("cluster-id", c.clusterID)) - c.isInitialized = true + if err := c.checkServiceModeChanged(); err != nil { + log.Warn("[pd] failed to check service mode and will check later", zap.Error(err)) } + c.wg.Add(2) + go c.updateMemberLoop() + go c.updateServiceModeLoop() + + c.isInitialized = true return nil } @@ -198,47 +224,61 @@ func (c *pdServiceDiscovery) updateMemberLoop() { ctx, cancel := context.WithCancel(c.ctx) defer cancel() + ticker := time.NewTicker(memberUpdateInterval) + defer ticker.Stop() for { select { - case <-c.checkMembershipCh: - case <-time.After(memberUpdateInterval): case <-ctx.Done(): return + case <-ticker.C: + case <-c.checkMembershipCh: } failpoint.Inject("skipUpdateMember", func() { failpoint.Continue() }) if err := c.updateMember(); err != nil { - log.Error("[pd] failed to update member", zap.Strings("urls", c.GetURLs()), errs.ZapError(err)) + log.Error("[pd] failed to update member", zap.Strings("urls", c.GetServiceURLs()), errs.ZapError(err)) } } } func (c *pdServiceDiscovery) updateServiceModeLoop() { defer c.wg.Done() + failpoint.Inject("skipUpdateServiceMode", func() { + failpoint.Return() + }) + failpoint.Inject("usePDServiceMode", func() { + c.serviceModeUpdateCb(pdpb.ServiceMode_PD_SVC_MODE) + failpoint.Return() + }) ctx, cancel := context.WithCancel(c.ctx) defer cancel() + ticker := time.NewTicker(serviceModeUpdateInterval) + defer ticker.Stop() for { select { - case <-time.After(serviceModeUpdateInterval): case <-ctx.Done(): return + case <-ticker.C: + } + if err := c.checkServiceModeChanged(); err != nil { + log.Error("[pd] failed to update service mode", + zap.Strings("urls", c.GetServiceURLs()), errs.ZapError(err)) + c.ScheduleCheckMemberChanged() // check if the leader changed } - - c.updateServiceMode() } } -// Close releases all resources +// Close releases all resources. func (c *pdServiceDiscovery) Close() { c.closeOnce.Do(func() { - log.Info("close pd service discovery") + log.Info("[pd] close pd service discovery client") c.clientConns.Range(func(key, cc interface{}) bool { if err := cc.(*grpc.ClientConn).Close(); err != nil { - log.Error("[pd] failed to close gRPC clientConn", errs.ZapError(errs.ErrCloseGRPCConn, err)) + log.Error("[pd] failed to close grpc clientConn", errs.ZapError(errs.ErrCloseGRPCConn, err)) } c.clientConns.Delete(key) return true @@ -251,13 +291,55 @@ func (c *pdServiceDiscovery) GetClusterID() uint64 { return c.clusterID } -// GetURLs returns the URLs of the servers. +// GetKeyspaceID returns the ID of the keyspace +func (c *pdServiceDiscovery) GetKeyspaceID() uint32 { + return c.keyspaceID +} + +// SetKeyspaceID sets the ID of the keyspace +func (c *pdServiceDiscovery) SetKeyspaceID(keyspaceID uint32) { + c.keyspaceID = keyspaceID +} + +// GetKeyspaceGroupID returns the ID of the keyspace group +func (c *pdServiceDiscovery) GetKeyspaceGroupID() uint32 { + // PD/API service only supports the default keyspace group + return defaultKeySpaceGroupID +} + +// DiscoverServiceURLs discovers the microservice with the specified type and returns the server urls. +func (c *pdServiceDiscovery) DiscoverMicroservice(svcType serviceType) (urls []string, err error) { + switch svcType { + case apiService: + urls = c.GetServiceURLs() + case tsoService: + leaderAddr := c.getLeaderAddr() + if len(leaderAddr) > 0 { + clusterInfo, err := c.getClusterInfo(c.ctx, leaderAddr, c.option.timeout) + if err != nil { + log.Error("[pd] failed to get cluster info", + zap.String("leader-addr", leaderAddr), errs.ZapError(err)) + return nil, err + } + urls = clusterInfo.TsoUrls + } else { + err = errors.New("failed to get leader addr") + return nil, err + } + default: + panic("invalid service type") + } + + return urls, nil +} + +// GetServiceURLs returns the URLs of the servers. // For testing use. It should only be called when the client is closed. -func (c *pdServiceDiscovery) GetURLs() []string { +func (c *pdServiceDiscovery) GetServiceURLs() []string { return c.urls.Load().([]string) } -// GetServingAddr returns the grpc client connection of the serving endpoint +// GetServingEndpointClientConn returns the grpc client connection of the serving endpoint // which is the leader in a quorum-based cluster or the primary in a primary/secondary // configured cluster. func (c *pdServiceDiscovery) GetServingEndpointClientConn() *grpc.ClientConn { @@ -348,21 +430,21 @@ func (c *pdServiceDiscovery) initClusterID() error { ctx, cancel := context.WithCancel(c.ctx) defer cancel() clusterID := uint64(0) - for _, url := range c.GetURLs() { - clusterInfo, err := c.getClusterInfo(ctx, url, c.option.timeout) - if err != nil || clusterInfo.GetHeader() == nil { + for _, url := range c.GetServiceURLs() { + members, err := c.getMembers(ctx, url, c.option.timeout) + if err != nil || members.GetHeader() == nil { log.Warn("[pd] failed to get cluster id", zap.String("url", url), errs.ZapError(err)) continue } if clusterID == 0 { - clusterID = clusterInfo.GetHeader().GetClusterId() + clusterID = members.GetHeader().GetClusterId() continue } failpoint.Inject("skipClusterIDCheck", func() { failpoint.Continue() }) // All URLs passed in should have the same cluster ID. - if clusterInfo.GetHeader().GetClusterId() != clusterID { + if members.GetHeader().GetClusterId() != clusterID { return errors.WithStack(errUnmatchedClusterID) } } @@ -374,22 +456,32 @@ func (c *pdServiceDiscovery) initClusterID() error { return nil } -func (c *pdServiceDiscovery) updateServiceMode() { +func (c *pdServiceDiscovery) checkServiceModeChanged() error { leaderAddr := c.getLeaderAddr() - if len(leaderAddr) > 0 { - clusterInfo, err := c.getClusterInfo(c.ctx, leaderAddr, c.option.timeout) - if err != nil { - log.Warn("[pd] failed to get cluster info for the leader", zap.String("leader-addr", leaderAddr), errs.ZapError(err)) - return + if len(leaderAddr) == 0 { + return errors.New("no leader found") + } + + clusterInfo, err := c.getClusterInfo(c.ctx, leaderAddr, c.option.timeout) + if err != nil { + if strings.Contains(err.Error(), "Unimplemented") { + // If the method is not supported, we set it to pd mode. + // TODO: it's a hack way to solve the compatibility issue. + // we need to remove this after all maintained version supports the method. + c.serviceModeUpdateCb(pdpb.ServiceMode_PD_SVC_MODE) + return nil } - c.serviceModeUpdateCb(clusterInfo.ServiceModes[0]) - } else { - log.Warn("[pd] no leader found") + return err } + if clusterInfo == nil || len(clusterInfo.ServiceModes) == 0 { + return errors.WithStack(errNoServiceModeReturned) + } + c.serviceModeUpdateCb(clusterInfo.ServiceModes[0]) + return nil } func (c *pdServiceDiscovery) updateMember() error { - for i, url := range c.GetURLs() { + for i, url := range c.GetServiceURLs() { failpoint.Inject("skipFirstUpdateMember", func() { if i == 0 { failpoint.Continue() @@ -405,7 +497,7 @@ func (c *pdServiceDiscovery) updateMember() error { var errTSO error if err == nil { if members.GetLeader() == nil || len(members.GetLeader().GetClientUrls()) == 0 { - err = errs.ErrClientGetLeader.FastGenByArgs("leader address don't exist") + err = errs.ErrClientGetLeader.FastGenByArgs("leader address doesn't exist") } // Still need to update TsoAllocatorLeaders, even if there is no PD leader errTSO = c.switchTSOAllocatorLeaders(members.GetTsoAllocatorLeaders()) @@ -482,7 +574,7 @@ func (c *pdServiceDiscovery) updateURLs(members []*pdpb.Member) { } sort.Strings(urls) - oldURLs := c.GetURLs() + oldURLs := c.GetServiceURLs() // the url list is same. if reflect.DeepEqual(oldURLs, urls) { return diff --git a/client/tso_batch_controller.go b/client/tso_batch_controller.go index 3ad05ca7cba..842c772abd9 100644 --- a/client/tso_batch_controller.go +++ b/client/tso_batch_controller.go @@ -44,16 +44,16 @@ func newTSOBatchController(tsoRequestCh chan *tsoRequest, maxBatchSize int) *tso // fetchPendingRequests will start a new round of the batch collecting from the channel. // It returns true if everything goes well, otherwise false which means we should stop the service. func (tbc *tsoBatchController) fetchPendingRequests(ctx context.Context, maxBatchWaitInterval time.Duration) error { - var firstTSORequest *tsoRequest + var firstRequest *tsoRequest select { case <-ctx.Done(): return ctx.Err() - case firstTSORequest = <-tbc.tsoRequestCh: + case firstRequest = <-tbc.tsoRequestCh: } // Start to batch when the first TSO request arrives. tbc.batchStartTime = time.Now() tbc.collectedRequestCount = 0 - tbc.pushRequest(firstTSORequest) + tbc.pushRequest(firstRequest) // This loop is for trying best to collect more requests, so we use `tbc.maxBatchSize` here. fetchPendingRequestsLoop: @@ -130,7 +130,7 @@ func (tbc *tsoBatchController) adjustBestBatchSize() { } } -func (tbc *tsoBatchController) revokePendingTSORequest(err error) { +func (tbc *tsoBatchController) revokePendingRequest(err error) { for i := 0; i < len(tbc.tsoRequestCh); i++ { req := <-tbc.tsoRequestCh req.done <- err diff --git a/client/tso_client.go b/client/tso_client.go index f1542ce4ed5..9aae31bba5a 100644 --- a/client/tso_client.go +++ b/client/tso_client.go @@ -22,8 +22,11 @@ import ( "time" "github.com/pingcap/errors" + "github.com/pingcap/kvproto/pkg/pdpb" + "github.com/pingcap/kvproto/pkg/tsopb" "github.com/pingcap/log" "github.com/tikv/pd/client/errs" + "github.com/tikv/pd/client/tsoutil" "go.uber.org/zap" "google.golang.org/grpc" healthpb "google.golang.org/grpc/health/grpc_health_v1" @@ -31,14 +34,17 @@ import ( // TSOClient is the client used to get timestamps. type TSOClient interface { - // GetTS gets a timestamp from PD. + // GetTS gets a timestamp from PD or TSO microservice. GetTS(ctx context.Context) (int64, int64, error) - // GetTSAsync gets a timestamp from PD, without block the caller. + // GetTSAsync gets a timestamp from PD or TSO microservice, without block the caller. GetTSAsync(ctx context.Context) TSFuture - // GetLocalTS gets a local timestamp from PD. + // GetLocalTS gets a local timestamp from PD or TSO microservice. GetLocalTS(ctx context.Context, dcLocation string) (int64, int64, error) - // GetLocalTSAsync gets a local timestamp from PD, without block the caller. + // GetLocalTSAsync gets a local timestamp from PD or TSO microservice, without block the caller. GetLocalTSAsync(ctx context.Context, dcLocation string) TSFuture + // GetMinTS gets a timestamp from PD or the minimal timestamp across all keyspace groups from + // the TSO microservice. + GetMinTS(ctx context.Context) (int64, int64, error) } type tsoRequest struct { @@ -48,7 +54,6 @@ type tsoRequest struct { done chan error physical int64 logical int64 - keyspaceID uint32 dcLocation string } @@ -91,8 +96,11 @@ type tsoClient struct { } // newTSOClient returns a new TSO client. -func newTSOClient(ctx context.Context, cancel context.CancelFunc, option *option, keyspaceID uint32, - svcDiscovery ServiceDiscovery, eventSrc tsoAllocatorEventSource, factory tsoStreamBuilderFactory) *tsoClient { +func newTSOClient( + ctx context.Context, option *option, keyspaceID uint32, + svcDiscovery ServiceDiscovery, factory tsoStreamBuilderFactory, +) *tsoClient { + ctx, cancel := context.WithCancel(ctx) c := &tsoClient{ ctx: ctx, cancel: cancel, @@ -105,6 +113,7 @@ func newTSOClient(ctx context.Context, cancel context.CancelFunc, option *option updateTSOConnectionCtxsCh: make(chan struct{}, 1), } + eventSrc := svcDiscovery.(tsoAllocatorEventSource) eventSrc.SetTSOLocalServAddrsUpdatedCallback(c.updateTSOLocalServAddrs) eventSrc.SetTSOGlobalServAddrUpdatedCallback(c.updateTSOGlobalServAddr) c.svcDiscovery.AddServiceAddrsSwitchedCallback(c.scheduleUpdateTSOConnectionCtxs) @@ -124,6 +133,9 @@ func (c *tsoClient) Setup() { // Close closes the TSO client func (c *tsoClient) Close() { + if c == nil { + return + } log.Info("closing tso client") c.cancel() @@ -134,7 +146,7 @@ func (c *tsoClient) Close() { if dispatcherInterface != nil { dispatcher := dispatcherInterface.(*tsoDispatcher) tsoErr := errors.WithStack(errClosing) - dispatcher.tsoBatchController.revokePendingTSORequest(tsoErr) + dispatcher.tsoBatchController.revokePendingRequest(tsoErr) dispatcher.dispatcherCancel() } return true @@ -269,3 +281,116 @@ func (c *tsoClient) backupClientConn() (*grpc.ClientConn, string) { } return nil, "" } + +// getMinTS gets a timestamp from PD or the minimal timestamp across all keyspace groups from the TSO microservice. +func (c *tsoClient) getMinTS(ctx context.Context) (physical, logical int64, err error) { + // Immediately refresh the TSO server/pod list + addrs, err := c.svcDiscovery.DiscoverMicroservice(tsoService) + if err != nil { + return 0, 0, errs.ErrClientGetMinTSO.Wrap(err).GenWithStackByCause() + } + if len(addrs) == 0 { + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs("no tso servers/pods discovered") + } + + // Get the minimal timestamp from the TSO servers/pods + var mutex sync.Mutex + resps := make([]*tsopb.GetMinTSResponse, 0) + wg := sync.WaitGroup{} + wg.Add(len(addrs)) + for _, addr := range addrs { + go func(addr string) { + defer wg.Done() + resp, err := c.getMinTSFromSingleServer(ctx, addr, c.option.timeout) + if err != nil || resp == nil { + log.Warn("[tso] failed to get min ts from tso server", + zap.String("address", addr), zap.Error(err)) + return + } + mutex.Lock() + defer mutex.Unlock() + resps = append(resps, resp) + }(addr) + } + wg.Wait() + + // Check the results. The returned minimal timestamp is valid if all the conditions are met: + // 1. The number of responses is equal to the number of TSO servers/pods. + // 2. The number of keyspace groups asked is equal to the number of TSO servers/pods. + // 3. The minimal timestamp is not zero. + var ( + minTS *pdpb.Timestamp + keyspaceGroupsAsked uint32 + ) + if len(resps) == 0 { + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs("none of tso server/pod responded") + } + emptyTS := &pdpb.Timestamp{} + keyspaceGroupsTotal := resps[0].KeyspaceGroupsTotal + for _, resp := range resps { + if resp.KeyspaceGroupsTotal == 0 { + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs("the tso service has no keyspace group") + } + if resp.KeyspaceGroupsTotal != keyspaceGroupsTotal { + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs( + "the tso service has inconsistent keyspace group total count") + } + keyspaceGroupsAsked += resp.KeyspaceGroupsServing + if tsoutil.CompareTimestamp(resp.Timestamp, emptyTS) > 0 && + (minTS == nil || tsoutil.CompareTimestamp(resp.Timestamp, minTS) < 0) { + minTS = resp.Timestamp + } + } + + if keyspaceGroupsAsked != keyspaceGroupsTotal { + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs( + fmt.Sprintf("can't query all the tso keyspace groups. Asked %d, expected %d", + keyspaceGroupsAsked, keyspaceGroupsTotal)) + } + + if minTS == nil { + return 0, 0, errs.ErrClientGetMinTSO.FastGenByArgs("the tso service is not ready") + } + + return minTS.Physical, tsoutil.AddLogical(minTS.Logical, 0, minTS.SuffixBits), nil +} + +func (c *tsoClient) getMinTSFromSingleServer( + ctx context.Context, tsoSrvAddr string, timeout time.Duration, +) (*tsopb.GetMinTSResponse, error) { + cc, err := c.svcDiscovery.GetOrCreateGRPCConn(tsoSrvAddr) + if err != nil { + return nil, errs.ErrClientGetMinTSO.FastGenByArgs( + fmt.Sprintf("can't connect to tso server %s", tsoSrvAddr)) + } + + cctx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + + resp, err := tsopb.NewTSOClient(cc).GetMinTS( + cctx, &tsopb.GetMinTSRequest{ + Header: &tsopb.RequestHeader{ + ClusterId: c.svcDiscovery.GetClusterID(), + KeyspaceId: c.svcDiscovery.GetKeyspaceID(), + KeyspaceGroupId: c.svcDiscovery.GetKeyspaceGroupID(), + }, + DcLocation: globalDCLocation, + }) + if err != nil { + attachErr := errors.Errorf("error:%s target:%s status:%s", + err, cc.Target(), cc.GetState().String()) + return nil, errs.ErrClientGetMinTSO.Wrap(attachErr).GenWithStackByCause() + } + if resp == nil { + attachErr := errors.Errorf("error:%s target:%s status:%s", + "no min ts info collected", cc.Target(), cc.GetState().String()) + return nil, errs.ErrClientGetMinTSO.Wrap(attachErr).GenWithStackByCause() + } + if resp.GetHeader().GetError() != nil { + attachErr := errors.Errorf("error:%s target:%s status:%s", + resp.GetHeader().GetError().String(), cc.Target(), cc.GetState().String()) + return nil, errs.ErrClientGetMinTSO.Wrap(attachErr).GenWithStackByCause() + } + + return resp, nil +} diff --git a/client/tso_dispatcher.go b/client/tso_dispatcher.go index 13bbda1a707..d2d62814619 100644 --- a/client/tso_dispatcher.go +++ b/client/tso_dispatcher.go @@ -27,6 +27,7 @@ import ( "github.com/pingcap/log" "github.com/tikv/pd/client/errs" "github.com/tikv/pd/client/grpcutil" + "github.com/tikv/pd/client/tsoutil" "go.uber.org/zap" "google.golang.org/grpc" "google.golang.org/grpc/codes" @@ -211,7 +212,7 @@ func (c *tsoClient) tsoDispatcherCheckLoop() { case <-ticker.C: case <-c.checkTSODispatcherCh: case <-loopCtx.Done(): - log.Info("exit tso dispacther loop") + log.Info("exit tso dispatcher loop") return } } @@ -376,7 +377,8 @@ tsoBatchLoop: zap.String("dc-location", dc)) } else { log.Error("[tso] fetch pending tso requests error", - zap.String("dc-location", dc), errs.ZapError(errs.ErrClientGetTSO, err)) + zap.String("dc-location", dc), + errs.ZapError(errs.ErrClientGetTSO.FastGenByArgs("when fetch pending tso requests"), err)) } return } @@ -404,7 +406,7 @@ tsoBatchLoop: err = errs.ErrClientCreateTSOStream.FastGenByArgs(errs.RetryTimeoutErr) log.Error("[tso] create tso stream error", zap.String("dc-location", dc), errs.ZapError(err)) c.svcDiscovery.ScheduleCheckMemberChanged() - c.finishTSORequest(tbc.getCollectedRequests(), 0, 0, 0, errors.WithStack(err)) + c.finishRequest(tbc.getCollectedRequests(), 0, 0, 0, errors.WithStack(err)) continue tsoBatchLoop case <-time.After(retryInterval): continue streamChoosingLoop @@ -440,7 +442,7 @@ tsoBatchLoop: case tsDeadlineCh.(chan deadline) <- dl: } opts = extractSpanReference(tbc, opts[:0]) - err = c.processTSORequests(stream, dc, tbc, opts) + err = c.processRequests(stream, dc, tbc, opts) close(done) // If error happens during tso stream handling, reset stream and run the next trial. if err != nil { @@ -450,7 +452,10 @@ tsoBatchLoop: default: } c.svcDiscovery.ScheduleCheckMemberChanged() - log.Error("[tso] getTS error", zap.String("dc-location", dc), zap.String("stream-addr", streamAddr), errs.ZapError(errs.ErrClientGetTSO, err)) + log.Error("[tso] getTS error", + zap.String("dc-location", dc), + zap.String("stream-addr", streamAddr), + errs.ZapError(errs.ErrClientGetTSO.FastGenByArgs("after processing requests"), err)) // Set `stream` to nil and remove this stream from the `connectionCtxs` due to error. connectionCtxs.Delete(streamAddr) cancel() @@ -497,8 +502,8 @@ func (c *tsoClient) chooseStream(connectionCtxs *sync.Map) (connectionCtx *tsoCo type tsoConnectionContext struct { streamAddr string - // Current stream to send gRPC requests, pdpb.PD_TsoClient for a leader/follower in the PD cluser, - // or tsopb.TSO_TsoClient for a primary/secondary in the TSO clusrer + // Current stream to send gRPC requests, pdpb.PD_TsoClient for a leader/follower in the PD cluster, + // or tsopb.TSO_TsoClient for a primary/secondary in the TSO cluster stream tsoStream ctx context.Context cancel context.CancelFunc @@ -615,7 +620,7 @@ func (c *tsoClient) tryConnectToTSO( // or of keyspace group primary/secondaries. func (c *tsoClient) getAllTSOStreamBuilders() map[string]tsoStreamBuilder { var ( - addrs = c.svcDiscovery.GetURLs() + addrs = c.svcDiscovery.GetServiceURLs() streamBuilders = make(map[string]tsoStreamBuilder, len(addrs)) cc *grpc.ClientConn err error @@ -662,7 +667,8 @@ func (c *tsoClient) tryConnectToTSOWithProxy(dispatcherCtx context.Context, dc s cctx, cancel := context.WithCancel(dispatcherCtx) // Do not proxy the leader client. if addr != leaderAddr { - log.Info("[tso] use follower to forward tso stream to do the proxy", zap.String("dc", dc), zap.String("addr", addr)) + log.Info("[tso] use follower to forward tso stream to do the proxy", + zap.String("dc", dc), zap.String("addr", addr)) cctx = grpcutil.BuildForwardContext(cctx, forwardedHost) } // Create the TSO stream. @@ -676,7 +682,8 @@ func (c *tsoClient) tryConnectToTSOWithProxy(dispatcherCtx context.Context, dc s connectionCtxs.Store(addr, &tsoConnectionContext{addr, stream, cctx, cancel}) continue } - log.Error("[tso] create the tso stream failed", zap.String("dc", dc), zap.String("addr", addr), errs.ZapError(err)) + log.Error("[tso] create the tso stream failed", + zap.String("dc", dc), zap.String("addr", addr), errs.ZapError(err)) cancel() } return nil @@ -691,33 +698,32 @@ func extractSpanReference(tbc *tsoBatchController, opts []opentracing.StartSpanO return opts } -func (c *tsoClient) processTSORequests(stream tsoStream, dcLocation string, tbc *tsoBatchController, opts []opentracing.StartSpanOption) error { +func (c *tsoClient) processRequests( + stream tsoStream, dcLocation string, tbc *tsoBatchController, opts []opentracing.StartSpanOption, +) error { if len(opts) > 0 { - span := opentracing.StartSpan("pdclient.processTSORequests", opts...) + span := opentracing.StartSpan("pdclient.processRequests", opts...) defer span.Finish() } requests := tbc.getCollectedRequests() count := int64(len(requests)) - physical, logical, suffixBits, err := stream.processRequests(c.svcDiscovery.GetClusterID(), dcLocation, requests, tbc.batchStartTime) + physical, logical, suffixBits, err := stream.processRequests( + c.svcDiscovery.GetClusterID(), c.svcDiscovery.GetKeyspaceID(), c.svcDiscovery.GetKeyspaceGroupID(), + dcLocation, requests, tbc.batchStartTime) if err != nil { - c.finishTSORequest(requests, 0, 0, 0, err) + c.finishRequest(requests, 0, 0, 0, err) return err } // `logical` is the largest ts's logical part here, we need to do the subtracting before we finish each TSO request. - firstLogical := addLogical(logical, -count+1, suffixBits) + firstLogical := tsoutil.AddLogical(logical, -count+1, suffixBits) c.compareAndSwapTS(dcLocation, physical, firstLogical, suffixBits, count) - c.finishTSORequest(requests, physical, firstLogical, suffixBits, nil) + c.finishRequest(requests, physical, firstLogical, suffixBits, nil) return nil } -// Because of the suffix, we need to shift the count before we add it to the logical part. -func addLogical(logical, count int64, suffixBits uint32) int64 { - return logical + count</tso//primary". // The is 5 digits integer with leading zeros. - tspSvcDiscoveryFormat = msServiceRootPath + "/%d/" + tsoServiceName + "/%05d/primary" + tsoSvcDiscoveryFormat = msServiceRootPath + "/%d/" + tsoServiceName + "/%05d/primary" + // initRetryInterval is the rpc retry interval during the initialization phase. + initRetryInterval = time.Second + // tsoQueryRetryMaxTimes is the max retry times for querying TSO. + tsoQueryRetryMaxTimes = 10 + // tsoQueryRetryInterval is the retry interval for querying TSO. + tsoQueryRetryInterval = 500 * time.Millisecond ) var _ ServiceDiscovery = (*tsoServiceDiscovery)(nil) var _ tsoAllocatorEventSource = (*tsoServiceDiscovery)(nil) +// keyspaceGroupSvcDiscovery is used for discovering the serving endpoints of the keyspace +// group to which the keyspace belongs +type keyspaceGroupSvcDiscovery struct { + sync.RWMutex + group *tsopb.KeyspaceGroup + // primaryAddr is the primary serving address + primaryAddr string + // secondaryAddrs are TSO secondary serving addresses + secondaryAddrs []string + // addrs are the primary/secondary serving addresses + addrs []string +} + +func (k *keyspaceGroupSvcDiscovery) update( + keyspaceGroup *tsopb.KeyspaceGroup, + newPrimaryAddr string, + secondaryAddrs, addrs []string, +) (oldPrimaryAddr string, primarySwitched, secondaryChanged bool) { + k.Lock() + defer k.Unlock() + + // If the new primary address is empty, we don't switch the primary address. + oldPrimaryAddr = k.primaryAddr + if len(newPrimaryAddr) > 0 { + primarySwitched = !strings.EqualFold(oldPrimaryAddr, newPrimaryAddr) + k.primaryAddr = newPrimaryAddr + } + + if !reflect.DeepEqual(k.secondaryAddrs, secondaryAddrs) { + k.secondaryAddrs = secondaryAddrs + secondaryChanged = true + } + + k.group = keyspaceGroup + k.addrs = addrs + return +} + +// tsoServerDiscovery is for discovering the serving endpoints of the TSO servers +// TODO: dynamically update the TSO server addresses in the case of TSO server failover +// and scale-out/in. +type tsoServerDiscovery struct { + sync.RWMutex + addrs []string + // used for round-robin load balancing + selectIdx int + // failureCount counts the consecutive failures for communicating with the tso servers + failureCount int +} + +func (t *tsoServerDiscovery) countFailure() bool { + t.Lock() + defer t.Unlock() + t.failureCount++ + return t.failureCount >= len(t.addrs) +} + +func (t *tsoServerDiscovery) resetFailure() { + t.Lock() + defer t.Unlock() + t.failureCount = 0 +} + // tsoServiceDiscovery is the service discovery client of the independent TSO service + type tsoServiceDiscovery struct { - clusterID uint64 - keyspaceID uint32 - urls atomic.Value // Store as []string - // primary key is the etcd path used for discoverying the serving endpoint of this keyspace - primaryKey string - // TSO Primary URL - primary atomic.Value // Store as string - // TSO Secondary URLs - secondaries atomic.Value // Store as []string - metacli MetaStorageClient + metacli MetaStorageClient + apiSvcDiscovery ServiceDiscovery + clusterID uint64 + keyspaceID uint32 + + // defaultDiscoveryKey is the etcd path used for discovering the serving endpoints of + // the default keyspace group + defaultDiscoveryKey string + // tsoServersDiscovery is for discovering the serving endpoints of the TSO servers + *tsoServerDiscovery + + // keyspaceGroupSD is for discovering the serving endpoints of the keyspace group + keyspaceGroupSD *keyspaceGroupSvcDiscovery // addr -> a gRPC connection clientConns sync.Map // Store as map[string]*grpc.ClientConn // localAllocPrimariesUpdatedCb will be called when the local tso allocator primary list is updated. - // The input is a map {DC Localtion -> Leader Addr} + // The input is a map {DC Location -> Leader Addr} localAllocPrimariesUpdatedCb tsoLocalServAddrsUpdatedFunc // globalAllocPrimariesUpdatedCb will be called when the local tso allocator primary list is updated. globalAllocPrimariesUpdatedCb tsoGlobalServAddrUpdatedFunc checkMembershipCh chan struct{} - ctx context.Context - cancel context.CancelFunc - wg sync.WaitGroup + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup + printFallbackLogOnce sync.Once tlsCfg *tlsutil.TLSConfig @@ -79,29 +155,47 @@ type tsoServiceDiscovery struct { } // newTSOServiceDiscovery returns a new client-side service discovery for the independent TSO service. -func newTSOServiceDiscovery(ctx context.Context, cancel context.CancelFunc, metacli MetaStorageClient, - clusterID uint64, keyspaceID uint32, urls []string, tlsCfg *tlsutil.TLSConfig, option *option) ServiceDiscovery { +func newTSOServiceDiscovery( + ctx context.Context, metacli MetaStorageClient, apiSvcDiscovery ServiceDiscovery, + clusterID uint64, keyspaceID uint32, tlsCfg *tlsutil.TLSConfig, option *option, +) ServiceDiscovery { + ctx, cancel := context.WithCancel(ctx) c := &tsoServiceDiscovery{ ctx: ctx, cancel: cancel, metacli: metacli, + apiSvcDiscovery: apiSvcDiscovery, keyspaceID: keyspaceID, clusterID: clusterID, - primaryKey: fmt.Sprintf(tspSvcDiscoveryFormat, clusterID, keyspaceID), tlsCfg: tlsCfg, option: option, checkMembershipCh: make(chan struct{}, 1), } - c.urls.Store(urls) + c.keyspaceGroupSD = &keyspaceGroupSvcDiscovery{ + primaryAddr: "", + secondaryAddrs: make([]string, 0), + addrs: make([]string, 0), + } + c.tsoServerDiscovery = &tsoServerDiscovery{addrs: make([]string, 0)} + // Start with the default keyspace group. The actual keyspace group, to which the keyspace belongs, + // will be discovered later. + c.defaultDiscoveryKey = fmt.Sprintf(tsoSvcDiscoveryFormat, clusterID, defaultKeySpaceGroupID) - log.Info("created tso service discovery", zap.String("discovery-key", c.primaryKey)) + log.Info("created tso service discovery", + zap.Uint64("cluster-id", clusterID), + zap.Uint32("keyspace-id", keyspaceID), + zap.String("default-discovery-key", c.defaultDiscoveryKey)) return c } // Init initialize the concrete client underlying func (c *tsoServiceDiscovery) Init() error { - if err := c.initRetry(c.updateMember); err != nil { + log.Info("initializing tso service discovery", + zap.Int("max-retry-times", c.option.maxRetryTimes), + zap.Duration("retry-interval", initRetryInterval)) + if err := c.retry(c.option.maxRetryTimes, initRetryInterval, c.updateMember); err != nil { + log.Error("failed to update member. initialization failed.", zap.Error(err)) c.cancel() return err } @@ -110,16 +204,18 @@ func (c *tsoServiceDiscovery) Init() error { return nil } -func (c *tsoServiceDiscovery) initRetry(f func() error) error { +func (c *tsoServiceDiscovery) retry( + maxRetryTimes int, retryInterval time.Duration, f func() error, +) error { var err error - for i := 0; i < c.option.maxRetryTimes; i++ { + for i := 0; i < maxRetryTimes; i++ { if err = f(); err == nil { return nil } select { case <-c.ctx.Done(): return err - case <-time.After(time.Second): + case <-time.After(retryInterval): } } return errors.WithStack(err) @@ -157,7 +253,10 @@ func (c *tsoServiceDiscovery) startCheckMemberLoop() { log.Info("[tso] exit check member loop") return } - if err := c.updateMember(); err != nil { + // Make sure tsoQueryRetryMaxTimes * tsoQueryRetryInterval is far less than memberUpdateInterval, + // so that we can speed up the process of tso service discovery when failover happens on the + // tso service side and also ensures it won't call updateMember too frequently during normal time. + if err := c.retry(tsoQueryRetryMaxTimes, tsoQueryRetryInterval, c.updateMember); err != nil { log.Error("[tso] failed to update member", errs.ZapError(err)) } } @@ -168,10 +267,48 @@ func (c *tsoServiceDiscovery) GetClusterID() uint64 { return c.clusterID } -// GetURLs returns the URLs of the servers. +// GetKeyspaceID returns the ID of the keyspace +func (c *tsoServiceDiscovery) GetKeyspaceID() uint32 { + return c.keyspaceID +} + +// SetKeyspaceID sets the ID of the keyspace +func (c *tsoServiceDiscovery) SetKeyspaceID(keyspaceID uint32) { + c.keyspaceID = keyspaceID +} + +// GetKeyspaceGroupID returns the ID of the keyspace group. If the keyspace group is unknown, +// it returns the default keyspace group ID. +func (c *tsoServiceDiscovery) GetKeyspaceGroupID() uint32 { + c.keyspaceGroupSD.RLock() + defer c.keyspaceGroupSD.RUnlock() + if c.keyspaceGroupSD.group == nil { + return defaultKeySpaceGroupID + } + return c.keyspaceGroupSD.group.Id +} + +// DiscoverServiceURLs discovers the microservice with the specified type and returns the server urls. +func (c *tsoServiceDiscovery) DiscoverMicroservice(svcType serviceType) ([]string, error) { + var urls []string + + switch svcType { + case apiService: + case tsoService: + return c.apiSvcDiscovery.DiscoverMicroservice(tsoService) + default: + panic("invalid service type") + } + + return urls, nil +} + +// GetServiceURLs returns the URLs of the tso primary/secondary addresses of this keyspace group. // For testing use. It should only be called when the client is closed. -func (c *tsoServiceDiscovery) GetURLs() []string { - return c.urls.Load().([]string) +func (c *tsoServiceDiscovery) GetServiceURLs() []string { + c.keyspaceGroupSD.RLock() + defer c.keyspaceGroupSD.RUnlock() + return c.keyspaceGroupSD.addrs } // GetServingAddr returns the grpc client connection of the serving endpoint @@ -183,7 +320,7 @@ func (c *tsoServiceDiscovery) GetServingEndpointClientConn() *grpc.ClientConn { return nil } -// GetClientConns returns the mapping {addr -> a gRPC connectio} +// GetClientConns returns the mapping {addr -> a gRPC connection} func (c *tsoServiceDiscovery) GetClientConns() *sync.Map { return &c.clientConns } @@ -195,7 +332,7 @@ func (c *tsoServiceDiscovery) GetServingAddr() string { } // GetBackupAddrs gets the addresses of the current reachable and healthy -// backup service endpoints randomly. Backup service endpoints are secondaries in +// backup service endpoints. Backup service endpoints are secondaries in // a primary/secondary configured cluster. func (c *tsoServiceDiscovery) GetBackupAddrs() []string { return c.getSecondaryAddrs() @@ -206,7 +343,7 @@ func (c *tsoServiceDiscovery) GetOrCreateGRPCConn(addr string) (*grpc.ClientConn return grpcutil.GetOrCreateGRPCConn(c.ctx, &c.clientConns, addr, c.tlsCfg, c.option.gRPCDialOptions...) } -// ScheduleCheckMemberChanged is used to trigger a check to see if there is any change in ervice endpoints. +// ScheduleCheckMemberChanged is used to trigger a check to see if there is any change in service endpoints. func (c *tsoServiceDiscovery) ScheduleCheckMemberChanged() { select { case c.checkMembershipCh <- struct{}{}: @@ -217,7 +354,12 @@ func (c *tsoServiceDiscovery) ScheduleCheckMemberChanged() { // Immediately check if there is any membership change among the primary/secondaries in // a primary/secondary configured cluster. func (c *tsoServiceDiscovery) CheckMemberChanged() error { - return c.updateMember() + c.apiSvcDiscovery.CheckMemberChanged() + if err := c.retry(tsoQueryRetryMaxTimes, tsoQueryRetryInterval, c.updateMember); err != nil { + log.Error("[tso] failed to update member", errs.ZapError(err)) + return err + } + return nil } // AddServingAddrSwitchedCallback adds callbacks which will be called when the primary in @@ -248,69 +390,238 @@ func (c *tsoServiceDiscovery) SetTSOGlobalServAddrUpdatedCallback(callback tsoGl // getPrimaryAddr returns the primary address. func (c *tsoServiceDiscovery) getPrimaryAddr() string { - primaryAddr := c.primary.Load() - if primaryAddr == nil { - return "" - } - return primaryAddr.(string) + c.keyspaceGroupSD.RLock() + defer c.keyspaceGroupSD.RUnlock() + return c.keyspaceGroupSD.primaryAddr } // getSecondaryAddrs returns the secondary addresses. func (c *tsoServiceDiscovery) getSecondaryAddrs() []string { - secondaryAddrs := c.secondaries.Load() - if secondaryAddrs == nil { - return []string{} - } - return secondaryAddrs.([]string) + c.keyspaceGroupSD.RLock() + defer c.keyspaceGroupSD.RUnlock() + return c.keyspaceGroupSD.secondaryAddrs } -func (c *tsoServiceDiscovery) switchPrimary(addrs []string) error { - // FIXME: How to safely compare primary urls? For now, only allows one client url. - addr := addrs[0] - oldPrimary := c.getPrimaryAddr() - if addr == oldPrimary { - return nil - } - - if _, err := c.GetOrCreateGRPCConn(addr); err != nil { - log.Warn("[tso] failed to connect primary", zap.String("primary", addr), errs.ZapError(err)) - return err - } - // Set PD primary and Global TSO Allocator (which is also the PD primary) - c.primary.Store(addr) +func (c *tsoServiceDiscovery) afterPrimarySwitched(oldPrimary, newPrimary string) error { // Run callbacks if c.globalAllocPrimariesUpdatedCb != nil { - if err := c.globalAllocPrimariesUpdatedCb(addr); err != nil { + if err := c.globalAllocPrimariesUpdatedCb(newPrimary); err != nil { return err } } - log.Info("[tso] switch primary", zap.String("new-primary", addr), zap.String("old-primary", oldPrimary)) + log.Info("[tso] switch primary", + zap.String("new-primary", newPrimary), + zap.String("old-primary", oldPrimary)) return nil } func (c *tsoServiceDiscovery) updateMember() error { - resp, err := c.metacli.Get(c.ctx, []byte(c.primaryKey)) + // The keyspace membership or the primary serving address of the keyspace group, to which this + // keyspace belongs, might have been changed. We need to query tso servers to get the latest info. + tsoServerAddr, err := c.getTSOServer(c.apiSvcDiscovery) if err != nil { - log.Error("[tso] failed to get the keyspace serving endpoint", zap.String("primary-key", c.primaryKey), errs.ZapError(err)) + log.Error("[tso] failed to get tso server", errs.ZapError(err)) return err } + var keyspaceGroup *tsopb.KeyspaceGroup + if len(tsoServerAddr) > 0 { + keyspaceGroup, err = c.findGroupByKeyspaceID(c.keyspaceID, tsoServerAddr, updateMemberTimeout) + if err != nil { + if c.tsoServerDiscovery.countFailure() { + log.Error("[tso] failed to find the keyspace group", errs.ZapError(err)) + } + return err + } + c.tsoServerDiscovery.resetFailure() + } else { + // There is no error but no tso server address found, which means + // the server side hasn't been upgraded to the version that + // processes and returns GetClusterInfoResponse.TsoUrls. In this case, + // we fall back to the old way of discovering the tso primary addresses + // from etcd directly. + c.printFallbackLogOnce.Do(func() { + log.Warn("[tso] no tso server address found,"+ + " fallback to the legacy path to discover from etcd directly", + zap.String("discovery-key", c.defaultDiscoveryKey)) + }) + addrs, err := c.discoverWithLegacyPath() + if err != nil { + return err + } + if len(addrs) == 0 { + return errors.New("no tso server address found") + } + members := make([]*tsopb.KeyspaceGroupMember, 0, len(addrs)) + for _, addr := range addrs { + members = append(members, &tsopb.KeyspaceGroupMember{Address: addr}) + } + members[0].IsPrimary = true + keyspaceGroup = &tsopb.KeyspaceGroup{ + Id: defaultKeySpaceGroupID, + Members: members, + } + } + + // Initialize the serving addresses from the returned keyspace group info. + primaryAddr := "" + secondaryAddrs := make([]string, 0) + addrs := make([]string, 0, len(keyspaceGroup.Members)) + for _, m := range keyspaceGroup.Members { + addrs = append(addrs, m.Address) + if m.IsPrimary { + primaryAddr = m.Address + } else { + secondaryAddrs = append(secondaryAddrs, m.Address) + } + } + + // If the primary address is not empty, we need to create a grpc connection to it, and do it + // out of the critical section of the keyspace group service discovery. + if len(primaryAddr) > 0 { + if primarySwitched := !strings.EqualFold(primaryAddr, c.getPrimaryAddr()); primarySwitched { + if _, err := c.GetOrCreateGRPCConn(primaryAddr); err != nil { + log.Warn("[tso] failed to connect the next primary", + zap.String("next-primary", primaryAddr), errs.ZapError(err)) + return err + } + } + } + + oldPrimary, primarySwitched, secondaryChanged := + c.keyspaceGroupSD.update(keyspaceGroup, primaryAddr, secondaryAddrs, addrs) + if primarySwitched { + if err := c.afterPrimarySwitched(oldPrimary, primaryAddr); err != nil { + return err + } + } + if primarySwitched || secondaryChanged { + log.Info("[tso] updated keyspace group service discovery info", + zap.String("keyspace-group-service", keyspaceGroup.String())) + } + + // Even if the primary address is empty, we still updated other returned info above, including the + // keyspace group info and the secondary addresses. + if len(primaryAddr) == 0 { + return errors.New("no primary address found") + } + + return nil +} + +// Query the keyspace group info from the tso server by the keyspace ID. The server side will return +// the info of the keyspace group to which this keyspace belongs. +func (c *tsoServiceDiscovery) findGroupByKeyspaceID( + keyspaceID uint32, tsoSrvAddr string, timeout time.Duration, +) (*tsopb.KeyspaceGroup, error) { + failpoint.Inject("unexpectedCallOfFindGroupByKeyspaceID", func(val failpoint.Value) { + keyspaceToCheck, ok := val.(int) + if ok && keyspaceID == uint32(keyspaceToCheck) { + panic("findGroupByKeyspaceID is called unexpectedly") + } + }) + ctx, cancel := context.WithTimeout(c.ctx, timeout) + defer cancel() + + cc, err := c.GetOrCreateGRPCConn(tsoSrvAddr) + if err != nil { + return nil, err + } + + resp, err := tsopb.NewTSOClient(cc).FindGroupByKeyspaceID( + ctx, &tsopb.FindGroupByKeyspaceIDRequest{ + Header: &tsopb.RequestHeader{ + ClusterId: c.clusterID, + KeyspaceId: keyspaceID, + KeyspaceGroupId: defaultKeySpaceGroupID, + }, + KeyspaceId: keyspaceID, + }) + if err != nil { + attachErr := errors.Errorf("error:%s target:%s status:%s", + err, cc.Target(), cc.GetState().String()) + return nil, errs.ErrClientFindGroupByKeyspaceID.Wrap(attachErr).GenWithStackByCause() + } + if resp.GetHeader().GetError() != nil { + attachErr := errors.Errorf("error:%s target:%s status:%s", + resp.GetHeader().GetError().String(), cc.Target(), cc.GetState().String()) + return nil, errs.ErrClientFindGroupByKeyspaceID.Wrap(attachErr).GenWithStackByCause() + } + if resp.KeyspaceGroup == nil { + attachErr := errors.Errorf("error:%s target:%s status:%s", + "no keyspace group found", cc.Target(), cc.GetState().String()) + return nil, errs.ErrClientFindGroupByKeyspaceID.Wrap(attachErr).GenWithStackByCause() + } + + return resp.KeyspaceGroup, nil +} + +func (c *tsoServiceDiscovery) getTSOServer(sd ServiceDiscovery) (string, error) { + c.Lock() + defer c.Unlock() + + var ( + addrs []string + err error + ) + t := c.tsoServerDiscovery + if len(t.addrs) == 0 || t.failureCount == len(t.addrs) { + addrs, err = sd.DiscoverMicroservice(tsoService) + if err != nil { + return "", err + } + failpoint.Inject("serverReturnsNoTSOAddrs", func() { + log.Info("[failpoint] injected error: server returns no tso addrs") + addrs = nil + }) + if len(addrs) == 0 { + // There is no error but no tso server address found, which means + // the server side hasn't been upgraded to the version that + // processes and returns GetClusterInfoResponse.TsoUrls. Return here + // and handle the fallback logic outside of this function. + return "", nil + } + + log.Info("update tso server addresses", zap.Strings("addrs", addrs)) + + t.addrs = addrs + t.selectIdx = 0 + t.failureCount = 0 + } + + // Pick a TSO server in a round-robin way. + tsoServerAddr := t.addrs[t.selectIdx] + t.selectIdx++ + t.selectIdx %= len(t.addrs) + + return tsoServerAddr, nil +} + +func (c *tsoServiceDiscovery) discoverWithLegacyPath() ([]string, error) { + resp, err := c.metacli.Get(c.ctx, []byte(c.defaultDiscoveryKey)) + if err != nil { + log.Error("[tso] failed to get the keyspace serving endpoint", + zap.String("discovery-key", c.defaultDiscoveryKey), errs.ZapError(err)) + return nil, err + } if resp == nil || len(resp.Kvs) == 0 { - log.Error("[tso] didn't find the keyspace serving endpoint", zap.String("primary-key", c.primaryKey)) - return errs.ErrClientGetServingEndpoint + log.Error("[tso] didn't find the keyspace serving endpoint", + zap.String("primary-key", c.defaultDiscoveryKey)) + return nil, errs.ErrClientGetServingEndpoint } else if resp.Count > 1 { - return errs.ErrClientGetMultiResponse.FastGenByArgs(resp.Kvs) + return nil, errs.ErrClientGetMultiResponse.FastGenByArgs(resp.Kvs) } value := resp.Kvs[0].Value primary := &tsopb.Participant{} if err := proto.Unmarshal(value, primary); err != nil { - return errs.ErrClientProtoUnmarshal.Wrap(err).GenWithStackByCause() + return nil, errs.ErrClientProtoUnmarshal.Wrap(err).GenWithStackByCause() } listenUrls := primary.GetListenUrls() if len(listenUrls) == 0 { - log.Error("[tso] the keyspace serving endpoint list is empty", zap.String("primary-key", c.primaryKey)) - return errs.ErrClientGetServingEndpoint + log.Error("[tso] the keyspace serving endpoint list is empty", + zap.String("discovery-key", c.defaultDiscoveryKey)) + return nil, errs.ErrClientGetServingEndpoint } - return c.switchPrimary(listenUrls) + return listenUrls, nil } diff --git a/client/tso_stream.go b/client/tso_stream.go index c8435abf9e5..5b658279cac 100644 --- a/client/tso_stream.go +++ b/client/tso_stream.go @@ -70,7 +70,9 @@ type tsoTSOStreamBuilder struct { client tsopb.TSOClient } -func (b *tsoTSOStreamBuilder) build(ctx context.Context, cancel context.CancelFunc, timeout time.Duration) (tsoStream, error) { +func (b *tsoTSOStreamBuilder) build( + ctx context.Context, cancel context.CancelFunc, timeout time.Duration, +) (tsoStream, error) { done := make(chan struct{}) // TODO: we need to handle a conner case that this goroutine is timeout while the stream is successfully created. go checkStreamTimeout(ctx, cancel, done, timeout) @@ -97,16 +99,19 @@ func checkStreamTimeout(ctx context.Context, cancel context.CancelFunc, done cha type tsoStream interface { // processRequests processes TSO requests in streaming mode to get timestamps - processRequests(clusterID uint64, dcLocation string, requests []*tsoRequest, - batchStartTime time.Time) (physical, logical int64, suffixBits uint32, err error) + processRequests( + clusterID uint64, keyspaceID, keyspaceGroupID uint32, dcLocation string, + requests []*tsoRequest, batchStartTime time.Time, + ) (physical, logical int64, suffixBits uint32, err error) } type pdTSOStream struct { stream pdpb.PD_TsoClient } -func (s *pdTSOStream) processRequests(clusterID uint64, dcLocation string, requests []*tsoRequest, - batchStartTime time.Time) (physical, logical int64, suffixBits uint32, err error) { +func (s *pdTSOStream) processRequests( + clusterID uint64, _, _ uint32, dcLocation string, requests []*tsoRequest, batchStartTime time.Time, +) (physical, logical int64, suffixBits uint32, err error) { start := time.Now() count := int64(len(requests)) req := &pdpb.TsoRequest{ @@ -143,7 +148,8 @@ func (s *pdTSOStream) processRequests(clusterID uint64, dcLocation string, reque return } - physical, logical, suffixBits = resp.GetTimestamp().GetPhysical(), resp.GetTimestamp().GetLogical(), resp.GetTimestamp().GetSuffixBits() + ts := resp.GetTimestamp() + physical, logical, suffixBits = ts.GetPhysical(), ts.GetLogical(), ts.GetSuffixBits() return } @@ -151,13 +157,17 @@ type tsoTSOStream struct { stream tsopb.TSO_TsoClient } -func (s *tsoTSOStream) processRequests(clusterID uint64, dcLocation string, requests []*tsoRequest, - batchStartTime time.Time) (physical, logical int64, suffixBits uint32, err error) { +func (s *tsoTSOStream) processRequests( + clusterID uint64, keyspaceID, keyspaceGroupID uint32, dcLocation string, + requests []*tsoRequest, batchStartTime time.Time, +) (physical, logical int64, suffixBits uint32, err error) { start := time.Now() count := int64(len(requests)) req := &tsopb.TsoRequest{ Header: &tsopb.RequestHeader{ - ClusterId: clusterID, + ClusterId: clusterID, + KeyspaceId: keyspaceID, + KeyspaceGroupId: keyspaceGroupID, }, Count: uint32(count), DcLocation: dcLocation, @@ -189,6 +199,7 @@ func (s *tsoTSOStream) processRequests(clusterID uint64, dcLocation string, requ return } - physical, logical, suffixBits = resp.GetTimestamp().GetPhysical(), resp.GetTimestamp().GetLogical(), resp.GetTimestamp().GetSuffixBits() + ts := resp.GetTimestamp() + physical, logical, suffixBits = ts.GetPhysical(), ts.GetLogical(), ts.GetSuffixBits() return } diff --git a/client/tsoutil/tsoutil.go b/client/tsoutil/tsoutil.go new file mode 100644 index 00000000000..ffc449640ac --- /dev/null +++ b/client/tsoutil/tsoutil.go @@ -0,0 +1,46 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tsoutil + +import ( + "github.com/pingcap/kvproto/pkg/pdpb" +) + +// AddLogical shifts the count before we add it to the logical part. +func AddLogical(logical, count int64, suffixBits uint32) int64 { + return logical + count< tsoTwo, returns 1. +// If tsoOne = tsoTwo, returns 0. +// If tsoOne < tsoTwo, returns -1. +func CompareTimestamp(tsoOne, tsoTwo *pdpb.Timestamp) int { + if tsoOne.GetPhysical() > tsoTwo.GetPhysical() || (tsoOne.GetPhysical() == tsoTwo.GetPhysical() && tsoOne.GetLogical() > tsoTwo.GetLogical()) { + return 1 + } + if tsoOne.GetPhysical() == tsoTwo.GetPhysical() && tsoOne.GetLogical() == tsoTwo.GetLogical() { + return 0 + } + return -1 +} diff --git a/errors.toml b/errors.toml index ede86964087..58cb8980e5b 100644 --- a/errors.toml +++ b/errors.toml @@ -63,7 +63,7 @@ create TSO stream failed, %s ["PD:client:ErrClientGetLeader"] error = ''' -get leader from %v error +get leader failed, %v ''' ["PD:client:ErrClientGetMember"] @@ -481,6 +481,11 @@ error = ''' init file log error, %s ''' +["PD:member:ErrCheckCampaign"] +error = ''' +check campaign failed +''' + ["PD:member:ErrEtcdLeaderNotFound"] error = ''' etcd leader not found @@ -711,11 +716,51 @@ error = ''' get allocator failed, %s ''' +["PD:tso:ErrGetAllocatorManager"] +error = ''' +get allocator manager failed, %s +''' + ["PD:tso:ErrGetLocalAllocator"] error = ''' get local allocator failed, %s ''' +["PD:tso:ErrGetMinTS"] +error = ''' +get min ts failed, %s +''' + +["PD:tso:ErrKeyspaceGroupIDInvalid"] +error = ''' +the keyspace group id is invalid, %s +''' + +["PD:tso:ErrKeyspaceGroupNotInitialized"] +error = ''' +the keyspace group %d isn't initialized +''' + +["PD:tso:ErrKeyspaceNotAssigned"] +error = ''' +the keyspace %d isn't assigned to any keyspace group +''' + +["PD:tso:ErrLoadKeyspaceGroupsRetryExhausted"] +error = ''' +load keyspace groups retry exhausted, %s +''' + +["PD:tso:ErrLoadKeyspaceGroupsTerminated"] +error = ''' +load keyspace groups terminated +''' + +["PD:tso:ErrLoadKeyspaceGroupsTimeout"] +error = ''' +load keyspace groups timeout +''' + ["PD:tso:ErrLogicOverflow"] error = ''' logic part overflow diff --git a/go.mod b/go.mod index 4f9c07bb06e..00e538362c4 100644 --- a/go.mod +++ b/go.mod @@ -25,8 +25,8 @@ require ( github.com/phf/go-queue v0.0.0-20170504031614-9abe38d0371d github.com/pingcap/errcode v0.3.0 github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c - github.com/pingcap/failpoint v0.0.0-20200702092429-9f69995143ce - github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba + github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 + github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 github.com/pingcap/sysutil v0.0.0-20211208032423-041a72e5860d github.com/pingcap/tidb-dashboard v0.0.0-20230209052558-a58fc2a7e924 @@ -37,8 +37,8 @@ require ( github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072 github.com/spf13/cobra v1.0.0 github.com/spf13/pflag v1.0.5 - github.com/stretchr/testify v1.8.1 - github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba + github.com/stretchr/testify v1.8.2 + github.com/swaggo/http-swagger v1.2.6 github.com/swaggo/swag v1.8.3 github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 github.com/unrolled/render v1.0.1 @@ -47,13 +47,15 @@ require ( go.uber.org/goleak v1.1.12 go.uber.org/zap v1.19.1 golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a - golang.org/x/text v0.4.0 + golang.org/x/text v0.9.0 golang.org/x/time v0.1.0 - golang.org/x/tools v0.2.0 + golang.org/x/tools v0.6.0 google.golang.org/grpc v1.51.0 gotest.tools/gotestsum v1.7.0 ) +require github.com/shirou/gopsutil v3.21.3+incompatible // indirect + require ( github.com/KyleBanks/depth v1.2.1 // indirect github.com/Masterminds/semver v1.5.0 // indirect @@ -98,7 +100,7 @@ require ( github.com/golang/snappy v0.0.4 // indirect github.com/google/pprof v0.0.0-20211122183932-1daafda22083 // indirect github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect - github.com/google/uuid v1.3.0 // indirect + github.com/google/uuid v1.3.0 github.com/gorilla/websocket v1.4.2 // indirect github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4 // indirect github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect @@ -140,14 +142,13 @@ require ( github.com/rs/cors v1.7.0 // indirect github.com/russross/blackfriday/v2 v2.0.1 // indirect github.com/samber/lo v1.37.0 // indirect - github.com/sergi/go-diff v1.0.1-0.20180205163309-da645544ed44 // indirect - github.com/shirou/gopsutil v3.21.3+incompatible // indirect + github.com/sergi/go-diff v1.1.0 // indirect github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 // indirect github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect github.com/sirupsen/logrus v1.4.2 // indirect github.com/soheilhy/cmux v0.1.4 github.com/stretchr/objx v0.5.0 // indirect - github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 // indirect + github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 // indirect github.com/tidwall/gjson v1.9.3 // indirect github.com/tklauser/go-sysconf v0.3.11 // indirect github.com/tklauser/numcpus v0.6.0 // indirect @@ -165,13 +166,13 @@ require ( go.uber.org/fx v1.12.0 // indirect go.uber.org/multierr v1.7.0 // indirect golang.org/x/crypto v0.1.0 // indirect - golang.org/x/image v0.0.0-20200119044424-58c23975cae1 // indirect - golang.org/x/mod v0.6.0 // indirect - golang.org/x/net v0.2.0 // indirect - golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 // indirect + golang.org/x/image v0.5.0 // indirect + golang.org/x/mod v0.8.0 // indirect + golang.org/x/net v0.9.0 // indirect + golang.org/x/oauth2 v0.4.0 // indirect golang.org/x/sync v0.1.0 // indirect - golang.org/x/sys v0.3.0 // indirect - golang.org/x/term v0.2.0 // indirect + golang.org/x/sys v0.7.0 // indirect + golang.org/x/term v0.7.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/genproto v0.0.0-20221202195650-67e5cbc046fd // indirect google.golang.org/protobuf v1.28.1 // indirect diff --git a/go.sum b/go.sum index 34dd7bbba18..7d2b6f0c746 100644 --- a/go.sum +++ b/go.sum @@ -9,7 +9,6 @@ github.com/KyleBanks/depth v1.2.1/go.mod h1:jzSb9d0L43HxTQfT+oSA1EEp2q+ne2uh6Xge github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= -github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= @@ -22,8 +21,8 @@ github.com/VividCortex/mysqlerr v1.0.0 h1:5pZ2TZA+YnzPgzBfiUWGqWmKDVNBdrkf9g+DNe github.com/VividCortex/mysqlerr v1.0.0/go.mod h1:xERx8E4tBhLvpjzdUyQiSfUxeMcATEQrflDAfXsqcAE= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502 h1:L8IbaI/W6h5Cwgh0n4zGeZpVK78r/jBf9ASurHo9+/o= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502/go.mod h1:pmnBM9bxWSiHvC/gSWunUIyDvGn33EkP2CUjxFKtTTM= +github.com/agiledragon/gomonkey/v2 v2.3.1/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= -github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/alvaroloes/enumer v1.1.2/go.mod h1:FxrjvuXoDAx9isTJrv4c+T410zFi0DtXIT0m65DJ+Wo= github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY= @@ -118,16 +117,12 @@ github.com/gin-contrib/cors v1.4.0/go.mod h1:bs9pNM0x/UsmHPBWT2xZz9ROh8xYjYkiURU github.com/gin-contrib/gzip v0.0.1 h1:ezvKOL6jH+jlzdHNE4h9h8q8uMpDQjyl0NN0Jd7jozc= github.com/gin-contrib/gzip v0.0.1/go.mod h1:fGBJBCdt6qCZuCAOwWuFhBB4OOq9EFqlo5dEaFhhu5w= github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= -github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE= github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI= github.com/gin-gonic/gin v1.3.0/go.mod h1:7cKuhb5qV2ggCFctp2fJQ+ErvciLZrIeoOSOm6mUr7Y= -github.com/gin-gonic/gin v1.4.0/go.mod h1:OW2EZn3DO8Ln9oIKOvM++LBO+5UPHJJDH72/q/3rZdM= github.com/gin-gonic/gin v1.6.3/go.mod h1:75u5sXoLsGZoRN5Sgbi1eraJ4GU3++wFwWzhwvtwp4M= github.com/gin-gonic/gin v1.8.1 h1:4+fr/el88TOO3ewCmQr8cx/CtZ/umlIRIs5M4NTNjf8= github.com/gin-gonic/gin v1.8.1/go.mod h1:ji8BvRH1azfM+SYow9zQ6SZMvR8qOMZHmsCuWR9tTTk= -github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAUcHlgXVRs= -github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ= github.com/go-echarts/go-echarts v1.0.0 h1:n181E4iXwj4zrU9VYmdM2m8dyhERt2w9k9YhHqdp6A8= github.com/go-echarts/go-echarts v1.0.0/go.mod h1:qbmyAb/Rl1f2w7wKba1D4LoNq4U164yO4/wedFbcWyo= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= @@ -137,18 +132,13 @@ github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7 github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM= github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= -github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= -github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= -github.com/go-openapi/jsonreference v0.19.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= github.com/go-openapi/jsonreference v0.19.6 h1:UBIxjkht+AWIgYzCDSv2GN+E/togfwXUJFRTWhl2Jjs= github.com/go-openapi/jsonreference v0.19.6/go.mod h1:diGHMEHg2IqXZGKxqyvWdfWU/aim5Dprw5bqpKkTvns= -github.com/go-openapi/spec v0.19.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= github.com/go-openapi/spec v0.20.4 h1:O8hJrt0UMnhHcluhIdUgCLRWyM2x7QkBXRvOs7m+O1M= github.com/go-openapi/spec v0.20.4/go.mod h1:faYFR1CvsJZ0mNsmsphTMSoRrNV3TEDoAM7FOEWeq8I= -github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= github.com/go-openapi/swag v0.19.15 h1:D2NRCBzS9/pEY3gP9Nl8aDqGUcPFrwG2p+CNFrLyrCM= github.com/go-openapi/swag v0.19.15/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= @@ -223,7 +213,7 @@ github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= +github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/mux v1.7.4 h1:VuZ8uybHlWmqV03+zRzdwKL4tUnIp1MAQtp1mIFE1bc= github.com/gorilla/mux v1.7.4/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= @@ -282,6 +272,7 @@ github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/u github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= @@ -304,7 +295,6 @@ github.com/leodido/go-urn v1.2.1/go.mod h1:zt4jvISO2HfUBqxjfIshjdMTYS56ZS/qv49ic github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4= github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= -github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA= @@ -314,7 +304,6 @@ github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVc github.com/mattn/go-colorable v0.1.8 h1:c1ghPdyEDarC70ftn0y+A/Ee++9zz8ljHG1b13eJ0s8= github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= -github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= @@ -363,6 +352,11 @@ github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/onsi/gomega v1.20.1 h1:PA/3qinGoukvymdIDV8pii6tiZgC8kbmJO6Z5+b002Q= github.com/onsi/gomega v1.20.1/go.mod h1:DtrZpjmvpn2mPm4YWQa0/ALMDj9v4YxLgojwPeREyVo= +github.com/otiai10/copy v1.7.0/go.mod h1:rmRl6QPdJj6EiUqXQ/4Nn2lLXoNQjFCQbbNrxgc/t3U= +github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE= +github.com/otiai10/curr v1.0.0/go.mod h1:LskTG5wDwr8Rs+nNQ+1LlxRjAtTZZjtJW4rMXl6j4vs= +github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo= +github.com/otiai10/mint v1.3.3/go.mod h1:/yxELlJQ0ufhjUwhshSj+wFjZ78CnZ48/1wtmBH1OTc= github.com/pascaldekloe/name v0.0.0-20180628100202-0fd16699aae1/go.mod h1:eD5JxqMiuNYyFNmyY9rkJ/slN8y59oEu4Ei7F8OoKWQ= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml/v2 v2.0.1 h1:8e3L2cCQzLFi2CR4g7vGFuFxX7Jl1kKX8gW+iV0GUKU= @@ -381,11 +375,11 @@ github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTw github.com/pingcap/errors v0.11.5-0.20190809092503-95897b64e011/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8= github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c h1:xpW9bvK+HuuTmyFqUwr+jcCvpVkK7sumiz+ko5H9eq4= github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ueLEUZgtx2cIogM0v4Zj5uvvzhuuiu7Pn8HzMPg= -github.com/pingcap/failpoint v0.0.0-20200702092429-9f69995143ce h1:Y1kCxlCtlPTMtVcOkjUcuQKh+YrluSo7+7YMCQSzy30= -github.com/pingcap/failpoint v0.0.0-20200702092429-9f69995143ce/go.mod h1:w4PEZ5y16LeofeeGwdgZB4ddv9bLyDuIX+ljstgKZyk= +github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 h1:C3N3itkduZXDZFh4N3vQ5HEtld3S+Y+StULhWVvumU0= +github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00/go.mod h1:4qGtCB0QK0wBzKtFEGDhxXnSnbQApw1gc9siScUl8ew= github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba h1:7g2yM0llENlRqtjboBKFBJ8N9SE01hPDpKuTwxBLpLM= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba/go.mod h1:RjuuhxITxwATlt5adgTedg3ehKk01M03L1U4jNHdeeQ= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 h1:VXQ6Du/nKZ9IQnI9NWMzKbftWu8NV5pQkSLKIRzzGN4= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1/go.mod h1:guCyM5N+o+ru0TsoZ1hi9lDjUMs2sIBjW3ARTEpVbnk= github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8= github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= @@ -439,8 +433,8 @@ github.com/samber/lo v1.37.0 h1:XjVcB8g6tgUp8rsPsJ2CvhClfImrpL04YpQHXeHPhRw= github.com/samber/lo v1.37.0/go.mod h1:9vaz2O4o8oOnK23pd2TrXufcbdbJIa3b6cstBWKpopA= github.com/sasha-s/go-deadlock v0.2.0 h1:lMqc+fUb7RrFS3gQLtoQsJ7/6TV/pAIFvBsqX73DK8Y= github.com/sasha-s/go-deadlock v0.2.0/go.mod h1:StQn567HiB1fF2yJ44N9au7wOhrPS3iZqiDbRupzT10= -github.com/sergi/go-diff v1.0.1-0.20180205163309-da645544ed44 h1:tB9NOR21++IjLyVx3/PCPhWMwqGNCMQEH96A6dMZ/gc= -github.com/sergi/go-diff v1.0.1-0.20180205163309-da645544ed44/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo= +github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0= +github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= github.com/shirou/gopsutil v3.21.3+incompatible h1:uenXGGa8ESCQq+dbgtl916dmg6PSAz2cXov0uORQ9v8= github.com/shirou/gopsutil v3.21.3+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA= github.com/shirou/gopsutil/v3 v3.22.12 h1:oG0ns6poeUSxf78JtOsfygNWuEHYYz8hnnNg7P04TJs= @@ -454,6 +448,8 @@ github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4 github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072 h1:Txo4SXVJq/OgEjwgkWoxkMoTjGlcrgsQE/XSghjmu0w= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072/go.mod h1:+4nWMF0+CqEcU74SnX2NxaGqZ8zX4pcQ8Jcs77DbX5A= +github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= +github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= @@ -482,15 +478,14 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 h1:PyYN9JH5jY9j6av01SpfRMb+1DWg/i3MbGOKPxJ2wjM= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14/go.mod h1:gxQT6pBGRuIGunNf/+tSOB5OHvguWi8Tbt82WOkf35E= -github.com/swaggo/gin-swagger v1.2.0/go.mod h1:qlH2+W7zXGZkczuL+r2nEBR2JTT+/lX05Nn6vPhc7OI= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba h1:lUPlXKqgbqT2SVg2Y+eT9mu5wbqMnG+i/+Q9nK7C0Rs= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba/go.mod h1:O1lAbCgAAX/KZ80LM/OXwtWFI/5TvZlwxSg8Cq08PV0= -github.com/swaggo/swag v1.5.1/go.mod h1:1Bl9F/ZBpVWh22nY0zmYyASPO1lI/zIwRDrpZU+tv8Y= -github.com/swaggo/swag v1.6.3/go.mod h1:wcc83tB4Mb2aNiL/HP4MFeQdpHUrca+Rp/DRNgWAUio= +github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= +github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 h1:+iNTcqQJy0OZ5jk6a5NLib47eqXK8uYcPX+O4+cBpEM= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2/go.mod h1:lKJPbtWzJ9JhsTN1k1gZgleJWY/cqq0psdoMmaThG3w= +github.com/swaggo/http-swagger v1.2.6 h1:ihTjChUoSRMpFMjWw+0AkL1Ti4r6v8pCgVYLmQVRlRw= +github.com/swaggo/http-swagger v1.2.6/go.mod h1:CcoICgY3yVDk2u1LQUCMHbAj0fjlxIX+873psXlIKNA= +github.com/swaggo/swag v1.7.9/go.mod h1:gZ+TJ2w/Ve1RwQsA2IRoSOTidHz6DX+PIG8GWvbnoLU= github.com/swaggo/swag v1.8.3 h1:3pZSSCQ//gAH88lfmxM3Cd1+JCsxV8Md6f36b9hrZ5s= github.com/swaggo/swag v1.8.3/go.mod h1:jMLeXOOmYyjk8PvHTsXBdrubsNd9gUJTTCzL5iBnseg= github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 h1:1oFLiOyVl+W7bnBzGhf7BbIv9loSFQcieWWYIjLqcAw= @@ -514,11 +509,9 @@ github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1 github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966 h1:j6JEOq5QWFker+d7mFQYOhjTZonQ7YkLTHm56dbn+yM= github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= -github.com/ugorji/go v1.1.5-pre/go.mod h1:FwP/aQVg39TXzItUBMwnWp9T9gPQnXw4Poh4/oBQZ/0= github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw= github.com/ugorji/go v1.2.7/go.mod h1:nF9osbDWLy6bDVv/Rtoh6QgnvNDpmCalQV5urGCCS6M= github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= -github.com/ugorji/go/codec v1.1.5-pre/go.mod h1:tULtS6Gy1AE1yCENaw4Vb//HLH5njI2tfCQDUqRd8fI= github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY= github.com/ugorji/go/codec v1.2.7 h1:YPXUKf7fYbp/y8xloBqZOw2qaVggbfwMlI8WM3wZUJ0= github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95kRgeqEY= @@ -539,6 +532,8 @@ github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1: github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg= github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= @@ -586,14 +581,16 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20200204104054-c9f3fb736b72/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU= golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a h1:tlXy25amD5A7gOfbXdqCGN5k8ESEed/Ee1E5RcrYnqU= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= -golang.org/x/image v0.0.0-20200119044424-58c23975cae1 h1:5h3ngYt7+vXCDZCup/HkCQgW5XwmSvR/nA2JmJ0RErg= golang.org/x/image v0.0.0-20200119044424-58c23975cae1/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= +golang.org/x/image v0.5.0 h1:5JMiNunQeQw++mMOz48/ISeNu3Iweh/JaZU8ZLqHRrI= +golang.org/x/image v0.5.0/go.mod h1:FVC7BI/5Ym8R25iw5OLsgshdUBbT1h5jZTpA+mvAdZ4= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= @@ -605,8 +602,9 @@ golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.6.0 h1:b9gGHsz9/HhJ3HF5DHQytPpuwocVTChQJK3AvoLRD5I= -golang.org/x/mod v0.6.0/go.mod h1:4mET923SAdbXp2ki8ey+zGs1SLqsuM2Y0uvdZR/fUNI= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= +golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -617,10 +615,8 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= -golang.org/x/net v0.0.0-20190611141213-3f473d35a33a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= @@ -631,13 +627,15 @@ golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwY golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM= +golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.2.0 h1:sZfSu1wtKLGlWI4ZZayP0ck9Y73K1ynO6gqzTdBVdPU= -golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 h1:nt+Q6cXKz4MosCSpnbMtqiQ8Oz0pxTef2B4Vca2lvfk= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg= +golang.org/x/oauth2 v0.4.0 h1:NF0gk8LVPg1Ml7SSbGyySuoxdsXitj7TvgvuRxIMc/M= +golang.org/x/oauth2 v0.4.0/go.mod h1:RznEsdpjGAINPTOF0UH/t+xJ75L18YO3Ho6Pyn+uRec= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -645,6 +643,7 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -657,7 +656,6 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190610200419-93c9922d18ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190801041406-cbf593c0f2f3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -678,21 +676,27 @@ golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ= golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= -golang.org/x/term v0.2.0 h1:z85xZCsEl7bi/KwbNADeBYoOP0++7W1ipu+aGnpwzRM= -golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.7.0 h1:BEvjmm5fURWqcfbSKTdpkDXYBrUS1c0m8agp14W48vQ= +golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg= -golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.1.0 h1:xYY+Bajn2a7VBmTM5GikTmnK8ZuX8YgnQCqZpbBNtmA= @@ -702,10 +706,9 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190524210228-3d17549cdc6b/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.0.0-20190606050223-4d9ae51c2468/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190611222205-d73e1c7e250b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190617190820-da514acc4774/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190624222133-a101b041ded4/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= @@ -724,8 +727,10 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f golang.org/x/tools v0.0.0-20210112230658-8b4aab62c064/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.2.0 h1:G6AHpWxTMGY1KyEYoAQ5WTtIekUUvDNjan3ugu60JvE= -golang.org/x/tools v0.2.0/go.mod h1:y4OqIKeOV/fWJetJ8bXPU1sEVniLMIyDAZWeHdV+NTA= +golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -758,6 +763,7 @@ google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqw gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= @@ -776,6 +782,7 @@ gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bl gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= diff --git a/pd.code-workspace b/pd.code-workspace index bcab8bfcb8e..c4603084a93 100644 --- a/pd.code-workspace +++ b/pd.code-workspace @@ -1,21 +1,24 @@ { "folders": [ { + "name": "pd-server", "path": "." }, { + "name": "pd-client", "path": "client" }, { - "path": "tests/client" + "name": "pd-client-tests", + "path": "tests/integrations/client" }, { "name": "mcs-tests", - "path": "tests/mcs" + "path": "tests/integrations/mcs" }, { "name": "tso-tests", - "path": "tests/tso" + "path": "tests/integrations/tso" }, { "name": "pd-tso-bench", diff --git a/pkg/balancer/balancer.go b/pkg/balancer/balancer.go new file mode 100644 index 00000000000..e3105f4c0b8 --- /dev/null +++ b/pkg/balancer/balancer.go @@ -0,0 +1,61 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package balancer + +// Policy is the policy of balancer. +type Policy int + +const ( + // PolicyRoundRobin is the round robin policy. + PolicyRoundRobin Policy = iota + // PolicyLeast is the policy to return the least used node. + // TODO: move indexed heap to pkg and use it. + PolicyLeast +) + +func (p Policy) String() string { + switch p { + case PolicyRoundRobin: + return "round-robin" + case PolicyLeast: + return "least" + default: + return "unknown" + } +} + +// Balancer is the interface for balancer. +type Balancer[T uint32 | string] interface { + // Next returns next one. + Next() T + // Put puts one into balancer. + Put(T) + // Delete deletes one from balancer. + Delete(T) + // GetAll returns all nodes. + GetAll() []T + // Len returns the length of nodes. + Len() int +} + +// GenByPolicy generates a balancer by policy. +func GenByPolicy[T uint32 | string](policy Policy) Balancer[T] { + switch policy { + case PolicyRoundRobin: + return NewRoundRobin[T]() + default: // only round-robin is supported now. + return NewRoundRobin[T]() + } +} diff --git a/pkg/balancer/balancer_test.go b/pkg/balancer/balancer_test.go new file mode 100644 index 00000000000..f95487a4cc7 --- /dev/null +++ b/pkg/balancer/balancer_test.go @@ -0,0 +1,102 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package balancer + +import ( + "math/rand" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestBalancerPutAndDelete(t *testing.T) { + t.Parallel() + re := require.New(t) + balancers := []Balancer[uint32]{ + NewRoundRobin[uint32](), + } + for _, balancer := range balancers { + re.Equal(uint32(0), balancer.Next()) + // test put + exists := make(map[uint32]struct{}) + for i := 0; i < 100; i++ { + num := rand.Uint32() + balancer.Put(num) + exists[num] = struct{}{} + re.Equal(len(balancer.GetAll()), len(exists)) + t := balancer.Next() + re.Contains(exists, t) + } + // test delete + for num := range exists { + balancer.Delete(num) + delete(exists, num) + re.Equal(len(balancer.GetAll()), len(exists)) + if len(exists) == 0 { + break + } + t := balancer.Next() + re.NotEqual(t, num) + re.Contains(exists, t) + } + re.Equal(uint32(0), balancer.Next()) + } +} + +func TestBalancerDuplicate(t *testing.T) { + t.Parallel() + re := require.New(t) + balancers := []Balancer[uint32]{ + NewRoundRobin[uint32](), + } + for _, balancer := range balancers { + re.Len(balancer.GetAll(), 0) + // test duplicate put + balancer.Put(1) + re.Len(balancer.GetAll(), 1) + balancer.Put(1) + re.Len(balancer.GetAll(), 1) + // test duplicate delete + balancer.Delete(1) + re.Len(balancer.GetAll(), 0) + balancer.Delete(1) + re.Len(balancer.GetAll(), 0) + } +} + +func TestRoundRobin(t *testing.T) { + t.Parallel() + re := require.New(t) + balancer := NewRoundRobin[uint32]() + for i := 0; i < 100; i++ { + num := rand.Uint32() + balancer.Put(num) + } + statistics := make(map[uint32]int) + for i := 0; i < 1000; i++ { + statistics[balancer.Next()]++ + } + min := 1000 + max := 0 + for _, v := range statistics { + if v < min { + min = v + } + if v > max { + max = v + } + } + re.LessOrEqual(max-min, 10) +} diff --git a/pkg/balancer/round_robin.go b/pkg/balancer/round_robin.go new file mode 100644 index 00000000000..cef35c43a5f --- /dev/null +++ b/pkg/balancer/round_robin.go @@ -0,0 +1,87 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package balancer + +import ( + "sync" + "sync/atomic" +) + +// RoundRobin is a balancer that selects nodes in a round-robin fashion. +type RoundRobin[T uint32 | string] struct { + sync.RWMutex + nodes []T + exists map[T]struct{} + next uint32 +} + +// NewRoundRobin creates a balancer that selects nodes in a round-robin fashion. +func NewRoundRobin[T uint32 | string]() *RoundRobin[T] { + return &RoundRobin[T]{ + nodes: make([]T, 0), + exists: make(map[T]struct{}), + } +} + +// Next returns next address +func (r *RoundRobin[T]) Next() (t T) { + r.RLock() + defer r.RUnlock() + if len(r.nodes) == 0 { + return + } + next := atomic.AddUint32(&r.next, 1) + node := r.nodes[(int(next)-1)%len(r.nodes)] + return node +} + +// GetAll returns all nodes +func (r *RoundRobin[T]) GetAll() []T { + r.RLock() + defer r.RUnlock() + return r.nodes +} + +// Put puts one into balancer. +func (r *RoundRobin[T]) Put(node T) { + r.Lock() + defer r.Unlock() + if _, ok := r.exists[node]; !ok { + r.nodes = append(r.nodes, node) + r.exists[node] = struct{}{} + } +} + +// Delete deletes one from balancer. +func (r *RoundRobin[T]) Delete(node T) { + r.Lock() + defer r.Unlock() + if _, ok := r.exists[node]; ok { + for i, n := range r.nodes { + if n == node { + r.nodes = append(r.nodes[:i], r.nodes[i+1:]...) + delete(r.exists, node) + break + } + } + } +} + +// Len returns the length of nodes. +func (r *RoundRobin[T]) Len() int { + r.RLock() + defer r.RUnlock() + return len(r.nodes) +} diff --git a/pkg/core/storelimit/limit_test.go b/pkg/core/storelimit/limit_test.go index 34eb0440454..1cad8ff9baa 100644 --- a/pkg/core/storelimit/limit_test.go +++ b/pkg/core/storelimit/limit_test.go @@ -23,7 +23,8 @@ import ( func TestStoreLimit(t *testing.T) { re := require.New(t) rate := int64(15) - limit := NewStoreRateLimit(float64(rate)) + limit := NewStoreRateLimit(float64(rate)).(*StoreRateLimit) + re.Equal(limit.Rate(AddPeer), float64(15)) re.True(limit.Available(influence*rate, AddPeer)) re.True(limit.Take(influence*rate, AddPeer)) re.False(limit.Take(influence, AddPeer)) diff --git a/pkg/core/storelimit/store_limit.go b/pkg/core/storelimit/store_limit.go index 193a386392a..8e61e9cb298 100644 --- a/pkg/core/storelimit/store_limit.go +++ b/pkg/core/storelimit/store_limit.go @@ -83,6 +83,14 @@ func (l *StoreRateLimit) Available(cost int64, typ Type) bool { return l.limits[typ].Available(cost) } +// Rate returns the capacity of the store limit. +func (l *StoreRateLimit) Rate(typ Type) float64 { + if l.limits[typ] == nil { + return 0.0 + } + return l.limits[typ].ratePerSec +} + // Take takes count tokens from the bucket without blocking. func (l *StoreRateLimit) Take(cost int64, typ Type) bool { return l.limits[typ].Take(cost) diff --git a/pkg/election/leadership.go b/pkg/election/leadership.go index 30485a863da..4332183ae82 100644 --- a/pkg/election/leadership.go +++ b/pkg/election/leadership.go @@ -117,7 +117,7 @@ func (ls *Leadership) Campaign(leaseTimeout int64, leaderData string, cmps ...cl finalCmps = append(finalCmps, clientv3.Compare(clientv3.CreateRevision(ls.leaderKey), "=", 0)) resp, err := kv.NewSlowLogTxn(ls.client). If(finalCmps...). - Then(clientv3.OpPut(ls.leaderKey, leaderData, clientv3.WithLease(newLease.ID))). + Then(clientv3.OpPut(ls.leaderKey, leaderData, clientv3.WithLease(newLease.ID.Load().(clientv3.LeaseID)))). Commit() log.Info("check campaign resp", zap.Any("resp", resp)) if err != nil { diff --git a/pkg/election/lease.go b/pkg/election/lease.go index f7542bff042..a0db045256f 100644 --- a/pkg/election/lease.go +++ b/pkg/election/lease.go @@ -42,7 +42,7 @@ type lease struct { // etcd client and lease client *clientv3.Client lease clientv3.Lease - ID clientv3.LeaseID + ID atomic.Value // store as clientv3.LeaseID // leaseTimeout and expireTime are used to control the lease's lifetime leaseTimeout time.Duration expireTime atomic.Value @@ -64,7 +64,7 @@ func (l *lease) Grant(leaseTimeout int64) error { log.Warn("lease grants too slow", zap.Duration("cost", cost), zap.String("purpose", l.Purpose)) } log.Info("lease granted", zap.Int64("lease-id", int64(leaseResp.ID)), zap.Int64("lease-timeout", leaseTimeout), zap.String("purpose", l.Purpose)) - l.ID = leaseResp.ID + l.ID.Store(leaseResp.ID) l.leaseTimeout = time.Duration(leaseTimeout) * time.Second l.expireTime.Store(start.Add(time.Duration(leaseResp.TTL) * time.Second)) return nil @@ -80,7 +80,11 @@ func (l *lease) Close() error { // Try to revoke lease to make subsequent elections faster. ctx, cancel := context.WithTimeout(l.client.Ctx(), revokeLeaseTimeout) defer cancel() - l.lease.Revoke(ctx, l.ID) + var leaseID clientv3.LeaseID + if l.ID.Load() != nil { + leaseID = l.ID.Load().(clientv3.LeaseID) + } + l.lease.Revoke(ctx, leaseID) return l.lease.Close() } @@ -145,7 +149,11 @@ func (l *lease) keepAliveWorker(ctx context.Context, interval time.Duration) <-c start := time.Now() ctx1, cancel := context.WithTimeout(ctx, l.leaseTimeout) defer cancel() - res, err := l.lease.KeepAliveOnce(ctx1, l.ID) + var leaseID clientv3.LeaseID + if l.ID.Load() != nil { + leaseID = l.ID.Load().(clientv3.LeaseID) + } + res, err := l.lease.KeepAliveOnce(ctx1, leaseID) if err != nil { log.Warn("lease keep alive failed", zap.String("purpose", l.Purpose), errs.ZapError(err)) return diff --git a/pkg/errs/errno.go b/pkg/errs/errno.go index 155d7ef45a4..5fd8c51446e 100644 --- a/pkg/errs/errno.go +++ b/pkg/errs/errno.go @@ -17,10 +17,18 @@ package errs import "github.com/pingcap/errors" const ( - // NotLeaderErr indicates the the non-leader member received the requests which should be received by leader. + // NotLeaderErr indicates the non-leader member received the requests which should be received by leader. + // Note: keep the same as the ones defined on the client side, because the client side checks if an error message + // contains this string to judge whether the leader is changed. NotLeaderErr = "is not leader" - // MismatchLeaderErr indicates the the non-leader member received the requests which should be received by leader. + // MismatchLeaderErr indicates the non-leader member received the requests which should be received by leader. + // Note: keep the same as the ones defined on the client side, because the client side checks if an error message + // contains this string to judge whether the leader is changed. MismatchLeaderErr = "mismatch leader id" + // NotServedErr indicates an tso node/pod received the requests for the keyspace groups which are not served by it. + // Note: keep the same as the ones defined on the client side, because the client side checks if an error message + // contains this string to judge whether the leader is changed. + NotServedErr = "is not served" ) // common error in multiple packages @@ -31,20 +39,29 @@ var ( // tso errors var ( - ErrSetLocalTSOConfig = errors.Normalize("set local tso config failed, %s", errors.RFCCodeText("PD:tso:ErrSetLocalTSOConfig")) - ErrGetAllocator = errors.Normalize("get allocator failed, %s", errors.RFCCodeText("PD:tso:ErrGetAllocator")) - ErrGetLocalAllocator = errors.Normalize("get local allocator failed, %s", errors.RFCCodeText("PD:tso:ErrGetLocalAllocator")) - ErrSyncMaxTS = errors.Normalize("sync max ts failed, %s", errors.RFCCodeText("PD:tso:ErrSyncMaxTS")) - ErrResetUserTimestamp = errors.Normalize("reset user timestamp failed, %s", errors.RFCCodeText("PD:tso:ErrResetUserTimestamp")) - ErrGenerateTimestamp = errors.Normalize("generate timestamp failed, %s", errors.RFCCodeText("PD:tso:ErrGenerateTimestamp")) - ErrLogicOverflow = errors.Normalize("logic part overflow", errors.RFCCodeText("PD:tso:ErrLogicOverflow")) - ErrProxyTSOTimeout = errors.Normalize("proxy tso timeout", errors.RFCCodeText("PD:tso:ErrProxyTSOTimeout")) + ErrSetLocalTSOConfig = errors.Normalize("set local tso config failed, %s", errors.RFCCodeText("PD:tso:ErrSetLocalTSOConfig")) + ErrGetAllocator = errors.Normalize("get allocator failed, %s", errors.RFCCodeText("PD:tso:ErrGetAllocator")) + ErrGetLocalAllocator = errors.Normalize("get local allocator failed, %s", errors.RFCCodeText("PD:tso:ErrGetLocalAllocator")) + ErrSyncMaxTS = errors.Normalize("sync max ts failed, %s", errors.RFCCodeText("PD:tso:ErrSyncMaxTS")) + ErrResetUserTimestamp = errors.Normalize("reset user timestamp failed, %s", errors.RFCCodeText("PD:tso:ErrResetUserTimestamp")) + ErrGenerateTimestamp = errors.Normalize("generate timestamp failed, %s", errors.RFCCodeText("PD:tso:ErrGenerateTimestamp")) + ErrLogicOverflow = errors.Normalize("logic part overflow", errors.RFCCodeText("PD:tso:ErrLogicOverflow")) + ErrProxyTSOTimeout = errors.Normalize("proxy tso timeout", errors.RFCCodeText("PD:tso:ErrProxyTSOTimeout")) + ErrKeyspaceGroupIDInvalid = errors.Normalize("the keyspace group id is invalid, %s", errors.RFCCodeText("PD:tso:ErrKeyspaceGroupIDInvalid")) + ErrGetAllocatorManager = errors.Normalize("get allocator manager failed, %s", errors.RFCCodeText("PD:tso:ErrGetAllocatorManager")) + ErrLoadKeyspaceGroupsTimeout = errors.Normalize("load keyspace groups timeout", errors.RFCCodeText("PD:tso:ErrLoadKeyspaceGroupsTimeout")) + ErrLoadKeyspaceGroupsTerminated = errors.Normalize("load keyspace groups terminated", errors.RFCCodeText("PD:tso:ErrLoadKeyspaceGroupsTerminated")) + ErrLoadKeyspaceGroupsRetryExhausted = errors.Normalize("load keyspace groups retry exhausted, %s", errors.RFCCodeText("PD:tso:ErrLoadKeyspaceGroupsRetryExhausted")) + ErrKeyspaceGroupNotInitialized = errors.Normalize("the keyspace group %d isn't initialized", errors.RFCCodeText("PD:tso:ErrKeyspaceGroupNotInitialized")) + ErrKeyspaceNotAssigned = errors.Normalize("the keyspace %d isn't assigned to any keyspace group", errors.RFCCodeText("PD:tso:ErrKeyspaceNotAssigned")) + ErrGetMinTS = errors.Normalize("get min ts failed, %s", errors.RFCCodeText("PD:tso:ErrGetMinTS")) ) // member errors var ( ErrEtcdLeaderNotFound = errors.Normalize("etcd leader not found", errors.RFCCodeText("PD:member:ErrEtcdLeaderNotFound")) ErrMarshalLeader = errors.Normalize("marshal leader failed", errors.RFCCodeText("PD:member:ErrMarshalLeader")) + ErrCheckCampaign = errors.Normalize("check campaign failed", errors.RFCCodeText("PD:member:ErrCheckCampaign")) ) // core errors @@ -67,7 +84,7 @@ var ( ErrClientCreateTSOStream = errors.Normalize("create TSO stream failed, %s", errors.RFCCodeText("PD:client:ErrClientCreateTSOStream")) ErrClientGetTSOTimeout = errors.Normalize("get TSO timeout", errors.RFCCodeText("PD:client:ErrClientGetTSOTimeout")) ErrClientGetTSO = errors.Normalize("get TSO failed, %v", errors.RFCCodeText("PD:client:ErrClientGetTSO")) - ErrClientGetLeader = errors.Normalize("get leader from %v error", errors.RFCCodeText("PD:client:ErrClientGetLeader")) + ErrClientGetLeader = errors.Normalize("get leader failed, %v", errors.RFCCodeText("PD:client:ErrClientGetLeader")) ErrClientGetMember = errors.Normalize("get member failed", errors.RFCCodeText("PD:client:ErrClientGetMember")) ) diff --git a/server/keyspace/keyspace.go b/pkg/keyspace/keyspace.go similarity index 61% rename from server/keyspace/keyspace.go rename to pkg/keyspace/keyspace.go index 4343ae038ec..c81c1cbf86c 100644 --- a/server/keyspace/keyspace.go +++ b/pkg/keyspace/keyspace.go @@ -17,6 +17,7 @@ package keyspace import ( "bytes" "context" + "strconv" "time" "github.com/pingcap/errors" @@ -24,12 +25,12 @@ import ( "github.com/pingcap/kvproto/pkg/keyspacepb" "github.com/pingcap/log" "github.com/tikv/pd/pkg/id" + "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/slice" "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/pkg/storage/kv" "github.com/tikv/pd/pkg/utils/syncutil" - "github.com/tikv/pd/server/cluster" - "github.com/tikv/pd/server/config" + "github.com/tikv/pd/server/schedule" "github.com/tikv/pd/server/schedule/labeler" "go.uber.org/zap" ) @@ -40,16 +41,26 @@ const ( AllocStep = uint64(100) // AllocLabel is used to label keyspace idAllocator's metrics. AllocLabel = "keyspace-idAlloc" - // DefaultKeyspaceName is the name reserved for default keyspace. - DefaultKeyspaceName = "DEFAULT" - // DefaultKeyspaceID is the id of default keyspace. - DefaultKeyspaceID = uint32(0) // regionLabelIDPrefix is used to prefix the keyspace region label. regionLabelIDPrefix = "keyspaces/" // regionLabelKey is the key for keyspace id in keyspace region label. regionLabelKey = "id" + // UserKindKey is the key for user kind in keyspace config. + UserKindKey = "user_kind" + // TSOKeyspaceGroupIDKey is the key for tso keyspace group id in keyspace config. + TSOKeyspaceGroupIDKey = "tso_keyspace_group_id" + // keyspacePatrolBatchSize is the batch size for keyspace assignment patrol. + keyspacePatrolBatchSize = 256 ) +// Config is the interface for keyspace config. +type Config interface { + GetPreAlloc() []string + ToWaitRegionSplit() bool + GetWaitRegionSplitTimeout() time.Duration + GetCheckRegionSplitInterval() time.Duration +} + // Manager manages keyspace related data. // It validates requests and provides concurrency control. type Manager struct { @@ -60,11 +71,15 @@ type Manager struct { // store is the storage for keyspace related information. store endpoint.KeyspaceStorage // rc is the raft cluster of the server. - rc *cluster.RaftCluster + cluster schedule.Cluster // ctx is the context of the manager, to be used in transaction. ctx context.Context // config is the configurations of the manager. - config config.KeyspaceConfig + config Config + // kgm is the keyspace group manager of the server. + kgm *GroupManager + // nextPatrolStartID is the next start id of keyspace assignment patrol. + nextPatrolStartID uint32 } // CreateKeyspaceRequest represents necessary arguments to create a keyspace. @@ -73,68 +88,90 @@ type CreateKeyspaceRequest struct { // Using an existing name will result in error. Name string Config map[string]string - // Now is the timestamp used to record creation time. - Now int64 + // CreateTime is the timestamp used to record creation time. + CreateTime int64 // IsPreAlloc indicates whether the keyspace is pre-allocated when the cluster starts. IsPreAlloc bool } // NewKeyspaceManager creates a Manager of keyspace related data. -func NewKeyspaceManager(store endpoint.KeyspaceStorage, - rc *cluster.RaftCluster, +func NewKeyspaceManager( + ctx context.Context, + store endpoint.KeyspaceStorage, + cluster schedule.Cluster, idAllocator id.Allocator, - config config.KeyspaceConfig, + config Config, + kgm *GroupManager, ) *Manager { return &Manager{ - metaLock: syncutil.NewLockGroup(syncutil.WithHash(keyspaceIDHash)), - idAllocator: idAllocator, - store: store, - rc: rc, - ctx: context.TODO(), - config: config, + metaLock: syncutil.NewLockGroup(syncutil.WithHash(keyspaceIDHash)), + idAllocator: idAllocator, + store: store, + cluster: cluster, + ctx: ctx, + config: config, + kgm: kgm, + nextPatrolStartID: utils.DefaultKeyspaceID, } } // Bootstrap saves default keyspace info. func (manager *Manager) Bootstrap() error { // Split Keyspace Region for default keyspace. - if err := manager.splitKeyspaceRegion(DefaultKeyspaceID, false); err != nil { + if err := manager.splitKeyspaceRegion(utils.DefaultKeyspaceID, false); err != nil { return err } now := time.Now().Unix() - defaultKeyspace := &keyspacepb.KeyspaceMeta{ - Id: DefaultKeyspaceID, - Name: DefaultKeyspaceName, + defaultKeyspaceMeta := &keyspacepb.KeyspaceMeta{ + Id: utils.DefaultKeyspaceID, + Name: utils.DefaultKeyspaceName, State: keyspacepb.KeyspaceState_ENABLED, CreatedAt: now, StateChangedAt: now, } - err := manager.saveNewKeyspace(defaultKeyspace) + + config, err := manager.kgm.GetKeyspaceConfigByKind(endpoint.Basic) + if err != nil { + return err + } + defaultKeyspaceMeta.Config = config + err = manager.saveNewKeyspace(defaultKeyspaceMeta) // It's possible that default keyspace already exists in the storage (e.g. PD restart/recover), // so we ignore the keyspaceExists error. if err != nil && err != ErrKeyspaceExists { return err } - + if err := manager.kgm.UpdateKeyspaceForGroup(endpoint.Basic, config[TSOKeyspaceGroupIDKey], defaultKeyspaceMeta.GetId(), opAdd); err != nil { + return err + } // Initialize pre-alloc keyspace. - preAlloc := manager.config.PreAlloc + preAlloc := manager.config.GetPreAlloc() for _, keyspaceName := range preAlloc { - _, err = manager.CreateKeyspace(&CreateKeyspaceRequest{ + config, err := manager.kgm.GetKeyspaceConfigByKind(endpoint.Basic) + if err != nil { + return err + } + req := &CreateKeyspaceRequest{ Name: keyspaceName, - Now: now, + CreateTime: now, IsPreAlloc: true, - }) + Config: config, + } + keyspace, err := manager.CreateKeyspace(req) // Ignore the keyspaceExists error for the same reason as saving default keyspace. if err != nil && err != ErrKeyspaceExists { return err } + if err := manager.kgm.UpdateKeyspaceForGroup(endpoint.Basic, config[TSOKeyspaceGroupIDKey], keyspace.GetId(), opAdd); err != nil { + return err + } } return nil } // UpdateConfig update keyspace manager's config. -func (manager *Manager) UpdateConfig(cfg *config.KeyspaceConfig) { - manager.config = *cfg +func (manager *Manager) UpdateConfig(cfg Config) { + manager.config = cfg } // CreateKeyspace create a keyspace meta with given config and save it to storage. @@ -150,32 +187,48 @@ func (manager *Manager) CreateKeyspace(request *CreateKeyspaceRequest) (*keyspac } // If the request to create a keyspace is pre-allocated when the PD starts, // there is no need to wait for the region split, because TiKV has not started. - waitRegionSplit := !request.IsPreAlloc && manager.config.WaitRegionSplit + waitRegionSplit := !request.IsPreAlloc && manager.config.ToWaitRegionSplit() // Split keyspace region. err = manager.splitKeyspaceRegion(newID, waitRegionSplit) if err != nil { return nil, err } + userKind := endpoint.StringUserKind(request.Config[UserKindKey]) + config, err := manager.kgm.GetKeyspaceConfigByKind(userKind) + if err != nil { + return nil, err + } + if len(config) != 0 { + if request.Config == nil { + request.Config = config + } else { + request.Config[TSOKeyspaceGroupIDKey] = config[TSOKeyspaceGroupIDKey] + request.Config[UserKindKey] = config[UserKindKey] + } + } // Create and save keyspace metadata. keyspace := &keyspacepb.KeyspaceMeta{ Id: newID, Name: request.Name, State: keyspacepb.KeyspaceState_ENABLED, - CreatedAt: request.Now, - StateChangedAt: request.Now, + CreatedAt: request.CreateTime, + StateChangedAt: request.CreateTime, Config: request.Config, } err = manager.saveNewKeyspace(keyspace) if err != nil { log.Warn("[keyspace] failed to create keyspace", - zap.Uint32("ID", keyspace.GetId()), + zap.Uint32("keyspace-id", keyspace.GetId()), zap.String("name", keyspace.GetName()), zap.Error(err), ) return nil, err } + if err := manager.kgm.UpdateKeyspaceForGroup(userKind, config[TSOKeyspaceGroupIDKey], keyspace.GetId(), opAdd); err != nil { + return nil, err + } log.Info("[keyspace] keyspace created", - zap.Uint32("ID", keyspace.GetId()), + zap.Uint32("keyspace-id", keyspace.GetId()), zap.String("name", keyspace.GetName()), ) return keyspace, nil @@ -221,27 +274,35 @@ func (manager *Manager) splitKeyspaceRegion(id uint32, waitRegionSplit bool) (er start := time.Now() keyspaceRule := makeLabelRule(id) - err = manager.rc.GetRegionLabeler().SetLabelRule(keyspaceRule) + cl, ok := manager.cluster.(interface{ GetRegionLabeler() *labeler.RegionLabeler }) + if !ok { + return errors.New("cluster does not support region label") + } + err = cl.GetRegionLabeler().SetLabelRule(keyspaceRule) if err != nil { log.Warn("[keyspace] failed to add region label for keyspace", - zap.Uint32("keyspaceID", id), + zap.Uint32("keyspace-id", id), zap.Error(err), ) - return + return err } defer func() { if err != nil { - manager.rc.GetRegionLabeler().DeleteLabelRule(keyspaceRule.ID) + cl.GetRegionLabeler().DeleteLabelRule(keyspaceRule.ID) } }() if waitRegionSplit { ranges := keyspaceRule.Data.([]*labeler.KeyRangeRule) + if len(ranges) < 2 { + log.Warn("[keyspace] failed to split keyspace region with insufficient range", zap.Any("label-rule", keyspaceRule)) + return ErrRegionSplitFailed + } rawLeftBound, rawRightBound := ranges[0].StartKey, ranges[0].EndKey txnLeftBound, txnRightBound := ranges[1].StartKey, ranges[1].EndKey - ticker := time.NewTicker(manager.config.CheckRegionSplitInterval.Duration) - timer := time.NewTimer(manager.config.WaitRegionSplitTimeout.Duration) + ticker := time.NewTicker(manager.config.GetCheckRegionSplitInterval()) + timer := time.NewTimer(manager.config.GetWaitRegionSplitTimeout()) defer func() { ticker.Stop() timer.Stop() @@ -249,7 +310,7 @@ func (manager *Manager) splitKeyspaceRegion(id uint32, waitRegionSplit bool) (er for { select { case <-ticker.C: - regionsInfo := manager.rc.GetBasicCluster().RegionsInfo + regionsInfo := manager.cluster.GetBasicCluster().RegionsInfo region := regionsInfo.GetRegionByKey(rawLeftBound) if region == nil || !bytes.Equal(region.GetStartKey(), rawLeftBound) { continue @@ -268,20 +329,20 @@ func (manager *Manager) splitKeyspaceRegion(id uint32, waitRegionSplit bool) (er } case <-timer.C: log.Warn("[keyspace] wait region split timeout", - zap.Uint32("keyspaceID", id), + zap.Uint32("keyspace-id", id), zap.Error(err), ) err = ErrRegionSplitTimeout return } - log.Info("[keyspace] wait reigon split successfully", zap.Uint32("keyspaceID", id)) + log.Info("[keyspace] wait region split successfully", zap.Uint32("keyspace-id", id)) break } } log.Info("[keyspace] added region label for keyspace", - zap.Uint32("keyspaceID", id), - zap.Any("LabelRule", keyspaceRule), + zap.Uint32("keyspace-id", id), + zap.Any("label-rule", keyspaceRule), zap.Duration("takes", time.Since(start)), ) return @@ -356,6 +417,7 @@ const ( // It returns error if saving failed, operation not allowed, or if keyspace not exists. func (manager *Manager) UpdateKeyspaceConfig(name string, mutations []*Mutation) (*keyspacepb.KeyspaceMeta, error) { var meta *keyspacepb.KeyspaceMeta + oldConfig := make(map[string]string) err := manager.store.RunInTxn(manager.ctx, func(txn kv.Txn) error { // First get KeyspaceID from Name. loaded, id, err := manager.store.LoadKeyspaceID(txn, name) @@ -383,6 +445,9 @@ func (manager *Manager) UpdateKeyspaceConfig(name string, mutations []*Mutation) if meta.GetConfig() == nil { meta.Config = map[string]string{} } + for k, v := range meta.GetConfig() { + oldConfig[k] = v + } // Update keyspace config according to mutations. for _, mutation := range mutations { switch mutation.Op { @@ -394,22 +459,41 @@ func (manager *Manager) UpdateKeyspaceConfig(name string, mutations []*Mutation) return errIllegalOperation } } + newConfig := meta.GetConfig() + oldUserKind := endpoint.StringUserKind(oldConfig[UserKindKey]) + newUserKind := endpoint.StringUserKind(newConfig[UserKindKey]) + oldID := oldConfig[TSOKeyspaceGroupIDKey] + newID := newConfig[TSOKeyspaceGroupIDKey] + needUpdate := oldUserKind != newUserKind || oldID != newID + if needUpdate { + if err := manager.kgm.UpdateKeyspaceGroup(oldID, newID, oldUserKind, newUserKind, meta.GetId()); err != nil { + return err + } + } // Save the updated keyspace meta. - return manager.store.SaveKeyspaceMeta(txn, meta) + if err := manager.store.SaveKeyspaceMeta(txn, meta); err != nil { + if needUpdate { + if err := manager.kgm.UpdateKeyspaceGroup(newID, oldID, newUserKind, oldUserKind, meta.GetId()); err != nil { + log.Error("failed to revert keyspace group", zap.Error(err)) + } + } + return err + } + return nil }) if err != nil { log.Warn("[keyspace] failed to update keyspace config", - zap.Uint32("ID", meta.GetId()), + zap.Uint32("keyspace-id", meta.GetId()), zap.String("name", meta.GetName()), zap.Error(err), ) return nil, err } log.Info("[keyspace] keyspace config updated", - zap.Uint32("ID", meta.GetId()), + zap.Uint32("keyspace-id", meta.GetId()), zap.String("name", meta.GetName()), - zap.Any("new config", meta.GetConfig()), + zap.Any("new-config", meta.GetConfig()), ) return meta, nil } @@ -418,7 +502,7 @@ func (manager *Manager) UpdateKeyspaceConfig(name string, mutations []*Mutation) // It returns error if saving failed, operation not allowed, or if keyspace not exists. func (manager *Manager) UpdateKeyspaceState(name string, newState keyspacepb.KeyspaceState, now int64) (*keyspacepb.KeyspaceMeta, error) { // Changing the state of default keyspace is not allowed. - if name == DefaultKeyspaceName { + if name == utils.DefaultKeyspaceName { log.Warn("[keyspace] failed to update keyspace config", zap.Error(errModifyDefault), ) @@ -452,7 +536,7 @@ func (manager *Manager) UpdateKeyspaceState(name string, newState keyspacepb.Key }) if err != nil { log.Warn("[keyspace] failed to update keyspace config", - zap.Uint32("ID", meta.GetId()), + zap.Uint32("keyspace-id", meta.GetId()), zap.String("name", meta.GetName()), zap.Error(err), ) @@ -460,8 +544,8 @@ func (manager *Manager) UpdateKeyspaceState(name string, newState keyspacepb.Key } log.Info("[keyspace] keyspace state updated", zap.Uint32("ID", meta.GetId()), - zap.String("name", meta.GetName()), - zap.String("new state", newState.String()), + zap.String("keyspace-id", meta.GetName()), + zap.String("new-state", newState.String()), ) return meta, nil } @@ -470,7 +554,7 @@ func (manager *Manager) UpdateKeyspaceState(name string, newState keyspacepb.Key // It returns error if saving failed, operation not allowed, or if keyspace not exists. func (manager *Manager) UpdateKeyspaceStateByID(id uint32, newState keyspacepb.KeyspaceState, now int64) (*keyspacepb.KeyspaceMeta, error) { // Changing the state of default keyspace is not allowed. - if id == DefaultKeyspaceID { + if id == utils.DefaultKeyspaceID { log.Warn("[keyspace] failed to update keyspace config", zap.Error(errModifyDefault), ) @@ -497,16 +581,16 @@ func (manager *Manager) UpdateKeyspaceStateByID(id uint32, newState keyspacepb.K }) if err != nil { log.Warn("[keyspace] failed to update keyspace config", - zap.Uint32("ID", meta.GetId()), + zap.Uint32("keyspace-id", meta.GetId()), zap.String("name", meta.GetName()), zap.Error(err), ) return nil, err } log.Info("[keyspace] keyspace state updated", - zap.Uint32("ID", meta.GetId()), + zap.Uint32("keyspace-id", meta.GetId()), zap.String("name", meta.GetName()), - zap.String("new state", newState.String()), + zap.String("new-state", newState.String()), ) return meta, nil } @@ -533,7 +617,18 @@ func (manager *Manager) LoadRangeKeyspace(startID uint32, limit int) ([]*keyspac if startID > spaceIDMax { return nil, errors.Errorf("startID of the scan %d exceeds spaceID Max %d", startID, spaceIDMax) } - return manager.store.LoadRangeKeyspace(startID, limit) + var ( + keyspaces []*keyspacepb.KeyspaceMeta + err error + ) + err = manager.store.RunInTxn(manager.ctx, func(txn kv.Txn) error { + keyspaces, err = manager.store.LoadRangeKeyspace(txn, startID, limit) + return err + }) + if err != nil { + return nil, err + } + return keyspaces, nil } // allocID allocate a new keyspace id. @@ -548,3 +643,99 @@ func (manager *Manager) allocID() (uint32, error) { } return id32, nil } + +// PatrolKeyspaceAssignment is used to patrol all keyspaces and assign them to the keyspace groups. +func (manager *Manager) PatrolKeyspaceAssignment() error { + var ( + // The current start ID of the patrol, used for logging. + currentStartID = manager.nextPatrolStartID + // The next start ID of the patrol, used for the next patrol. + nextStartID = currentStartID + moreToPatrol = true + err error + ) + for moreToPatrol { + err = manager.store.RunInTxn(manager.ctx, func(txn kv.Txn) error { + defaultKeyspaceGroup, err := manager.kgm.store.LoadKeyspaceGroup(txn, utils.DefaultKeyspaceGroupID) + if err != nil { + return err + } + if defaultKeyspaceGroup == nil { + return errors.Errorf("default keyspace group %d not found", utils.DefaultKeyspaceGroupID) + } + if defaultKeyspaceGroup.IsSplitting() { + return ErrKeyspaceGroupInSplit + } + keyspaces, err := manager.store.LoadRangeKeyspace(txn, manager.nextPatrolStartID, keyspacePatrolBatchSize) + if err != nil { + return err + } + keyspaceNum := len(keyspaces) + // If there are more than one keyspace, update the current and next start IDs. + if keyspaceNum > 0 { + currentStartID = keyspaces[0].GetId() + nextStartID = keyspaces[keyspaceNum-1].GetId() + 1 + } + // If there are less than `keyspacePatrolBatchSize` keyspaces, + // we have reached the end of the keyspace list. + moreToPatrol = keyspaceNum == keyspacePatrolBatchSize + var ( + assigned = false + keyspaceIDsToUnlock = make([]uint32, 0, keyspaceNum) + ) + defer func() { + for _, id := range keyspaceIDsToUnlock { + manager.metaLock.Unlock(id) + } + }() + for _, ks := range keyspaces { + if ks == nil { + continue + } + manager.metaLock.Lock(ks.Id) + if ks.Config == nil { + ks.Config = make(map[string]string, 1) + } else if _, ok := ks.Config[TSOKeyspaceGroupIDKey]; ok { + // If the keyspace already has a group ID, skip it. + manager.metaLock.Unlock(ks.Id) + continue + } + // Unlock the keyspace meta lock after the whole txn. + keyspaceIDsToUnlock = append(keyspaceIDsToUnlock, ks.Id) + // If the keyspace doesn't have a group ID, assign it to the default keyspace group. + if !slice.Contains(defaultKeyspaceGroup.Keyspaces, ks.Id) { + defaultKeyspaceGroup.Keyspaces = append(defaultKeyspaceGroup.Keyspaces, ks.Id) + // Only save the keyspace group meta if any keyspace is assigned to it. + assigned = true + } + ks.Config[TSOKeyspaceGroupIDKey] = strconv.FormatUint(uint64(utils.DefaultKeyspaceGroupID), 10) + err = manager.store.SaveKeyspaceMeta(txn, ks) + if err != nil { + log.Error("[keyspace] failed to save keyspace meta during patrol", + zap.Int("batch-size", keyspacePatrolBatchSize), + zap.Uint32("current-start-id", currentStartID), + zap.Uint32("next-start-id", nextStartID), + zap.Uint32("keyspace-id", ks.Id), zap.Error(err)) + return err + } + } + if assigned { + err = manager.kgm.store.SaveKeyspaceGroup(txn, defaultKeyspaceGroup) + if err != nil { + log.Error("[keyspace] failed to save default keyspace group meta during patrol", + zap.Int("batch-size", keyspacePatrolBatchSize), + zap.Uint32("current-start-id", currentStartID), + zap.Uint32("next-start-id", nextStartID), zap.Error(err)) + return err + } + } + return nil + }) + if err != nil { + return err + } + // If all keyspaces in the current batch are assigned, update the next start ID. + manager.nextPatrolStartID = nextStartID + } + return nil +} diff --git a/server/keyspace/keyspace_test.go b/pkg/keyspace/keyspace_test.go similarity index 71% rename from server/keyspace/keyspace_test.go rename to pkg/keyspace/keyspace_test.go index 2eab01703e2..19e7d97c9d9 100644 --- a/server/keyspace/keyspace_test.go +++ b/pkg/keyspace/keyspace_test.go @@ -15,6 +15,7 @@ package keyspace import ( + "context" "fmt" "math" "strconv" @@ -26,10 +27,11 @@ import ( "github.com/pingcap/kvproto/pkg/keyspacepb" "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" + "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/mock/mockid" "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/pkg/storage/kv" - "github.com/tikv/pd/server/config" + "github.com/tikv/pd/pkg/utils/typeutil" ) const ( @@ -40,6 +42,8 @@ const ( type keyspaceTestSuite struct { suite.Suite + ctx context.Context + cancel context.CancelFunc manager *Manager } @@ -47,18 +51,49 @@ func TestKeyspaceTestSuite(t *testing.T) { suite.Run(t, new(keyspaceTestSuite)) } +type mockConfig struct { + PreAlloc []string + WaitRegionSplit bool + WaitRegionSplitTimeout typeutil.Duration + CheckRegionSplitInterval typeutil.Duration +} + +func (m *mockConfig) GetPreAlloc() []string { + return m.PreAlloc +} + +func (m *mockConfig) ToWaitRegionSplit() bool { + return m.WaitRegionSplit +} + +func (m *mockConfig) GetWaitRegionSplitTimeout() time.Duration { + return m.WaitRegionSplitTimeout.Duration +} + +func (m *mockConfig) GetCheckRegionSplitInterval() time.Duration { + return m.CheckRegionSplitInterval.Duration +} + func (suite *keyspaceTestSuite) SetupTest() { + suite.ctx, suite.cancel = context.WithCancel(context.Background()) store := endpoint.NewStorageEndpoint(kv.NewMemoryKV(), nil) allocator := mockid.NewIDAllocator() - suite.manager = NewKeyspaceManager(store, nil, allocator, config.KeyspaceConfig{}) + kgm := NewKeyspaceGroupManager(suite.ctx, store, nil, 0) + suite.manager = NewKeyspaceManager(suite.ctx, store, nil, allocator, &mockConfig{}, kgm) + suite.NoError(kgm.Bootstrap()) suite.NoError(suite.manager.Bootstrap()) } +func (suite *keyspaceTestSuite) TearDownTest() { + suite.cancel() +} + func (suite *keyspaceTestSuite) SetupSuite() { - suite.NoError(failpoint.Enable("github.com/tikv/pd/server/keyspace/skipSplitRegion", "return(true)")) + suite.NoError(failpoint.Enable("github.com/tikv/pd/pkg/keyspace/skipSplitRegion", "return(true)")) } + func (suite *keyspaceTestSuite) TearDownSuite() { - suite.NoError(failpoint.Disable("github.com/tikv/pd/server/keyspace/skipSplitRegion")) + suite.NoError(failpoint.Disable("github.com/tikv/pd/pkg/keyspace/skipSplitRegion")) } func makeCreateKeyspaceRequests(count int) []*CreateKeyspaceRequest { @@ -66,13 +101,13 @@ func makeCreateKeyspaceRequests(count int) []*CreateKeyspaceRequest { requests := make([]*CreateKeyspaceRequest, count) for i := 0; i < count; i++ { requests[i] = &CreateKeyspaceRequest{ - Name: fmt.Sprintf("test_keyspace%d", i), + Name: fmt.Sprintf("test_keyspace_%d", i), Config: map[string]string{ testConfig1: "100", testConfig2: "200", }, - Now: now, - IsPreAlloc: true, + CreateTime: now, + IsPreAlloc: true, // skip wait region split } } return requests @@ -148,8 +183,11 @@ func (suite *keyspaceTestSuite) TestUpdateKeyspaceConfig() { re.Error(err) } // Changing config of DEFAULT keyspace is allowed. - updated, err := manager.UpdateKeyspaceConfig(DefaultKeyspaceName, mutations) + updated, err := manager.UpdateKeyspaceConfig(utils.DefaultKeyspaceName, mutations) re.NoError(err) + // remove auto filled fields + delete(updated.Config, TSOKeyspaceGroupIDKey) + delete(updated.Config, UserKindKey) checkMutations(re, nil, updated.Config, mutations) } @@ -185,7 +223,7 @@ func (suite *keyspaceTestSuite) TestUpdateKeyspaceState() { _, err = manager.UpdateKeyspaceState(createRequest.Name, keyspacepb.KeyspaceState_ENABLED, newTime) re.Error(err) // Changing state of DEFAULT keyspace is not allowed. - _, err = manager.UpdateKeyspaceState(DefaultKeyspaceName, keyspacepb.KeyspaceState_DISABLED, newTime) + _, err = manager.UpdateKeyspaceState(utils.DefaultKeyspaceName, keyspacepb.KeyspaceState_DISABLED, newTime) re.Error(err) } } @@ -294,8 +332,8 @@ func (suite *keyspaceTestSuite) TestUpdateMultipleKeyspace() { // checkCreateRequest verifies a keyspace meta matches a create request. func checkCreateRequest(re *require.Assertions, request *CreateKeyspaceRequest, meta *keyspacepb.KeyspaceMeta) { re.Equal(request.Name, meta.GetName()) - re.Equal(request.Now, meta.GetCreatedAt()) - re.Equal(request.Now, meta.GetStateChangedAt()) + re.Equal(request.CreateTime, meta.GetCreatedAt()) + re.Equal(request.CreateTime, meta.GetStateChangedAt()) re.Equal(keyspacepb.KeyspaceState_ENABLED, meta.GetState()) re.Equal(request.Config, meta.GetConfig()) } @@ -336,3 +374,63 @@ func updateKeyspaceConfig(re *require.Assertions, manager *Manager, name string, oldMeta = updatedMeta } } + +func (suite *keyspaceTestSuite) TestPatrolKeyspaceAssignment() { + re := suite.Require() + // Create a keyspace without any keyspace group. + now := time.Now().Unix() + err := suite.manager.saveNewKeyspace(&keyspacepb.KeyspaceMeta{ + Id: 111, + Name: "111", + State: keyspacepb.KeyspaceState_ENABLED, + CreatedAt: now, + StateChangedAt: now, + }) + re.NoError(err) + // Check if the keyspace is not attached to the default group. + defaultKeyspaceGroup, err := suite.manager.kgm.GetKeyspaceGroupByID(utils.DefaultKeyspaceGroupID) + re.NoError(err) + re.NotNil(defaultKeyspaceGroup) + re.NotContains(defaultKeyspaceGroup.Keyspaces, uint32(111)) + // Patrol the keyspace assignment. + err = suite.manager.PatrolKeyspaceAssignment() + re.NoError(err) + // Check if the keyspace is attached to the default group. + defaultKeyspaceGroup, err = suite.manager.kgm.GetKeyspaceGroupByID(utils.DefaultKeyspaceGroupID) + re.NoError(err) + re.NotNil(defaultKeyspaceGroup) + re.Contains(defaultKeyspaceGroup.Keyspaces, uint32(111)) +} + +func (suite *keyspaceTestSuite) TestPatrolKeyspaceAssignmentInBatch() { + re := suite.Require() + // Create some keyspaces without any keyspace group. + for i := 1; i < keyspacePatrolBatchSize*2+1; i++ { + now := time.Now().Unix() + err := suite.manager.saveNewKeyspace(&keyspacepb.KeyspaceMeta{ + Id: uint32(i), + Name: strconv.Itoa(i), + State: keyspacepb.KeyspaceState_ENABLED, + CreatedAt: now, + StateChangedAt: now, + }) + re.NoError(err) + } + // Check if all the keyspaces are not attached to the default group. + defaultKeyspaceGroup, err := suite.manager.kgm.GetKeyspaceGroupByID(utils.DefaultKeyspaceGroupID) + re.NoError(err) + re.NotNil(defaultKeyspaceGroup) + for i := 1; i < keyspacePatrolBatchSize*2+1; i++ { + re.NotContains(defaultKeyspaceGroup.Keyspaces, uint32(i)) + } + // Patrol the keyspace assignment. + err = suite.manager.PatrolKeyspaceAssignment() + re.NoError(err) + // Check if all the keyspaces are attached to the default group. + defaultKeyspaceGroup, err = suite.manager.kgm.GetKeyspaceGroupByID(utils.DefaultKeyspaceGroupID) + re.NoError(err) + re.NotNil(defaultKeyspaceGroup) + for i := 1; i < keyspacePatrolBatchSize*2+1; i++ { + re.Contains(defaultKeyspaceGroup.Keyspaces, uint32(i)) + } +} diff --git a/pkg/keyspace/tso_keyspace_group.go b/pkg/keyspace/tso_keyspace_group.go new file mode 100644 index 00000000000..46810be92d5 --- /dev/null +++ b/pkg/keyspace/tso_keyspace_group.go @@ -0,0 +1,712 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package keyspace + +import ( + "context" + "encoding/json" + "strconv" + "sync" + "time" + + "github.com/pingcap/errors" + "github.com/pingcap/log" + "github.com/tikv/pd/pkg/balancer" + "github.com/tikv/pd/pkg/mcs/discovery" + "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/slice" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/storage/kv" + "github.com/tikv/pd/pkg/utils/etcdutil" + "github.com/tikv/pd/pkg/utils/logutil" + "go.etcd.io/etcd/clientv3" + "go.etcd.io/etcd/mvcc/mvccpb" + "go.uber.org/zap" +) + +const ( + defaultBalancerPolicy = balancer.PolicyRoundRobin + allocNodesToKeyspaceGroupsInterval = 1 * time.Second + allocNodesTimeout = 1 * time.Second + allocNodesInterval = 10 * time.Millisecond +) + +const ( + opAdd int = iota + opDelete +) + +// GroupManager is the manager of keyspace group related data. +type GroupManager struct { + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup + + sync.RWMutex + // groups is the cache of keyspace group related information. + // user kind -> keyspace group + groups map[endpoint.UserKind]*indexedHeap + + // store is the storage for keyspace group related information. + store endpoint.KeyspaceGroupStorage + + // nodeBalancer is the balancer for tso nodes. + // TODO: add user kind with different balancer when we ensure where the correspondence between tso node and user kind will be found + nodesBalancer balancer.Balancer[string] + // serviceRegistryMap stores the mapping from the service registry key to the service address. + // Note: it is only used in tsoNodesWatcher. + serviceRegistryMap map[string]string + // tsoNodesWatcher is the watcher for the registered tso servers. + tsoNodesWatcher *etcdutil.LoopWatcher +} + +// NewKeyspaceGroupManager creates a Manager of keyspace group related data. +func NewKeyspaceGroupManager( + ctx context.Context, + store endpoint.KeyspaceGroupStorage, + client *clientv3.Client, + clusterID uint64, +) *GroupManager { + ctx, cancel := context.WithCancel(ctx) + groups := make(map[endpoint.UserKind]*indexedHeap) + for i := 0; i < int(endpoint.UserKindCount); i++ { + groups[endpoint.UserKind(i)] = newIndexedHeap(int(utils.MaxKeyspaceGroupCountInUse)) + } + m := &GroupManager{ + ctx: ctx, + cancel: cancel, + store: store, + groups: groups, + nodesBalancer: balancer.GenByPolicy[string](defaultBalancerPolicy), + serviceRegistryMap: make(map[string]string), + } + + // If the etcd client is not nil, start the watch loop for the registered tso servers. + // The PD(TSO) Client relies on this info to discover tso servers. + if client != nil { + m.initTSONodesWatcher(client, clusterID) + m.wg.Add(2) + go m.tsoNodesWatcher.StartWatchLoop() + go m.allocNodesToAllKeyspaceGroups() + } + + return m +} + +// Bootstrap saves default keyspace group info and init group mapping in the memory. +func (m *GroupManager) Bootstrap() error { + // Force the membership restriction that the default keyspace must belong to default keyspace group. + // Have no information to specify the distribution of the default keyspace group replicas, so just + // leave the replica/member list empty. The TSO service will assign the default keyspace group replica + // to every tso node/pod by default. + defaultKeyspaceGroup := &endpoint.KeyspaceGroup{ + ID: utils.DefaultKeyspaceGroupID, + UserKind: endpoint.Basic.String(), + Keyspaces: []uint32{utils.DefaultKeyspaceID}, + } + + m.Lock() + defer m.Unlock() + + // Ignore the error if default keyspace group already exists in the storage (e.g. PD restart/recover). + err := m.saveKeyspaceGroups([]*endpoint.KeyspaceGroup{defaultKeyspaceGroup}, false) + if err != nil && err != ErrKeyspaceGroupExists { + return err + } + + // Load all the keyspace groups from the storage and add to the respective userKind groups. + groups, err := m.store.LoadKeyspaceGroups(utils.DefaultKeyspaceGroupID, 0) + if err != nil { + return err + } + for _, group := range groups { + userKind := endpoint.StringUserKind(group.UserKind) + m.groups[userKind].Put(group) + } + + return nil +} + +// Close closes the manager. +func (m *GroupManager) Close() { + m.cancel() + m.wg.Wait() +} + +func (m *GroupManager) allocNodesToAllKeyspaceGroups() { + defer logutil.LogPanic() + defer m.wg.Done() + ticker := time.NewTicker(allocNodesToKeyspaceGroupsInterval) + defer ticker.Stop() + for { + select { + case <-m.ctx.Done(): + return + case <-ticker.C: + } + countOfNodes := m.GetNodesCount() + if countOfNodes < utils.KeyspaceGroupDefaultReplicaCount { + continue + } + groups, err := m.store.LoadKeyspaceGroups(utils.DefaultKeyspaceGroupID, 0) + if err != nil { + log.Error("failed to load the all keyspace group", zap.Error(err)) + continue + } + withError := false + for _, group := range groups { + if len(group.Members) < utils.KeyspaceGroupDefaultReplicaCount { + nodes, err := m.AllocNodesForKeyspaceGroup(group.ID, utils.KeyspaceGroupDefaultReplicaCount) + if err != nil { + withError = true + log.Error("failed to alloc nodes for keyspace group", zap.Error(err)) + continue + } + group.Members = nodes + } + } + if !withError { + // all keyspace groups have equal or more than default replica count + return + } + } +} + +func (m *GroupManager) initTSONodesWatcher(client *clientv3.Client, clusterID uint64) { + tsoServiceKey := discovery.TSOPath(clusterID) + tsoServiceEndKey := clientv3.GetPrefixRangeEnd(tsoServiceKey) + "/" + + putFn := func(kv *mvccpb.KeyValue) error { + s := &discovery.ServiceRegistryEntry{} + if err := json.Unmarshal(kv.Value, s); err != nil { + log.Warn("failed to unmarshal service registry entry", + zap.String("event-kv-key", string(kv.Key)), zap.Error(err)) + return err + } + m.nodesBalancer.Put(s.ServiceAddr) + m.serviceRegistryMap[string(kv.Key)] = s.ServiceAddr + return nil + } + deleteFn := func(kv *mvccpb.KeyValue) error { + key := string(kv.Key) + if serviceAddr, ok := m.serviceRegistryMap[key]; ok { + delete(m.serviceRegistryMap, key) + m.nodesBalancer.Delete(serviceAddr) + return nil + } + return errors.Errorf("failed to find the service address for key %s", key) + } + + m.tsoNodesWatcher = etcdutil.NewLoopWatcher( + m.ctx, + &m.wg, + client, + "tso-nodes-watcher", + tsoServiceKey, + putFn, + deleteFn, + func() error { return nil }, + clientv3.WithRange(tsoServiceEndKey), + ) +} + +// CreateKeyspaceGroups creates keyspace groups. +func (m *GroupManager) CreateKeyspaceGroups(keyspaceGroups []*endpoint.KeyspaceGroup) error { + m.Lock() + defer m.Unlock() + if err := m.saveKeyspaceGroups(keyspaceGroups, false); err != nil { + return err + } + + for _, keyspaceGroup := range keyspaceGroups { + userKind := endpoint.StringUserKind(keyspaceGroup.UserKind) + m.groups[userKind].Put(keyspaceGroup) + } + + return nil +} + +// GetTSOServiceAddrs gets all TSO service addresses. +func (m *GroupManager) GetTSOServiceAddrs() []string { + if m == nil || m.nodesBalancer == nil { + return nil + } + return m.nodesBalancer.GetAll() +} + +// GetKeyspaceGroups gets keyspace groups from the start ID with limit. +// If limit is 0, it will load all keyspace groups from the start ID. +func (m *GroupManager) GetKeyspaceGroups(startID uint32, limit int) ([]*endpoint.KeyspaceGroup, error) { + return m.store.LoadKeyspaceGroups(startID, limit) +} + +// GetKeyspaceGroupByID returns the keyspace group by ID. +func (m *GroupManager) GetKeyspaceGroupByID(id uint32) (*endpoint.KeyspaceGroup, error) { + var ( + kg *endpoint.KeyspaceGroup + err error + ) + + if err := m.store.RunInTxn(m.ctx, func(txn kv.Txn) error { + kg, err = m.store.LoadKeyspaceGroup(txn, id) + if err != nil { + return err + } + return nil + }); err != nil { + return nil, err + } + return kg, nil +} + +// DeleteKeyspaceGroupByID deletes the keyspace group by ID. +func (m *GroupManager) DeleteKeyspaceGroupByID(id uint32) (*endpoint.KeyspaceGroup, error) { + var ( + kg *endpoint.KeyspaceGroup + err error + ) + + m.Lock() + defer m.Unlock() + if err := m.store.RunInTxn(m.ctx, func(txn kv.Txn) error { + kg, err = m.store.LoadKeyspaceGroup(txn, id) + if err != nil { + return err + } + if kg == nil { + return nil + } + if kg.IsSplitting() { + return ErrKeyspaceGroupInSplit + } + return m.store.DeleteKeyspaceGroup(txn, id) + }); err != nil { + return nil, err + } + + userKind := endpoint.StringUserKind(kg.UserKind) + // TODO: move out the keyspace to another group + // we don't need the keyspace group as the return value + m.groups[userKind].Remove(id) + + return kg, nil +} + +// saveKeyspaceGroups will try to save the given keyspace groups into the storage. +// If any keyspace group already exists and `overwrite` is false, it will return ErrKeyspaceGroupExists. +func (m *GroupManager) saveKeyspaceGroups(keyspaceGroups []*endpoint.KeyspaceGroup, overwrite bool) error { + return m.store.RunInTxn(m.ctx, func(txn kv.Txn) error { + for _, keyspaceGroup := range keyspaceGroups { + // Check if keyspace group has already existed. + oldKG, err := m.store.LoadKeyspaceGroup(txn, keyspaceGroup.ID) + if err != nil { + return err + } + if oldKG != nil && !overwrite { + return ErrKeyspaceGroupExists + } + if oldKG.IsSplitting() && overwrite { + return ErrKeyspaceGroupInSplit + } + newKG := &endpoint.KeyspaceGroup{ + ID: keyspaceGroup.ID, + UserKind: keyspaceGroup.UserKind, + Members: keyspaceGroup.Members, + Keyspaces: keyspaceGroup.Keyspaces, + } + if oldKG.IsSplitting() { + newKG.SplitState = &endpoint.SplitState{ + SplitSource: oldKG.SplitState.SplitSource, + } + } + err = m.store.SaveKeyspaceGroup(txn, newKG) + if err != nil { + return err + } + } + return nil + }) +} + +// GetKeyspaceConfigByKind returns the keyspace config for the given user kind. +func (m *GroupManager) GetKeyspaceConfigByKind(userKind endpoint.UserKind) (map[string]string, error) { + // when server is not in API mode, we don't need to return the keyspace config + if m == nil { + return map[string]string{}, nil + } + m.RLock() + defer m.RUnlock() + return m.getKeyspaceConfigByKindLocked(userKind) +} + +func (m *GroupManager) getKeyspaceConfigByKindLocked(userKind endpoint.UserKind) (map[string]string, error) { + groups, ok := m.groups[userKind] + if !ok { + return map[string]string{}, errors.Errorf("user kind %s not found", userKind) + } + kg := groups.Top() + if kg == nil { + return map[string]string{}, errors.Errorf("no keyspace group for user kind %s", userKind) + } + id := strconv.FormatUint(uint64(kg.ID), 10) + config := map[string]string{ + UserKindKey: userKind.String(), + TSOKeyspaceGroupIDKey: id, + } + return config, nil +} + +// UpdateKeyspaceForGroup updates the keyspace field for the keyspace group. +func (m *GroupManager) UpdateKeyspaceForGroup(userKind endpoint.UserKind, groupID string, keyspaceID uint32, mutation int) error { + // when server is not in API mode, we don't need to update the keyspace for keyspace group + if m == nil { + return nil + } + id, err := strconv.ParseUint(groupID, 10, 64) + if err != nil { + return err + } + + m.Lock() + defer m.Unlock() + return m.updateKeyspaceForGroupLocked(userKind, id, keyspaceID, mutation) +} + +func (m *GroupManager) updateKeyspaceForGroupLocked(userKind endpoint.UserKind, groupID uint64, keyspaceID uint32, mutation int) error { + kg := m.groups[userKind].Get(uint32(groupID)) + if kg == nil { + return errors.Errorf("keyspace group %d not found", groupID) + } + if kg.IsSplitting() { + return ErrKeyspaceGroupInSplit + } + + changed := false + + switch mutation { + case opAdd: + if !slice.Contains(kg.Keyspaces, keyspaceID) { + kg.Keyspaces = append(kg.Keyspaces, keyspaceID) + changed = true + } + case opDelete: + lenOfKeyspaces := len(kg.Keyspaces) + kg.Keyspaces = slice.Remove(kg.Keyspaces, keyspaceID) + if lenOfKeyspaces != len(kg.Keyspaces) { + changed = true + } + } + + if changed { + if err := m.saveKeyspaceGroups([]*endpoint.KeyspaceGroup{kg}, true); err != nil { + return err + } + + m.groups[userKind].Put(kg) + } + return nil +} + +// UpdateKeyspaceGroup updates the keyspace group. +func (m *GroupManager) UpdateKeyspaceGroup(oldGroupID, newGroupID string, oldUserKind, newUserKind endpoint.UserKind, keyspaceID uint32) error { + // when server is not in API mode, we don't need to update the keyspace group + if m == nil { + return nil + } + oldID, err := strconv.ParseUint(oldGroupID, 10, 64) + if err != nil { + return err + } + newID, err := strconv.ParseUint(newGroupID, 10, 64) + if err != nil { + return err + } + + m.Lock() + defer m.Unlock() + oldKG := m.groups[oldUserKind].Get(uint32(oldID)) + if oldKG == nil { + return errors.Errorf("keyspace group %s not found in %s group", oldGroupID, oldUserKind) + } + newKG := m.groups[newUserKind].Get(uint32(newID)) + if newKG == nil { + return errors.Errorf("keyspace group %s not found in %s group", newGroupID, newUserKind) + } + if oldKG.IsSplitting() || newKG.IsSplitting() { + return ErrKeyspaceGroupInSplit + } + + var updateOld, updateNew bool + if !slice.Contains(newKG.Keyspaces, keyspaceID) { + newKG.Keyspaces = append(newKG.Keyspaces, keyspaceID) + updateNew = true + } + + lenOfOldKeyspaces := len(oldKG.Keyspaces) + oldKG.Keyspaces = slice.Remove(oldKG.Keyspaces, keyspaceID) + if lenOfOldKeyspaces != len(oldKG.Keyspaces) { + updateOld = true + } + + if err := m.saveKeyspaceGroups([]*endpoint.KeyspaceGroup{oldKG, newKG}, true); err != nil { + return err + } + + if updateOld { + m.groups[oldUserKind].Put(oldKG) + } + + if updateNew { + m.groups[newUserKind].Put(newKG) + } + + return nil +} + +// SplitKeyspaceGroupByID splits the keyspace group by ID into a new keyspace group with the given new ID. +// And the keyspaces in the old keyspace group will be moved to the new keyspace group. +func (m *GroupManager) SplitKeyspaceGroupByID(splitSourceID, splitTargetID uint32, keyspaces []uint32) error { + var splitSourceKg, splitTargetKg *endpoint.KeyspaceGroup + m.Lock() + defer m.Unlock() + if err := m.store.RunInTxn(m.ctx, func(txn kv.Txn) (err error) { + // Load the old keyspace group first. + splitSourceKg, err = m.store.LoadKeyspaceGroup(txn, splitSourceID) + if err != nil { + return err + } + if splitSourceKg == nil { + return ErrKeyspaceGroupNotExists + } + // A keyspace group can not take part in multiple split processes. + if splitSourceKg.IsSplitting() { + return ErrKeyspaceGroupInSplit + } + // Check if the source keyspace group has enough replicas. + if len(splitSourceKg.Members) < utils.KeyspaceGroupDefaultReplicaCount { + return ErrKeyspaceGroupNotEnoughReplicas + } + // Check if the new keyspace group already exists. + splitTargetKg, err = m.store.LoadKeyspaceGroup(txn, splitTargetID) + if err != nil { + return err + } + if splitTargetKg != nil { + return ErrKeyspaceGroupExists + } + keyspaceNum := len(keyspaces) + sourceKeyspaceNum := len(splitSourceKg.Keyspaces) + // Check if the keyspaces are all in the old keyspace group. + if keyspaceNum == 0 || keyspaceNum > sourceKeyspaceNum { + return ErrKeyspaceNotInKeyspaceGroup + } + var ( + oldKeyspaceMap = make(map[uint32]struct{}, sourceKeyspaceNum) + newKeyspaceMap = make(map[uint32]struct{}, keyspaceNum) + ) + for _, keyspace := range splitSourceKg.Keyspaces { + oldKeyspaceMap[keyspace] = struct{}{} + } + for _, keyspace := range keyspaces { + if _, ok := oldKeyspaceMap[keyspace]; !ok { + return ErrKeyspaceNotInKeyspaceGroup + } + newKeyspaceMap[keyspace] = struct{}{} + } + // Get the split keyspace group for the old keyspace group. + splitKeyspaces := make([]uint32, 0, sourceKeyspaceNum-keyspaceNum) + for _, keyspace := range splitSourceKg.Keyspaces { + if _, ok := newKeyspaceMap[keyspace]; !ok { + splitKeyspaces = append(splitKeyspaces, keyspace) + } + } + // Update the old keyspace group. + splitSourceKg.Keyspaces = splitKeyspaces + splitSourceKg.SplitState = &endpoint.SplitState{ + SplitSource: splitSourceKg.ID, + } + if err = m.store.SaveKeyspaceGroup(txn, splitSourceKg); err != nil { + return err + } + splitTargetKg = &endpoint.KeyspaceGroup{ + ID: splitTargetID, + // Keep the same user kind and members as the old keyspace group. + UserKind: splitSourceKg.UserKind, + Members: splitSourceKg.Members, + Keyspaces: keyspaces, + SplitState: &endpoint.SplitState{ + SplitSource: splitSourceKg.ID, + }, + } + // Create the new split keyspace group. + return m.store.SaveKeyspaceGroup(txn, splitTargetKg) + }); err != nil { + return err + } + // Update the keyspace group cache. + m.groups[endpoint.StringUserKind(splitSourceKg.UserKind)].Put(splitSourceKg) + m.groups[endpoint.StringUserKind(splitTargetKg.UserKind)].Put(splitTargetKg) + return nil +} + +// FinishSplitKeyspaceByID finishes the split keyspace group by the split target ID. +func (m *GroupManager) FinishSplitKeyspaceByID(splitTargetID uint32) error { + var splitTargetKg, splitSourceKg *endpoint.KeyspaceGroup + m.Lock() + defer m.Unlock() + if err := m.store.RunInTxn(m.ctx, func(txn kv.Txn) (err error) { + // Load the split target keyspace group first. + splitTargetKg, err = m.store.LoadKeyspaceGroup(txn, splitTargetID) + if err != nil { + return err + } + if splitTargetKg == nil { + return ErrKeyspaceGroupNotExists + } + // Check if it's in the split state. + if !splitTargetKg.IsSplitTarget() { + return ErrKeyspaceGroupNotInSplit + } + // Load the split source keyspace group then. + splitSourceKg, err = m.store.LoadKeyspaceGroup(txn, splitTargetKg.SplitSource()) + if err != nil { + return err + } + if splitSourceKg == nil { + return ErrKeyspaceGroupNotExists + } + if !splitSourceKg.IsSplitSource() { + return ErrKeyspaceGroupNotInSplit + } + splitTargetKg.SplitState = nil + splitSourceKg.SplitState = nil + err = m.store.SaveKeyspaceGroup(txn, splitTargetKg) + if err != nil { + return err + } + err = m.store.SaveKeyspaceGroup(txn, splitSourceKg) + if err != nil { + return err + } + return nil + }); err != nil { + return err + } + // Update the keyspace group cache. + m.groups[endpoint.StringUserKind(splitTargetKg.UserKind)].Put(splitTargetKg) + m.groups[endpoint.StringUserKind(splitSourceKg.UserKind)].Put(splitSourceKg) + return nil +} + +// GetNodesCount returns the count of nodes. +func (m *GroupManager) GetNodesCount() int { + if m.nodesBalancer == nil { + return 0 + } + return m.nodesBalancer.Len() +} + +// AllocNodesForKeyspaceGroup allocates nodes for the keyspace group. +func (m *GroupManager) AllocNodesForKeyspaceGroup(id uint32, desiredReplicaCount int) ([]endpoint.KeyspaceGroupMember, error) { + m.Lock() + defer m.Unlock() + ctx, cancel := context.WithTimeout(m.ctx, allocNodesTimeout) + defer cancel() + ticker := time.NewTicker(allocNodesInterval) + defer ticker.Stop() + nodes := make([]endpoint.KeyspaceGroupMember, 0, desiredReplicaCount) + err := m.store.RunInTxn(m.ctx, func(txn kv.Txn) error { + kg, err := m.store.LoadKeyspaceGroup(txn, id) + if err != nil { + return err + } + if kg == nil { + return ErrKeyspaceGroupNotExists + } + if kg.IsSplitting() { + return ErrKeyspaceGroupInSplit + } + exists := make(map[string]struct{}) + for _, member := range kg.Members { + exists[member.Address] = struct{}{} + nodes = append(nodes, member) + } + if len(exists) >= desiredReplicaCount { + return nil + } + for len(exists) < desiredReplicaCount { + select { + case <-ctx.Done(): + return nil + case <-ticker.C: + } + countOfNodes := m.GetNodesCount() + if countOfNodes < desiredReplicaCount || countOfNodes == 0 { // double check + return ErrNoAvailableNode + } + addr := m.nodesBalancer.Next() + if addr == "" { + return ErrNoAvailableNode + } + if _, ok := exists[addr]; ok { + continue + } + exists[addr] = struct{}{} + nodes = append(nodes, endpoint.KeyspaceGroupMember{Address: addr}) + } + kg.Members = nodes + return m.store.SaveKeyspaceGroup(txn, kg) + }) + if err != nil { + return nil, err + } + log.Info("alloc nodes for keyspace group", zap.Uint32("id", id), zap.Reflect("nodes", nodes)) + return nodes, nil +} + +// SetNodesForKeyspaceGroup sets the nodes for the keyspace group. +func (m *GroupManager) SetNodesForKeyspaceGroup(id uint32, nodes []string) error { + m.Lock() + defer m.Unlock() + return m.store.RunInTxn(m.ctx, func(txn kv.Txn) error { + kg, err := m.store.LoadKeyspaceGroup(txn, id) + if err != nil { + return err + } + if kg == nil { + return ErrKeyspaceGroupNotExists + } + if kg.IsSplitting() { + return ErrKeyspaceGroupInSplit + } + members := make([]endpoint.KeyspaceGroupMember, 0, len(nodes)) + for _, node := range nodes { + members = append(members, endpoint.KeyspaceGroupMember{Address: node}) + } + kg.Members = members + return m.store.SaveKeyspaceGroup(txn, kg) + }) +} + +// IsExistNode checks if the node exists. +func (m *GroupManager) IsExistNode(addr string) bool { + nodes := m.nodesBalancer.GetAll() + for _, node := range nodes { + if node == addr { + return true + } + } + return false +} diff --git a/pkg/keyspace/tso_keyspace_group_test.go b/pkg/keyspace/tso_keyspace_group_test.go new file mode 100644 index 00000000000..e7a7fe246f7 --- /dev/null +++ b/pkg/keyspace/tso_keyspace_group_test.go @@ -0,0 +1,322 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package keyspace + +import ( + "context" + "fmt" + "testing" + "time" + + "github.com/stretchr/testify/suite" + "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/mock/mockcluster" + "github.com/tikv/pd/pkg/mock/mockid" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/storage/kv" + "github.com/tikv/pd/server/config" +) + +type keyspaceGroupTestSuite struct { + suite.Suite + ctx context.Context + cancel context.CancelFunc + kgm *GroupManager + kg *Manager +} + +func TestKeyspaceGroupTestSuite(t *testing.T) { + suite.Run(t, new(keyspaceGroupTestSuite)) +} + +func (suite *keyspaceGroupTestSuite) SetupTest() { + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + store := endpoint.NewStorageEndpoint(kv.NewMemoryKV(), nil) + suite.kgm = NewKeyspaceGroupManager(suite.ctx, store, nil, 0) + idAllocator := mockid.NewIDAllocator() + cluster := mockcluster.NewCluster(suite.ctx, config.NewTestOptions()) + suite.kg = NewKeyspaceManager(suite.ctx, store, cluster, idAllocator, &mockConfig{}, suite.kgm) + suite.NoError(suite.kgm.Bootstrap()) +} + +func (suite *keyspaceGroupTestSuite) TearDownTest() { + suite.cancel() +} + +func (suite *keyspaceGroupTestSuite) TestKeyspaceGroupOperations() { + re := suite.Require() + + keyspaceGroups := []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Standard.String(), + }, + { + ID: uint32(2), + UserKind: endpoint.Standard.String(), + Keyspaces: []uint32{111, 222, 333}, + }, + { + ID: uint32(3), + UserKind: endpoint.Standard.String(), + }, + } + err := suite.kgm.CreateKeyspaceGroups(keyspaceGroups) + re.NoError(err) + // list all keyspace groups + kgs, err := suite.kgm.GetKeyspaceGroups(uint32(0), 0) + re.NoError(err) + re.Len(kgs, 4) + // list part of keyspace groups + kgs, err = suite.kgm.GetKeyspaceGroups(uint32(1), 2) + re.NoError(err) + re.Len(kgs, 2) + // get the default keyspace group + kg, err := suite.kgm.GetKeyspaceGroupByID(0) + re.NoError(err) + re.Equal(uint32(0), kg.ID) + re.Equal(endpoint.Basic.String(), kg.UserKind) + re.False(kg.IsSplitting()) + kg, err = suite.kgm.GetKeyspaceGroupByID(3) + re.NoError(err) + re.Equal(uint32(3), kg.ID) + re.Equal(endpoint.Standard.String(), kg.UserKind) + re.False(kg.IsSplitting()) + // remove the keyspace group 3 + kg, err = suite.kgm.DeleteKeyspaceGroupByID(3) + re.NoError(err) + re.Equal(uint32(3), kg.ID) + // get non-existing keyspace group + kg, err = suite.kgm.GetKeyspaceGroupByID(3) + re.NoError(err) + re.Empty(kg) + // create an existing keyspace group + keyspaceGroups = []*endpoint.KeyspaceGroup{{ID: uint32(1), UserKind: endpoint.Standard.String()}} + err = suite.kgm.CreateKeyspaceGroups(keyspaceGroups) + re.Error(err) +} + +func (suite *keyspaceGroupTestSuite) TestKeyspaceAssignment() { + re := suite.Require() + + keyspaceGroups := []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Standard.String(), + }, + { + ID: uint32(2), + UserKind: endpoint.Standard.String(), + }, + { + ID: uint32(3), + UserKind: endpoint.Standard.String(), + }, + } + err := suite.kgm.CreateKeyspaceGroups(keyspaceGroups) + re.NoError(err) + // list all keyspace groups + kgs, err := suite.kgm.GetKeyspaceGroups(uint32(0), 0) + re.NoError(err) + re.Len(kgs, 4) + + for i := 0; i < 99; i++ { + _, err := suite.kg.CreateKeyspace(&CreateKeyspaceRequest{ + Name: fmt.Sprintf("test%d", i), + Config: map[string]string{ + UserKindKey: endpoint.Standard.String(), + }, + CreateTime: time.Now().Unix(), + }) + re.NoError(err) + } + + for i := 1; i <= 3; i++ { + kg, err := suite.kgm.GetKeyspaceGroupByID(uint32(i)) + re.NoError(err) + re.Len(kg.Keyspaces, 33) + } +} + +func (suite *keyspaceGroupTestSuite) TestUpdateKeyspace() { + re := suite.Require() + + keyspaceGroups := []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Basic.String(), + }, + { + ID: uint32(2), + UserKind: endpoint.Standard.String(), + }, + { + ID: uint32(3), + UserKind: endpoint.Enterprise.String(), + }, + } + err := suite.kgm.CreateKeyspaceGroups(keyspaceGroups) + re.NoError(err) + // list all keyspace groups + _, err = suite.kgm.GetKeyspaceGroups(uint32(0), 0) + re.NoError(err) + re.Equal(2, suite.kgm.groups[endpoint.Basic].Len()) + re.Equal(1, suite.kgm.groups[endpoint.Standard].Len()) + re.Equal(1, suite.kgm.groups[endpoint.Enterprise].Len()) + + _, err = suite.kg.CreateKeyspace(&CreateKeyspaceRequest{ + Name: "test", + Config: map[string]string{ + UserKindKey: endpoint.Standard.String(), + }, + CreateTime: time.Now().Unix(), + }) + re.NoError(err) + kg2, err := suite.kgm.GetKeyspaceGroupByID(2) + re.NoError(err) + re.Len(kg2.Keyspaces, 1) + kg3, err := suite.kgm.GetKeyspaceGroupByID(3) + re.NoError(err) + re.Len(kg3.Keyspaces, 0) + + _, err = suite.kg.UpdateKeyspaceConfig("test", []*Mutation{ + { + Op: OpPut, + Key: UserKindKey, + Value: endpoint.Enterprise.String(), + }, + { + Op: OpPut, + Key: TSOKeyspaceGroupIDKey, + Value: "2", + }, + }) + re.Error(err) + kg2, err = suite.kgm.GetKeyspaceGroupByID(2) + re.NoError(err) + re.Len(kg2.Keyspaces, 1) + kg3, err = suite.kgm.GetKeyspaceGroupByID(3) + re.NoError(err) + re.Len(kg3.Keyspaces, 0) + _, err = suite.kg.UpdateKeyspaceConfig("test", []*Mutation{ + { + Op: OpPut, + Key: UserKindKey, + Value: endpoint.Enterprise.String(), + }, + { + Op: OpPut, + Key: TSOKeyspaceGroupIDKey, + Value: "3", + }, + }) + re.NoError(err) + kg2, err = suite.kgm.GetKeyspaceGroupByID(2) + re.NoError(err) + re.Len(kg2.Keyspaces, 0) + kg3, err = suite.kgm.GetKeyspaceGroupByID(3) + re.NoError(err) + re.Len(kg3.Keyspaces, 1) +} + +func (suite *keyspaceGroupTestSuite) TestKeyspaceGroupSplit() { + re := suite.Require() + + keyspaceGroups := []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Basic.String(), + }, + { + ID: uint32(2), + UserKind: endpoint.Standard.String(), + Keyspaces: []uint32{111, 222, 333}, + Members: make([]endpoint.KeyspaceGroupMember, utils.KeyspaceGroupDefaultReplicaCount), + }, + } + err := suite.kgm.CreateKeyspaceGroups(keyspaceGroups) + re.NoError(err) + // split the keyspace group 1 to 4 + err = suite.kgm.SplitKeyspaceGroupByID(1, 4, []uint32{333}) + re.ErrorIs(err, ErrKeyspaceGroupNotEnoughReplicas) + // split the keyspace group 2 to 4 without giving any keyspace + err = suite.kgm.SplitKeyspaceGroupByID(2, 4, []uint32{}) + re.ErrorIs(err, ErrKeyspaceNotInKeyspaceGroup) + // split the keyspace group 2 to 4 + err = suite.kgm.SplitKeyspaceGroupByID(2, 4, []uint32{333}) + re.NoError(err) + kg2, err := suite.kgm.GetKeyspaceGroupByID(2) + re.NoError(err) + re.Equal(uint32(2), kg2.ID) + re.Equal([]uint32{111, 222}, kg2.Keyspaces) + re.True(kg2.IsSplitSource()) + re.Equal(kg2.ID, kg2.SplitSource()) + kg4, err := suite.kgm.GetKeyspaceGroupByID(4) + re.NoError(err) + re.Equal(uint32(4), kg4.ID) + re.Equal([]uint32{333}, kg4.Keyspaces) + re.True(kg4.IsSplitTarget()) + re.Equal(kg2.ID, kg4.SplitSource()) + re.Equal(kg2.UserKind, kg4.UserKind) + re.Equal(kg2.Members, kg4.Members) + + // finish the split of the keyspace group 2 + err = suite.kgm.FinishSplitKeyspaceByID(2) + re.ErrorIs(err, ErrKeyspaceGroupNotInSplit) + // finish the split of a non-existing keyspace group + err = suite.kgm.FinishSplitKeyspaceByID(5) + re.ErrorIs(err, ErrKeyspaceGroupNotExists) + // split the in-split keyspace group + err = suite.kgm.SplitKeyspaceGroupByID(2, 4, []uint32{333}) + re.ErrorIs(err, ErrKeyspaceGroupInSplit) + // remove the in-split keyspace group + kg2, err = suite.kgm.DeleteKeyspaceGroupByID(2) + re.Nil(kg2) + re.ErrorIs(err, ErrKeyspaceGroupInSplit) + kg4, err = suite.kgm.DeleteKeyspaceGroupByID(4) + re.Nil(kg4) + re.ErrorIs(err, ErrKeyspaceGroupInSplit) + // update the in-split keyspace group + err = suite.kg.kgm.UpdateKeyspaceForGroup(endpoint.Standard, "2", 444, opAdd) + re.ErrorIs(err, ErrKeyspaceGroupInSplit) + err = suite.kg.kgm.UpdateKeyspaceForGroup(endpoint.Standard, "4", 444, opAdd) + re.ErrorIs(err, ErrKeyspaceGroupInSplit) + + // finish the split of keyspace group 4 + err = suite.kgm.FinishSplitKeyspaceByID(4) + re.NoError(err) + kg2, err = suite.kgm.GetKeyspaceGroupByID(2) + re.NoError(err) + re.Equal(uint32(2), kg2.ID) + re.Equal([]uint32{111, 222}, kg2.Keyspaces) + re.False(kg2.IsSplitting()) + kg4, err = suite.kgm.GetKeyspaceGroupByID(4) + re.NoError(err) + re.Equal(uint32(4), kg4.ID) + re.Equal([]uint32{333}, kg4.Keyspaces) + re.False(kg4.IsSplitting()) + re.Equal(kg2.UserKind, kg4.UserKind) + re.Equal(kg2.Members, kg4.Members) + + // split a non-existing keyspace group + err = suite.kgm.SplitKeyspaceGroupByID(3, 5, nil) + re.ErrorIs(err, ErrKeyspaceGroupNotExists) + // split into an existing keyspace group + err = suite.kgm.SplitKeyspaceGroupByID(2, 4, nil) + re.ErrorIs(err, ErrKeyspaceGroupExists) + // split with the wrong keyspaces. + err = suite.kgm.SplitKeyspaceGroupByID(2, 5, []uint32{111, 222, 444}) + re.ErrorIs(err, ErrKeyspaceNotInKeyspaceGroup) +} diff --git a/server/keyspace/util.go b/pkg/keyspace/util.go similarity index 63% rename from server/keyspace/util.go rename to pkg/keyspace/util.go index 336ee4ee151..ff7d6e90cb9 100644 --- a/server/keyspace/util.go +++ b/pkg/keyspace/util.go @@ -15,6 +15,7 @@ package keyspace import ( + "container/heap" "encoding/binary" "encoding/hex" "regexp" @@ -23,6 +24,7 @@ import ( "github.com/pingcap/errors" "github.com/pingcap/kvproto/pkg/keyspacepb" "github.com/tikv/pd/pkg/codec" + "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/server/schedule/labeler" ) @@ -37,12 +39,27 @@ const ( var ( // ErrKeyspaceNotFound is used to indicate target keyspace does not exist. ErrKeyspaceNotFound = errors.New("keyspace does not exist") - // ErrKeyspaceExists indicates target keyspace already exists. - // Used when creating a new keyspace. - ErrKeyspaceExists = errors.New("keyspace already exists") // ErrRegionSplitTimeout indices to split region timeout ErrRegionSplitTimeout = errors.New("region split timeout") - + // ErrRegionSplitFailed indices to split region failed + ErrRegionSplitFailed = errors.New("region split failed") + // ErrKeyspaceExists indicates target keyspace already exists. + // It's used when creating a new keyspace. + ErrKeyspaceExists = errors.New("keyspace already exists") + // ErrKeyspaceGroupExists indicates target keyspace group already exists. + ErrKeyspaceGroupExists = errors.New("keyspace group already exists") + // ErrKeyspaceGroupNotExists is used to indicate target keyspace group does not exist. + ErrKeyspaceGroupNotExists = errors.New("keyspace group does not exist") + // ErrKeyspaceGroupInSplit is used to indicate target keyspace group is in split state. + ErrKeyspaceGroupInSplit = errors.New("keyspace group is in split state") + // ErrKeyspaceGroupNotInSplit is used to indicate target keyspace group is not in split state. + ErrKeyspaceGroupNotInSplit = errors.New("keyspace group is not in split state") + // ErrKeyspaceNotInKeyspaceGroup is used to indicate target keyspace is not in this keyspace group. + ErrKeyspaceNotInKeyspaceGroup = errors.New("keyspace is not in this keyspace group") + // ErrKeyspaceGroupNotEnoughReplicas is used to indicate not enough replicas in the keyspace group. + ErrKeyspaceGroupNotEnoughReplicas = errors.New("not enough replicas in the keyspace group") + // ErrNoAvailableNode is used to indicate no available node in the keyspace group. + ErrNoAvailableNode = errors.New("no available node") errModifyDefault = errors.New("cannot modify default keyspace's state") errIllegalOperation = errors.New("unknown operation") @@ -65,7 +82,7 @@ func validateID(id uint32) error { if id > spaceIDMax { return errors.Errorf("illegal keyspace id %d, larger than spaceID Max %d", id, spaceIDMax) } - if id == DefaultKeyspaceID { + if id == utils.DefaultKeyspaceID { return errors.Errorf("illegal keyspace id %d, collides with default keyspace id", id) } return nil @@ -82,7 +99,7 @@ func validateName(name string) error { if !isValid { return errors.Errorf("illegal keyspace name %s, should contain only alphanumerical and underline", name) } - if name == DefaultKeyspaceName { + if name == utils.DefaultKeyspaceName { return errors.Errorf("illegal keyspace name %s, collides with default keyspace name", name) } return nil @@ -165,3 +182,98 @@ func makeLabelRule(id uint32) *labeler.LabelRule { Data: makeKeyRanges(id), } } + +// indexedHeap is a heap with index. +type indexedHeap struct { + items []*endpoint.KeyspaceGroup + // keyspace group id -> position in items + index map[uint32]int +} + +func newIndexedHeap(hint int) *indexedHeap { + return &indexedHeap{ + items: make([]*endpoint.KeyspaceGroup, 0, hint), + index: map[uint32]int{}, + } +} + +// Implementing heap.Interface. +func (hp *indexedHeap) Len() int { + return len(hp.items) +} + +// Implementing heap.Interface. +func (hp *indexedHeap) Less(i, j int) bool { + // Gives the keyspace group with the least number of keyspaces first + return len(hp.items[j].Keyspaces) > len(hp.items[i].Keyspaces) +} + +// Implementing heap.Interface. +func (hp *indexedHeap) Swap(i, j int) { + lid := hp.items[i].ID + rid := hp.items[j].ID + hp.items[i], hp.items[j] = hp.items[j], hp.items[i] + hp.index[lid] = j + hp.index[rid] = i +} + +// Implementing heap.Interface. +func (hp *indexedHeap) Push(x interface{}) { + item := x.(*endpoint.KeyspaceGroup) + hp.index[item.ID] = hp.Len() + hp.items = append(hp.items, item) +} + +// Implementing heap.Interface. +func (hp *indexedHeap) Pop() interface{} { + l := hp.Len() + item := hp.items[l-1] + hp.items = hp.items[:l-1] + delete(hp.index, item.ID) + return item +} + +// Top returns the top item. +func (hp *indexedHeap) Top() *endpoint.KeyspaceGroup { + if hp.Len() <= 0 { + return nil + } + return hp.items[0] +} + +// Get returns item with the given ID. +func (hp *indexedHeap) Get(id uint32) *endpoint.KeyspaceGroup { + idx, ok := hp.index[id] + if !ok { + return nil + } + item := hp.items[idx] + return item +} + +// GetAll returns all the items. +func (hp *indexedHeap) GetAll() []*endpoint.KeyspaceGroup { + all := make([]*endpoint.KeyspaceGroup, len(hp.items)) + copy(all, hp.items) + return all +} + +// Put inserts item or updates the old item if it exists. +func (hp *indexedHeap) Put(item *endpoint.KeyspaceGroup) (isUpdate bool) { + if idx, ok := hp.index[item.ID]; ok { + hp.items[idx] = item + heap.Fix(hp, idx) + return true + } + heap.Push(hp, item) + return false +} + +// Remove deletes item by ID and returns it. +func (hp *indexedHeap) Remove(id uint32) *endpoint.KeyspaceGroup { + if idx, ok := hp.index[id]; ok { + item := heap.Remove(hp, idx) + return item.(*endpoint.KeyspaceGroup) + } + return nil +} diff --git a/server/keyspace/util_test.go b/pkg/keyspace/util_test.go similarity index 94% rename from server/keyspace/util_test.go rename to pkg/keyspace/util_test.go index 89b856ff9eb..ae55bf8e004 100644 --- a/server/keyspace/util_test.go +++ b/pkg/keyspace/util_test.go @@ -21,6 +21,7 @@ import ( "github.com/stretchr/testify/require" "github.com/tikv/pd/pkg/codec" + "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/server/schedule/labeler" ) @@ -30,7 +31,7 @@ func TestValidateID(t *testing.T) { id uint32 hasErr bool }{ - {DefaultKeyspaceID, true}, // Reserved id should result in error. + {utils.DefaultKeyspaceID, true}, // Reserved id should result in error. {100, false}, {spaceIDMax - 1, false}, {spaceIDMax, false}, @@ -48,7 +49,7 @@ func TestValidateName(t *testing.T) { name string hasErr bool }{ - {DefaultKeyspaceName, true}, // Reserved name should result in error. + {utils.DefaultKeyspaceName, true}, // Reserved name should result in error. {"keyspaceName1", false}, {"keyspace_name_1", false}, {"10", false}, diff --git a/pkg/mcs/discovery/discover.go b/pkg/mcs/discovery/discover.go index f6716e924cf..362652dbe5d 100644 --- a/pkg/mcs/discovery/discover.go +++ b/pkg/mcs/discovery/discover.go @@ -20,8 +20,8 @@ import ( ) // Discover is used to get all the service instances of the specified service name. -func Discover(cli *clientv3.Client, serviceName string) ([]string, error) { - key := discoveryPath(serviceName) + "/" +func Discover(cli *clientv3.Client, clusterID, serviceName string) ([]string, error) { + key := discoveryPath(clusterID, serviceName) + "/" endKey := clientv3.GetPrefixRangeEnd(key) + "/" withRange := clientv3.WithRange(endKey) diff --git a/pkg/mcs/discovery/discover_test.go b/pkg/mcs/discovery/discover_test.go index 32ced204bec..fed1d7844a0 100644 --- a/pkg/mcs/discovery/discover_test.go +++ b/pkg/mcs/discovery/discover_test.go @@ -41,14 +41,14 @@ func TestDiscover(t *testing.T) { re.NoError(err) <-etcd.Server.ReadyNotify() - sr1 := NewServiceRegister(context.Background(), client, "test_service", "127.0.0.1:1", "127.0.0.1:1", 1) + sr1 := NewServiceRegister(context.Background(), client, "12345", "test_service", "127.0.0.1:1", "127.0.0.1:1", 1) err = sr1.Register() re.NoError(err) - sr2 := NewServiceRegister(context.Background(), client, "test_service", "127.0.0.1:2", "127.0.0.1:2", 1) + sr2 := NewServiceRegister(context.Background(), client, "12345", "test_service", "127.0.0.1:2", "127.0.0.1:2", 1) err = sr2.Register() re.NoError(err) - endpoints, err := Discover(client, "test_service") + endpoints, err := Discover(client, "12345", "test_service") re.NoError(err) re.Len(endpoints, 2) re.Equal("127.0.0.1:1", endpoints[0]) @@ -57,7 +57,7 @@ func TestDiscover(t *testing.T) { sr1.cancel() sr2.cancel() time.Sleep(3 * time.Second) - endpoints, err = Discover(client, "test_service") + endpoints, err = Discover(client, "12345", "test_service") re.NoError(err) re.Empty(endpoints) } @@ -81,17 +81,17 @@ func TestServiceRegistryEntry(t *testing.T) { entry1 := &ServiceRegistryEntry{ServiceAddr: "127.0.0.1:1"} s1, err := entry1.Serialize() re.NoError(err) - sr1 := NewServiceRegister(context.Background(), client, "test_service", "127.0.0.1:1", s1, 1) + sr1 := NewServiceRegister(context.Background(), client, "12345", "test_service", "127.0.0.1:1", s1, 1) err = sr1.Register() re.NoError(err) entry2 := &ServiceRegistryEntry{ServiceAddr: "127.0.0.1:2"} s2, err := entry2.Serialize() re.NoError(err) - sr2 := NewServiceRegister(context.Background(), client, "test_service", "127.0.0.1:2", s2, 1) + sr2 := NewServiceRegister(context.Background(), client, "12345", "test_service", "127.0.0.1:2", s2, 1) err = sr2.Register() re.NoError(err) - endpoints, err := Discover(client, "test_service") + endpoints, err := Discover(client, "12345", "test_service") re.NoError(err) re.Len(endpoints, 2) returnedEntry1 := &ServiceRegistryEntry{} @@ -104,7 +104,7 @@ func TestServiceRegistryEntry(t *testing.T) { sr1.cancel() sr2.cancel() time.Sleep(3 * time.Second) - endpoints, err = Discover(client, "test_service") + endpoints, err = Discover(client, "12345", "test_service") re.NoError(err) re.Empty(endpoints) } diff --git a/pkg/mcs/discovery/key_path.go b/pkg/mcs/discovery/key_path.go index 171dbff736b..0e53b21c9fe 100644 --- a/pkg/mcs/discovery/key_path.go +++ b/pkg/mcs/discovery/key_path.go @@ -14,17 +14,25 @@ package discovery -import "path" +import ( + "strconv" + "strings" +) const ( - registryPrefix = "/pd/microservice" + registryPrefix = "/ms" registryKey = "registry" ) -func registryPath(serviceName, serviceAddr string) string { - return path.Join(registryPrefix, serviceName, registryKey, serviceAddr) +func registryPath(clusterID, serviceName, serviceAddr string) string { + return strings.Join([]string{registryPrefix, clusterID, serviceName, registryKey, serviceAddr}, "/") +} + +func discoveryPath(clusterID, serviceName string) string { + return strings.Join([]string{registryPrefix, clusterID, serviceName, registryKey}, "/") } -func discoveryPath(serviceName string) string { - return path.Join(registryPrefix, serviceName, registryKey) +// TSOPath returns the path to store TSO addresses. +func TSOPath(clusterID uint64) string { + return discoveryPath(strconv.FormatUint(clusterID, 10), "tso") + "/" } diff --git a/pkg/mcs/discovery/register.go b/pkg/mcs/discovery/register.go index fd99f3fcca7..617c1520b8d 100644 --- a/pkg/mcs/discovery/register.go +++ b/pkg/mcs/discovery/register.go @@ -39,9 +39,9 @@ type ServiceRegister struct { } // NewServiceRegister creates a new ServiceRegister. -func NewServiceRegister(ctx context.Context, cli *clientv3.Client, serviceName, serviceAddr, serializedValue string, ttl int64) *ServiceRegister { +func NewServiceRegister(ctx context.Context, cli *clientv3.Client, clusterID, serviceName, serviceAddr, serializedValue string, ttl int64) *ServiceRegister { cctx, cancel := context.WithCancel(ctx) - serviceKey := registryPath(serviceName, serviceAddr) + serviceKey := registryPath(clusterID, serviceName, serviceAddr) return &ServiceRegister{ ctx: cctx, cancel: cancel, diff --git a/pkg/mcs/discovery/register_test.go b/pkg/mcs/discovery/register_test.go index c8594ba0162..6d0bf8a7066 100644 --- a/pkg/mcs/discovery/register_test.go +++ b/pkg/mcs/discovery/register_test.go @@ -39,13 +39,15 @@ func TestRegister(t *testing.T) { re.NoError(err) <-etcd.Server.ReadyNotify() - sr := NewServiceRegister(context.Background(), client, "test_service", "127.0.0.1:1", "127.0.0.1:1", 10) + // with http prefix + sr := NewServiceRegister(context.Background(), client, "12345", "test_service", "http://127.0.0.1:1", "http://127.0.0.1:1", 10) re.NoError(err) err = sr.Register() re.NoError(err) + re.Equal("/ms/12345/test_service/registry/http://127.0.0.1:1", sr.key) resp, err := client.Get(context.Background(), sr.key) re.NoError(err) - re.Equal("127.0.0.1:1", string(resp.Kvs[0].Value)) + re.Equal("http://127.0.0.1:1", string(resp.Kvs[0].Value)) err = sr.Deregister() re.NoError(err) @@ -53,7 +55,7 @@ func TestRegister(t *testing.T) { re.NoError(err) re.Empty(resp.Kvs) - sr = NewServiceRegister(context.Background(), client, "test_service", "127.0.0.1:2", "127.0.0.1:2", 1) + sr = NewServiceRegister(context.Background(), client, "12345", "test_service", "127.0.0.1:2", "127.0.0.1:2", 1) re.NoError(err) err = sr.Register() re.NoError(err) diff --git a/pkg/mcs/resource_manager/server/server.go b/pkg/mcs/resource_manager/server/server.go index f53b0070386..9edca817f77 100644 --- a/pkg/mcs/resource_manager/server/server.go +++ b/pkg/mcs/resource_manager/server/server.go @@ -47,8 +47,8 @@ import ( // Server is the resource manager server, and it implements bs.Server. // nolint type Server struct { - // Server state. 0 is not serving, 1 is serving. - isServing int64 + // Server state. 0 is not running, 1 is running. + isRunning int64 ctx context.Context serverLoopWg sync.WaitGroup @@ -96,7 +96,7 @@ func (s *Server) Run() (err error) { // Close closes the server. func (s *Server) Close() { - if !atomic.CompareAndSwapInt64(&s.isServing, 1, 0) { + if !atomic.CompareAndSwapInt64(&s.isRunning, 1, 0) { // server is already closed return } @@ -141,12 +141,12 @@ func (s *Server) AddStartCallback(callbacks ...func()) { // IsServing returns whether the server is the leader, if there is embedded etcd, or the primary otherwise. func (s *Server) IsServing() bool { // TODO: implement this function with primary. - return atomic.LoadInt64(&s.isServing) == 1 + return atomic.LoadInt64(&s.isRunning) == 1 } // IsClosed checks if the server loop is closed func (s *Server) IsClosed() bool { - return s != nil && atomic.LoadInt64(&s.isServing) == 0 + return s != nil && atomic.LoadInt64(&s.isRunning) == 0 } // AddServiceReadyCallback adds callbacks when the server becomes the leader, if there is embedded etcd, or the primary otherwise. @@ -290,7 +290,7 @@ func (s *Server) startServer() error { } // Server has started. - atomic.StoreInt64(&s.isServing, 1) + atomic.StoreInt64(&s.isRunning, 1) return nil } diff --git a/pkg/mcs/tso/server/apis/v1/api.go b/pkg/mcs/tso/server/apis/v1/api.go index af269cc95da..42db95d8134 100644 --- a/pkg/mcs/tso/server/apis/v1/api.go +++ b/pkg/mcs/tso/server/apis/v1/api.go @@ -22,16 +22,22 @@ import ( "github.com/gin-contrib/gzip" "github.com/gin-gonic/gin" "github.com/joho/godotenv" + "github.com/pingcap/kvproto/pkg/tsopb" + "github.com/pingcap/log" tsoserver "github.com/tikv/pd/pkg/mcs/tso/server" "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/pkg/tso" "github.com/tikv/pd/pkg/utils/apiutil" "github.com/tikv/pd/pkg/utils/apiutil/multiservicesapi" "github.com/unrolled/render" + "go.uber.org/zap" ) -// APIPathPrefix is the prefix of the API path. -const APIPathPrefix = "/tso/api/v1" +const ( + // APIPathPrefix is the prefix of the API path. + APIPathPrefix = "/tso/api/v1" +) var ( once sync.Once @@ -46,14 +52,14 @@ var ( func init() { tsoserver.SetUpRestHandler = func(srv *tsoserver.Service) (http.Handler, apiutil.APIServiceGroup) { s := NewService(srv) - return s.handler(), apiServiceGroup + return s.apiHandlerEngine, apiServiceGroup } } // Service is the tso service. type Service struct { apiHandlerEngine *gin.Engine - baseEndpoint *gin.RouterGroup + root *gin.RouterGroup srv *tsoserver.Service rd *render.Render @@ -77,30 +83,64 @@ func NewService(srv *tsoserver.Service) *Service { apiHandlerEngine.Use(cors.Default()) apiHandlerEngine.Use(gzip.Gzip(gzip.DefaultCompression)) apiHandlerEngine.Use(func(c *gin.Context) { - c.Set("service", srv) + c.Set(multiservicesapi.ServiceContextKey, srv) c.Next() }) apiHandlerEngine.Use(multiservicesapi.ServiceRedirector()) apiHandlerEngine.GET("metrics", utils.PromHandler()) - endpoint := apiHandlerEngine.Group(APIPathPrefix) + root := apiHandlerEngine.Group(APIPathPrefix) s := &Service{ srv: srv, apiHandlerEngine: apiHandlerEngine, - baseEndpoint: endpoint, + root: root, rd: createIndentRender(), } - s.RegisterRouter() + s.RegisterAdminRouter() + s.RegisterKeyspaceGroupRouter() return s } -// RegisterRouter registers the router of the service. -func (s *Service) RegisterRouter() { +// RegisterAdminRouter registers the router of the TSO admin handler. +func (s *Service) RegisterAdminRouter() { + router := s.root.Group("admin") tsoAdminHandler := tso.NewAdminHandler(s.srv.GetHandler(), s.rd) - s.baseEndpoint.POST("/admin/reset-ts", gin.WrapF(tsoAdminHandler.ResetTS)) + router.POST("/reset-ts", gin.WrapF(tsoAdminHandler.ResetTS)) } -func (s *Service) handler() http.Handler { - return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - s.apiHandlerEngine.ServeHTTP(w, r) - }) +// RegisterKeyspaceGroupRouter registers the router of the TSO keyspace group handler. +func (s *Service) RegisterKeyspaceGroupRouter() { + router := s.root.Group("keyspace-groups") + router.GET("/members", GetKeyspaceGroupMembers) +} + +// KeyspaceGroupMember contains the keyspace group and its member information. +type KeyspaceGroupMember struct { + Group *endpoint.KeyspaceGroup + Member *tsopb.Participant + IsPrimary bool `json:"is_primary"` + PrimaryID uint64 `json:"primary_id"` +} + +// GetKeyspaceGroupMembers gets the keyspace group members that the TSO service is serving. +func GetKeyspaceGroupMembers(c *gin.Context) { + svr := c.MustGet(multiservicesapi.ServiceContextKey).(*tsoserver.Service) + kgm := svr.GetKeyspaceGroupManager() + keyspaceGroups := kgm.GetKeyspaceGroups() + members := make(map[uint32]*KeyspaceGroupMember, len(keyspaceGroups)) + for id, group := range keyspaceGroups { + am, err := kgm.GetAllocatorManager(id) + if err != nil { + log.Error("failed to get allocator manager", + zap.Uint32("keyspace-group-id", id), zap.Error(err)) + continue + } + member := am.GetMember() + members[id] = &KeyspaceGroupMember{ + Group: group, + Member: member.GetMember().(*tsopb.Participant), + IsPrimary: member.IsLeader(), + PrimaryID: member.GetLeaderID(), + } + } + c.IndentedJSON(http.StatusOK, members) } diff --git a/pkg/mcs/tso/server/config.go b/pkg/mcs/tso/server/config.go index 2e5225747a2..aedb7ebc1c0 100644 --- a/pkg/mcs/tso/server/config.go +++ b/pkg/mcs/tso/server/config.go @@ -26,6 +26,7 @@ import ( "github.com/pkg/errors" "github.com/spf13/pflag" "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/tso" "github.com/tikv/pd/pkg/utils/configutil" "github.com/tikv/pd/pkg/utils/grpcutil" "github.com/tikv/pd/pkg/utils/metricutil" @@ -46,6 +47,8 @@ const ( minTSOUpdatePhysicalInterval = 1 * time.Millisecond ) +var _ tso.ServiceConfig = (*Config)(nil) + // Config is the configuration for the TSO. type Config struct { BackendEndpoints string `toml:"backend-endpoints" json:"backend-endpoints"` @@ -99,6 +102,31 @@ func NewConfig() *Config { return &Config{} } +// GetName returns the Name +func (c *Config) GetName() string { + return c.Name +} + +// GeBackendEndpoints returns the BackendEndpoints +func (c *Config) GeBackendEndpoints() string { + return c.BackendEndpoints +} + +// GetListenAddr returns the ListenAddr +func (c *Config) GetListenAddr() string { + return c.ListenAddr +} + +// GetAdvertiseListenAddr returns the AdvertiseListenAddr +func (c *Config) GetAdvertiseListenAddr() string { + return c.AdvertiseListenAddr +} + +// GetLeaderLease returns the leader lease. +func (c *Config) GetLeaderLease() int64 { + return c.LeaderLease +} + // IsLocalTSOEnabled returns if the local TSO is enabled. func (c *Config) IsLocalTSOEnabled() bool { return c.EnableLocalTSO @@ -114,6 +142,11 @@ func (c *Config) GetTSOSaveInterval() time.Duration { return c.TSOSaveInterval.Duration } +// GetMaxResetTSGap returns the MaxResetTSGap. +func (c *Config) GetMaxResetTSGap() time.Duration { + return c.MaxResetTSGap.Duration +} + // GetTLSConfig returns the TLS config. func (c *Config) GetTLSConfig() *grpcutil.TLSConfig { return &c.Security.TLSConfig diff --git a/pkg/mcs/tso/server/config_test.go b/pkg/mcs/tso/server/config_test.go new file mode 100644 index 00000000000..9f5bc298964 --- /dev/null +++ b/pkg/mcs/tso/server/config_test.go @@ -0,0 +1,99 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "strings" + "testing" + "time" + + "github.com/BurntSushi/toml" + "github.com/stretchr/testify/require" + "github.com/tikv/pd/pkg/mcs/utils" +) + +func TestConfigBasic(t *testing.T) { + re := require.New(t) + + cfg := NewConfig() + cfg, err := GenerateConfig(cfg) + re.NoError(err) + + // Test default values. + re.True(strings.HasPrefix(cfg.GetName(), defaultName)) + re.Equal(defaultBackendEndpoints, cfg.BackendEndpoints) + re.Equal(defaultListenAddr, cfg.ListenAddr) + re.Equal(utils.DefaultLeaderLease, cfg.LeaderLease) + re.False(cfg.EnableLocalTSO) + re.True(cfg.EnableGRPCGateway) + re.Equal(defaultTSOSaveInterval, cfg.TSOSaveInterval.Duration) + re.Equal(defaultTSOUpdatePhysicalInterval, cfg.TSOUpdatePhysicalInterval.Duration) + re.Equal(defaultMaxResetTSGap, cfg.MaxResetTSGap.Duration) + + // Test setting values. + cfg.Name = "test-name" + cfg.BackendEndpoints = "test-endpoints" + cfg.ListenAddr = "test-listen-addr" + cfg.AdvertiseListenAddr = "test-advertise-listen-addr" + cfg.LeaderLease = 123 + cfg.EnableLocalTSO = true + cfg.TSOSaveInterval.Duration = time.Duration(10) * time.Second + cfg.TSOUpdatePhysicalInterval.Duration = time.Duration(100) * time.Millisecond + cfg.MaxResetTSGap.Duration = time.Duration(1) * time.Hour + + re.Equal("test-name", cfg.GetName()) + re.Equal("test-endpoints", cfg.GeBackendEndpoints()) + re.Equal("test-listen-addr", cfg.GetListenAddr()) + re.Equal("test-advertise-listen-addr", cfg.GetAdvertiseListenAddr()) + re.Equal(int64(123), cfg.GetLeaderLease()) + re.True(cfg.EnableLocalTSO) + re.Equal(time.Duration(10)*time.Second, cfg.TSOSaveInterval.Duration) + re.Equal(time.Duration(100)*time.Millisecond, cfg.TSOUpdatePhysicalInterval.Duration) + re.Equal(time.Duration(1)*time.Hour, cfg.MaxResetTSGap.Duration) +} + +func TestLoadFromConfig(t *testing.T) { + re := require.New(t) + cfgData := ` +backend-endpoints = "test-endpoints" +listen-addr = "test-listen-addr" +advertise-listen-addr = "test-advertise-listen-addr" +name = "tso-test-name" +data-dir = "/var/lib/tso" +enable-grpc-gateway = false +lease = 123 +enable-local-tso = true +tso-save-interval = "10s" +tso-update-physical-interval = "100ms" +max-gap-reset-ts = "1h" +` + + cfg := NewConfig() + meta, err := toml.Decode(cfgData, &cfg) + re.NoError(err) + err = cfg.Adjust(&meta, false) + re.NoError(err) + + re.Equal("tso-test-name", cfg.GetName()) + re.Equal("test-endpoints", cfg.GeBackendEndpoints()) + re.Equal("test-listen-addr", cfg.GetListenAddr()) + re.Equal("test-advertise-listen-addr", cfg.GetAdvertiseListenAddr()) + re.Equal("/var/lib/tso", cfg.DataDir) + re.Equal(int64(123), cfg.GetLeaderLease()) + re.True(cfg.EnableLocalTSO) + re.Equal(time.Duration(10)*time.Second, cfg.TSOSaveInterval.Duration) + re.Equal(time.Duration(100)*time.Millisecond, cfg.TSOUpdatePhysicalInterval.Duration) + re.Equal(time.Duration(1)*time.Hour, cfg.MaxResetTSGap.Duration) +} diff --git a/pkg/mcs/tso/server/grpc_service.go b/pkg/mcs/tso/server/grpc_service.go index c650c4910ad..dd0a96b1cba 100644 --- a/pkg/mcs/tso/server/grpc_service.go +++ b/pkg/mcs/tso/server/grpc_service.go @@ -18,33 +18,26 @@ import ( "context" "io" "net/http" + "strings" "time" - "github.com/pingcap/kvproto/pkg/pdpb" "github.com/pingcap/kvproto/pkg/tsopb" "github.com/pingcap/log" "github.com/pkg/errors" bs "github.com/tikv/pd/pkg/basicserver" - "github.com/tikv/pd/pkg/errs" "github.com/tikv/pd/pkg/mcs/registry" "github.com/tikv/pd/pkg/utils/apiutil" "github.com/tikv/pd/pkg/utils/grpcutil" - "github.com/tikv/pd/pkg/utils/logutil" - "go.uber.org/zap" + "github.com/tikv/pd/pkg/utils/tsoutil" "google.golang.org/grpc" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) -const ( - // tso - maxMergeTSORequests = 10000 - defaultTSOProxyTimeout = 3 * time.Second -) - // gRPC errors var ( - ErrNotStarted = status.Errorf(codes.Unavailable, "server not started") + ErrNotStarted = status.Errorf(codes.Unavailable, "server not started") + ErrClusterMismatched = status.Errorf(codes.Unavailable, "cluster mismatched") ) var _ tsopb.TSOServer = (*Service)(nil) @@ -116,16 +109,20 @@ func (s *Service) Tso(stream tsopb.TSO_TsoServer) error { streamCtx := stream.Context() forwardedHost := grpcutil.GetForwardedHost(streamCtx) if !s.IsLocalRequest(forwardedHost) { + clientConn, err := s.GetDelegateClient(s.ctx, forwardedHost) + if err != nil { + return errors.WithStack(err) + } + if errCh == nil { doneCh = make(chan struct{}) defer close(doneCh) errCh = make(chan error) } - s.dispatchTSORequest(ctx, &tsoRequest{ - forwardedHost, - request, - stream, - }, forwardedHost, doneCh, errCh) + + tsoProtoFactory := s.tsoProtoFactory + tsoRequest := tsoutil.NewTSOProtoRequest(forwardedHost, clientConn, request, stream) + s.tsoDispatcher.DispatchRequest(ctx, tsoRequest, tsoProtoFactory, doneCh, errCh) continue } @@ -135,16 +132,19 @@ func (s *Service) Tso(stream tsopb.TSO_TsoServer) error { return status.Errorf(codes.Unknown, "server not started") } if request.GetHeader().GetClusterId() != s.clusterID { - return status.Errorf(codes.FailedPrecondition, "mismatch cluster id, need %d but got %d", s.clusterID, request.GetHeader().GetClusterId()) + return status.Errorf( + codes.FailedPrecondition, "mismatch cluster id, need %d but got %d", + s.clusterID, request.GetHeader().GetClusterId()) } count := request.GetCount() - ts, err := s.tsoAllocatorManager.HandleTSORequest(request.GetDcLocation(), count) + ts, keyspaceGroupBelongTo, err := s.keyspaceGroupManager.HandleTSORequest( + request.Header.KeyspaceId, request.Header.KeyspaceGroupId, request.GetDcLocation(), count) if err != nil { return status.Errorf(codes.Unknown, err.Error()) } tsoHandleDuration.Observe(time.Since(start).Seconds()) response := &tsopb.TsoResponse{ - Header: s.header(), + Header: s.header(keyspaceGroupBelongTo), Timestamp: &ts, Count: count, } @@ -154,201 +154,118 @@ func (s *Service) Tso(stream tsopb.TSO_TsoServer) error { } } -func (s *Service) header() *tsopb.ResponseHeader { - if s.clusterID == 0 { - return s.wrapErrorToHeader(tsopb.ErrorType_NOT_BOOTSTRAPPED, "cluster id is not ready") +// FindGroupByKeyspaceID returns the keyspace group that the keyspace belongs to. +func (s *Service) FindGroupByKeyspaceID( + ctx context.Context, request *tsopb.FindGroupByKeyspaceIDRequest, +) (*tsopb.FindGroupByKeyspaceIDResponse, error) { + respKeyspaceGroup := request.GetHeader().GetKeyspaceGroupId() + if errorType, err := s.validRequest(request.GetHeader()); err != nil { + return &tsopb.FindGroupByKeyspaceIDResponse{ + Header: s.wrapErrorToHeader(errorType, err.Error(), respKeyspaceGroup), + }, nil } - return &tsopb.ResponseHeader{ClusterId: s.clusterID} -} - -func (s *Service) wrapErrorToHeader(errorType tsopb.ErrorType, message string) *tsopb.ResponseHeader { - return s.errorHeader(&tsopb.Error{ - Type: errorType, - Message: message, - }) -} -func (s *Service) errorHeader(err *tsopb.Error) *tsopb.ResponseHeader { - return &tsopb.ResponseHeader{ - ClusterId: s.clusterID, - Error: err, + keyspaceID := request.GetKeyspaceId() + am, keyspaceGroup, keyspaceGroupID, err := s.keyspaceGroupManager.FindGroupByKeyspaceID(keyspaceID) + if err != nil { + return &tsopb.FindGroupByKeyspaceIDResponse{ + Header: s.wrapErrorToHeader(tsopb.ErrorType_UNKNOWN, err.Error(), keyspaceGroupID), + }, nil } -} - -type tsoRequest struct { - forwardedHost string - request *tsopb.TsoRequest - stream tsopb.TSO_TsoServer -} - -func (s *Service) dispatchTSORequest(ctx context.Context, request *tsoRequest, forwardedHost string, doneCh <-chan struct{}, errCh chan<- error) { - tsoRequestChInterface, loaded := s.tsoDispatcher.LoadOrStore(forwardedHost, make(chan *tsoRequest, maxMergeTSORequests)) - if !loaded { - tsDeadlineCh := make(chan deadline, 1) - go s.handleDispatcher(ctx, forwardedHost, tsoRequestChInterface.(chan *tsoRequest), tsDeadlineCh, doneCh, errCh) - go watchTSDeadline(ctx, tsDeadlineCh) + if keyspaceGroup == nil { + return &tsopb.FindGroupByKeyspaceIDResponse{ + Header: s.wrapErrorToHeader( + tsopb.ErrorType_UNKNOWN, "keyspace group not found", keyspaceGroupID), + }, nil } - tsoRequestChInterface.(chan *tsoRequest) <- request -} - -func (s *Service) handleDispatcher(ctx context.Context, forwardedHost string, tsoRequestCh <-chan *tsoRequest, tsDeadlineCh chan<- deadline, doneCh <-chan struct{}, errCh chan<- error) { - defer logutil.LogPanic() - dispatcherCtx, ctxCancel := context.WithCancel(ctx) - defer ctxCancel() - defer s.tsoDispatcher.Delete(forwardedHost) - var ( - forwardStream tsopb.TSO_TsoClient - cancel context.CancelFunc - ) - client, err := s.GetDelegateClient(ctx, forwardedHost) - if err != nil { - goto errHandling - } - log.Info("create tso forward stream", zap.String("forwarded-host", forwardedHost)) - forwardStream, cancel, err = s.CreateTsoForwardStream(client) -errHandling: - if err != nil || forwardStream == nil { - log.Error("create tso forwarding stream error", zap.String("forwarded-host", forwardedHost), errs.ZapError(errs.ErrGRPCCreateStream, err)) - select { - case <-dispatcherCtx.Done(): - return - case _, ok := <-doneCh: - if !ok { - return - } - case errCh <- err: - close(errCh) - return - } + members := make([]*tsopb.KeyspaceGroupMember, 0, len(keyspaceGroup.Members)) + for _, member := range keyspaceGroup.Members { + members = append(members, &tsopb.KeyspaceGroupMember{ + Address: member.Address, + // TODO: watch the keyspace groups' primary serving address changes + // to get the latest primary serving addresses of all keyspace groups. + IsPrimary: strings.EqualFold(member.Address, am.GetLeaderAddr()), + }) } - defer cancel() - requests := make([]*tsoRequest, maxMergeTSORequests+1) - for { - select { - case first := <-tsoRequestCh: - pendingTSOReqCount := len(tsoRequestCh) + 1 - requests[0] = first - for i := 1; i < pendingTSOReqCount; i++ { - requests[i] = <-tsoRequestCh - } - done := make(chan struct{}) - dl := deadline{ - timer: time.After(defaultTSOProxyTimeout), - done: done, - cancel: cancel, - } - select { - case tsDeadlineCh <- dl: - case <-dispatcherCtx.Done(): - return - } - err = s.processTSORequests(forwardStream, requests[:pendingTSOReqCount]) - close(done) - if err != nil { - log.Error("proxy forward tso error", zap.String("forwarded-host", forwardedHost), errs.ZapError(errs.ErrGRPCSend, err)) - select { - case <-dispatcherCtx.Done(): - return - case _, ok := <-doneCh: - if !ok { - return - } - case errCh <- err: - close(errCh) - return - } - } - case <-dispatcherCtx.Done(): - return + var splitState *tsopb.SplitState + if keyspaceGroup.SplitState != nil { + splitState = &tsopb.SplitState{ + SplitSource: keyspaceGroup.SplitState.SplitSource, } } + + return &tsopb.FindGroupByKeyspaceIDResponse{ + Header: s.header(keyspaceGroupID), + KeyspaceGroup: &tsopb.KeyspaceGroup{ + Id: keyspaceGroupID, + UserKind: keyspaceGroup.UserKind, + SplitState: splitState, + Members: members, + }, + }, nil } -func (s *Service) processTSORequests(forwardStream tsopb.TSO_TsoClient, requests []*tsoRequest) error { - start := time.Now() - // Merge the requests - count := uint32(0) - for _, request := range requests { - count += request.request.GetCount() - } - req := &tsopb.TsoRequest{ - Header: requests[0].request.GetHeader(), - Count: count, - // TODO: support Local TSO proxy forwarding. - DcLocation: requests[0].request.GetDcLocation(), +// GetMinTS gets the minimum timestamp across all keyspace groups served by the TSO server +// who receives and handles the request. +func (s *Service) GetMinTS( + ctx context.Context, request *tsopb.GetMinTSRequest, +) (*tsopb.GetMinTSResponse, error) { + respKeyspaceGroup := request.GetHeader().GetKeyspaceGroupId() + if errorType, err := s.validRequest(request.GetHeader()); err != nil { + return &tsopb.GetMinTSResponse{ + Header: s.wrapErrorToHeader(errorType, err.Error(), respKeyspaceGroup), + }, nil } - // Send to the leader stream. - if err := forwardStream.Send(req); err != nil { - return err - } - resp, err := forwardStream.Recv() + + minTS, kgAskedCount, kgTotalCount, err := s.keyspaceGroupManager.GetMinTS(request.GetDcLocation()) if err != nil { - return err + return &tsopb.GetMinTSResponse{ + Header: s.wrapErrorToHeader( + tsopb.ErrorType_UNKNOWN, err.Error(), respKeyspaceGroup), + Timestamp: &minTS, + KeyspaceGroupsServing: kgAskedCount, + KeyspaceGroupsTotal: kgTotalCount, + }, nil } - tsoProxyHandleDuration.Observe(time.Since(start).Seconds()) - tsoProxyBatchSize.Observe(float64(count)) - // Split the response - physical, logical, suffixBits := resp.GetTimestamp().GetPhysical(), resp.GetTimestamp().GetLogical(), resp.GetTimestamp().GetSuffixBits() - // `logical` is the largest ts's logical part here, we need to do the subtracting before we finish each TSO request. - // This is different from the logic of client batch, for example, if we have a largest ts whose logical part is 10, - // count is 5, then the splitting results should be 5 and 10. - firstLogical := addLogical(logical, -int64(count), suffixBits) - return s.finishTSORequest(requests, physical, firstLogical, suffixBits) + + return &tsopb.GetMinTSResponse{ + Header: s.header(respKeyspaceGroup), + Timestamp: &minTS, + KeyspaceGroupsServing: kgAskedCount, + KeyspaceGroupsTotal: kgTotalCount, + }, nil } -// Because of the suffix, we need to shift the count before we add it to the logical part. -func addLogical(logical, count int64, suffixBits uint32) int64 { - return logical + count</tso/", and the entire key - // is in the format of "/ms//tso//primary". The is 5 digits - // integer with leading zeros. - tsoSvcDiscoveryPrefixFormat = msServiceRootPath + "/%d/" + mcsutils.TSOServiceName + "/%05d" - // maxRetryTimes is the max retry times for initializing the cluster ID. - maxRetryTimes = 180 - // retryInterval is the interval to retry. - retryInterval = 1 * time.Second + // tsoSvcRootPathFormat defines the root path for all etcd paths used for different purposes. + // format: "/ms/{cluster_id}/tso". + tsoSvcRootPathFormat = msServiceRootPath + "/%d/" + mcsutils.TSOServiceName + + // maxRetryTimesWaitAPIService is the max retry times for initializing the cluster ID. + maxRetryTimesWaitAPIService = 360 + // retryIntervalWaitAPIService is the interval to retry. + // Note: the interval must be less than the timeout of tidb and tikv, which is 2s by default in tikv. + retryIntervalWaitAPIService = 500 * time.Millisecond ) var _ bs.Server = (*Server)(nil) -var _ tso.Member = (*member.Participant)(nil) +var _ tso.ElectionMember = (*member.Participant)(nil) // Server is the TSO server, and it implements bs.Server. type Server struct { diagnosticspb.DiagnosticsServer - // Server state. 0 is not serving, 1 is serving. - isServing int64 + // Server state. 0 is not running, 1 is running. + isRunning int64 // Server start timestamp startTimestamp int64 @@ -99,34 +97,35 @@ type Server struct { handler *Handler - cfg *Config - clusterID uint64 - defaultGroupRootPath string - defaultGroupStorage endpoint.TSOStorage - listenURL *url.URL - backendUrls []url.URL + cfg *Config + clusterID uint64 + listenURL *url.URL + backendUrls []url.URL - // for the primary election in the TSO cluster - participant *member.Participant // etcd client etcdClient *clientv3.Client // http client httpClient *http.Client - muxListener net.Listener - service *Service - tsoAllocatorManager *tso.AllocatorManager + muxListener net.Listener + service *Service + keyspaceGroupManager *tso.KeyspaceGroupManager // Store as map[string]*grpc.ClientConn clientConns sync.Map - // Store as map[string]chan *tsoRequest - tsoDispatcher sync.Map + // tsoDispatcher is used to dispatch the TSO requests to + // the corresponding forwarding TSO channels. + tsoDispatcher *tsoutil.TSODispatcher + // tsoProtoFactory is the abstract factory for creating tso + // related data structures defined in the tso grpc protocol + tsoProtoFactory *tsoutil.TSOProtoFactory // Callback functions for different stages // startCallbacks will be called after the server is started. startCallbacks []func() - // primaryCallbacks will be called after the server becomes the primary. - primaryCallbacks []func(context.Context) - serviceRegister *discovery.ServiceRegister + + // for service registry + serviceID *discovery.ServiceRegistryEntry + serviceRegister *discovery.ServiceRegister } // Implement the following methods defined in bs.Server @@ -163,7 +162,7 @@ func (s *Server) Run() error { skipWaitAPIServiceReady = true }) if !skipWaitAPIServiceReady { - if err := s.waitAPIServiceReady(s.ctx); err != nil { + if err := s.waitAPIServiceReady(); err != nil { return err } } @@ -175,145 +174,19 @@ func (s *Server) Run() error { if err := s.initClient(); err != nil { return err } - if err := s.startServer(); err != nil { - return err - } - - s.startServerLoop() - - return nil -} - -func (s *Server) startServerLoop() { - s.serverLoopCtx, s.serverLoopCancel = context.WithCancel(s.ctx) - s.serverLoopWg.Add(2) - go s.primaryElectionLoop() - go s.tsoAllocatorLoop() -} - -// tsoAllocatorLoop is used to run the TSO Allocator updating daemon. -func (s *Server) tsoAllocatorLoop() { - defer logutil.LogPanic() - defer s.serverLoopWg.Done() - - ctx, cancel := context.WithCancel(s.serverLoopCtx) - defer cancel() - s.tsoAllocatorManager.AllocatorDaemon(ctx) - log.Info("tso server is closed, exit allocator loop") -} - -func (s *Server) primaryElectionLoop() { - defer logutil.LogPanic() - defer s.serverLoopWg.Done() - - for { - if s.IsClosed() { - log.Info("server is closed, exit tso primary election loop") - return - } - - primary, rev, checkAgain := s.participant.CheckLeader() - if checkAgain { - continue - } - if primary != nil { - // TODO: if enable-local-tso is true, check the cluster dc-location after the primary/leader is elected - // go s.tsoAllocatorManager.ClusterDCLocationChecker() - - log.Info("start to watch the primary/leader", zap.Stringer("tso-primary", primary)) - // WatchLeader will keep looping and never return unless the primary/leader has changed. - s.participant.WatchLeader(s.serverLoopCtx, primary, rev) - log.Info("the tso primary/leader has changed, try to re-campaign a primary/leader") - } - - s.campaignLeader() - } -} - -func (s *Server) campaignLeader() { - log.Info("start to campaign the primary/leader", zap.String("campaign-tso-primary-name", s.participant.Name())) - if err := s.participant.CampaignLeader(s.cfg.LeaderLease); err != nil { - if err.Error() == errs.ErrEtcdTxnConflict.Error() { - log.Info("campaign tso primary/leader meets error due to txn conflict, another tso server may campaign successfully", - zap.String("campaign-tso-primary-name", s.participant.Name())) - } else { - log.Error("campaign tso primary/leader meets error due to etcd error", - zap.String("campaign-tso-primary-name", s.participant.Name()), - errs.ZapError(err)) - } - return - } - - // Start keepalive the leadership and enable TSO service. - // TSO service is strictly enabled/disabled by the leader lease for 2 reasons: - // 1. lease based approach is not affected by thread pause, slow runtime schedule, etc. - // 2. load region could be slow. Based on lease we can recover TSO service faster. - ctx, cancel := context.WithCancel(s.serverLoopCtx) - var resetLeaderOnce sync.Once - defer resetLeaderOnce.Do(func() { - cancel() - s.participant.ResetLeader() - }) - - // maintain the the leadership, after this, TSO can be service. - s.participant.KeepLeader(ctx) - log.Info("campaign tso primary ok", zap.String("campaign-tso-primary-name", s.participant.Name())) - - allocator, err := s.tsoAllocatorManager.GetAllocator(tso.GlobalDCLocation) - if err != nil { - log.Error("failed to get the global tso allocator", errs.ZapError(err)) - return - } - log.Info("initializing the global tso allocator") - if err := allocator.Initialize(0); err != nil { - log.Error("failed to initialize the global tso allocator", errs.ZapError(err)) - return - } - defer func() { - s.tsoAllocatorManager.ResetAllocatorGroup(tso.GlobalDCLocation) - }() - - log.Info("triggering the primary callback functions") - for _, cb := range s.primaryCallbacks { - cb(ctx) - } - - s.participant.EnableLeader() - defer resetLeaderOnce.Do(func() { - cancel() - s.participant.ResetLeader() - }) - - // TODO: if enable-local-tso is true, check the cluster dc-location after the primary/leader is elected - // go s.tsoAllocatorManager.ClusterDCLocationChecker() - log.Info("tso primary is ready to serve", zap.String("tso-primary-name", s.participant.Name())) - - leaderTicker := time.NewTicker(mcsutils.LeaderTickInterval) - defer leaderTicker.Stop() - - for { - select { - case <-leaderTicker.C: - if !s.participant.IsLeader() { - log.Info("no longer a primary/leader because lease has expired, the tso primary/leader will step down") - return - } - case <-ctx.Done(): - // Server is closed and it should return nil. - log.Info("server is closed") - return - } - } + return s.startServer() } // Close closes the server. func (s *Server) Close() { - if !atomic.CompareAndSwapInt64(&s.isServing, 1, 0) { + if !atomic.CompareAndSwapInt64(&s.isRunning, 1, 0) { // server is already closed return } log.Info("closing tso server ...") + // close tso service loops in the keyspace group manager + s.keyspaceGroupManager.Close() s.serviceRegister.Deregister() s.muxListener.Close() s.serverLoopCancel() @@ -349,23 +222,62 @@ func (s *Server) AddStartCallback(callbacks ...func()) { // IsServing implements basicserver. It returns whether the server is the leader // if there is embedded etcd, or the primary otherwise. func (s *Server) IsServing() bool { - return atomic.LoadInt64(&s.isServing) == 1 && s.participant.IsLeader() + return s.IsKeyspaceServing(mcsutils.DefaultKeyspaceID, mcsutils.DefaultKeyspaceGroupID) +} + +// IsKeyspaceServing returns whether the server is the primary of the given keyspace. +// TODO: update basicserver interface to support keyspace. +func (s *Server) IsKeyspaceServing(keyspaceID, keyspaceGroupID uint32) bool { + if atomic.LoadInt64(&s.isRunning) == 0 { + return false + } + + member, err := s.keyspaceGroupManager.GetElectionMember( + keyspaceID, keyspaceGroupID) + if err != nil { + log.Error("failed to get election member", errs.ZapError(err)) + return false + } + return member.IsLeader() } // GetLeaderListenUrls gets service endpoints from the leader in election group. // The entry at the index 0 is the primary's service endpoint. func (s *Server) GetLeaderListenUrls() []string { - return s.participant.GetLeaderListenUrls() + member, err := s.keyspaceGroupManager.GetElectionMember( + mcsutils.DefaultKeyspaceID, mcsutils.DefaultKeyspaceGroupID) + if err != nil { + log.Error("failed to get election member", errs.ZapError(err)) + return nil + } + + return member.GetLeaderListenUrls() +} + +// GetMember returns the election member of the given keyspace and keyspace group. +func (s *Server) GetMember(keyspaceID, keyspaceGroupID uint32) (tso.ElectionMember, error) { + member, err := s.keyspaceGroupManager.GetElectionMember(keyspaceID, keyspaceGroupID) + if err != nil { + return nil, err + } + return member, nil } -// GetMemberName returns the name of the member in election group. -func (s *Server) GetMemberName() string { - return s.participant.Name() +// ResignPrimary resigns the primary of the given keyspace. +func (s *Server) ResignPrimary(keyspaceID, keyspaceGroupID uint32) error { + member, err := s.keyspaceGroupManager.GetElectionMember(keyspaceID, keyspaceGroupID) + if err != nil { + return err + } + member.ResetLeader() + return nil } -// AddServiceReadyCallback implements basicserver. It adds callbacks when the server becomes the primary. +// AddServiceReadyCallback implements basicserver. +// It adds callbacks when it's ready for providing tso service. func (s *Server) AddServiceReadyCallback(callbacks ...func(context.Context)) { - s.primaryCallbacks = append(s.primaryCallbacks, callbacks...) + // Do nothing here. The primary of each keyspace group assigned to this host + // will respond to the requests accordingly. } // Implement the other methods @@ -377,17 +289,17 @@ func (s *Server) ClusterID() uint64 { // IsClosed checks if the server loop is closed func (s *Server) IsClosed() bool { - return atomic.LoadInt64(&s.isServing) == 0 + return atomic.LoadInt64(&s.isRunning) == 0 } -// GetTSOAllocatorManager returns the manager of TSO Allocator. -func (s *Server) GetTSOAllocatorManager() *tso.AllocatorManager { - return s.tsoAllocatorManager +// GetKeyspaceGroupManager returns the manager of keyspace group. +func (s *Server) GetKeyspaceGroupManager() *tso.KeyspaceGroupManager { + return s.keyspaceGroupManager } -// GetTSODispatcher gets the TSO Dispatcher -func (s *Server) GetTSODispatcher() *sync.Map { - return &s.tsoDispatcher +// GetTSOAllocatorManager returns the manager of TSO Allocator. +func (s *Server) GetTSOAllocatorManager(keyspaceGroupID uint32) (*tso.AllocatorManager, error) { + return s.keyspaceGroupManager.GetAllocatorManager(keyspaceGroupID) } // IsLocalRequest checks if the forwarded host is the current host @@ -399,16 +311,6 @@ func (s *Server) IsLocalRequest(forwardedHost string) bool { return forwardedHost == "" } -// CreateTsoForwardStream creates the forward stream -func (s *Server) CreateTsoForwardStream(client *grpc.ClientConn) (tsopb.TSO_TsoClient, context.CancelFunc, error) { - done := make(chan struct{}) - ctx, cancel := context.WithCancel(s.ctx) - go checkStream(ctx, cancel, done) - forwardStream, err := tsopb.NewTSOClient(client).Tso(ctx) - done <- struct{}{} - return forwardStream, cancel, err -} - // GetDelegateClient returns grpc client connection talking to the forwarded host func (s *Server) GetDelegateClient(ctx context.Context, forwardedHost string) (*grpc.ClientConn, error) { client, ok := s.clientConns.Load(forwardedHost) @@ -461,18 +363,6 @@ func (s *Server) SetExternalTS(externalTS uint64) error { return nil } -func checkStream(streamCtx context.Context, cancel context.CancelFunc, done chan struct{}) { - defer logutil.LogPanic() - select { - case <-done: - return - case <-time.After(3 * time.Second): - cancel() - case <-streamCtx.Done(): - } - <-done -} - // GetConfig gets the config. func (s *Server) GetConfig() *Config { return s.cfg @@ -588,29 +478,24 @@ func (s *Server) startServer() (err error) { // The independent TSO service still reuses PD version info since PD and TSO are just // different service modes provided by the same pd-server binary serverInfo.WithLabelValues(versioninfo.PDReleaseVersion, versioninfo.PDGitHash).Set(float64(time.Now().Unix())) - s.defaultGroupRootPath = path.Join(pdRootPath, strconv.FormatUint(s.clusterID, 10)) s.listenURL, err = url.Parse(s.cfg.ListenAddr) if err != nil { return err } - uniqueName := s.cfg.AdvertiseListenAddr - uniqueID := memberutil.GenerateUniqueID(uniqueName) - log.Info("joining primary election", zap.String("participant-name", uniqueName), zap.Uint64("participant-id", uniqueID)) - - s.participant = member.NewParticipant(s.etcdClient) - s.participant.InitInfo(uniqueName, uniqueID, fmt.Sprintf(tsoSvcDiscoveryPrefixFormat, s.clusterID, mcsutils.DefaultKeyspaceID), - "primary", "keyspace group primary election", s.cfg.AdvertiseListenAddr) - - s.defaultGroupStorage = endpoint.NewStorageEndpoint(kv.NewEtcdKVBase(s.GetClient(), s.defaultGroupRootPath), nil) - s.tsoAllocatorManager = tso.NewAllocatorManager( - s.participant, s.defaultGroupRootPath, s.defaultGroupStorage, s.cfg.IsLocalTSOEnabled(), - s.cfg.GetTSOSaveInterval(), s.cfg.GetTSOUpdatePhysicalInterval(), - s.cfg.GetTLSConfig(), func() time.Duration { return s.cfg.MaxResetTSGap.Duration }) - // Set up the Global TSO Allocator here, it will be initialized once this TSO participant campaigns leader successfully. - s.tsoAllocatorManager.SetUpAllocator(s.ctx, tso.GlobalDCLocation, s.participant.GetLeadership()) + // Initialize the TSO service. + s.serverLoopCtx, s.serverLoopCancel = context.WithCancel(s.ctx) + legacySvcRootPath := path.Join(pdRootPath, strconv.FormatUint(s.clusterID, 10)) + tsoSvcRootPath := fmt.Sprintf(tsoSvcRootPathFormat, s.clusterID) + s.serviceID = &discovery.ServiceRegistryEntry{ServiceAddr: s.cfg.AdvertiseListenAddr} + s.keyspaceGroupManager = tso.NewKeyspaceGroupManager( + s.serverLoopCtx, s.serviceID, s.etcdClient, s.httpClient, s.cfg.AdvertiseListenAddr, legacySvcRootPath, tsoSvcRootPath, s.cfg) + if err := s.keyspaceGroupManager.Initialize(); err != nil { + return err + } + s.tsoProtoFactory = &tsoutil.TSOProtoFactory{} s.service = &Service{Server: s} tlsConfig, err := s.cfg.Security.ToTLSConfig() @@ -636,46 +521,50 @@ func (s *Server) startServer() (err error) { } // Server has started. - entry := &discovery.ServiceRegistryEntry{ServiceAddr: s.cfg.ListenAddr} - serializedEntry, err := entry.Serialize() + serializedEntry, err := s.serviceID.Serialize() if err != nil { return err } - s.serviceRegister = discovery.NewServiceRegister(s.ctx, s.etcdClient, mcsutils.TSOServiceName, s.cfg.ListenAddr, serializedEntry, discovery.DefaultLeaseInSeconds) + s.serviceRegister = discovery.NewServiceRegister(s.ctx, s.etcdClient, strconv.FormatUint(s.clusterID, 10), + mcsutils.TSOServiceName, s.cfg.AdvertiseListenAddr, serializedEntry, discovery.DefaultLeaseInSeconds) if err := s.serviceRegister.Register(); err != nil { - log.Error("failed to regiser the service", zap.String("service-name", mcsutils.TSOServiceName), errs.ZapError(err)) + log.Error("failed to register the service", zap.String("service-name", mcsutils.TSOServiceName), errs.ZapError(err)) return err } - atomic.StoreInt64(&s.isServing, 1) + atomic.StoreInt64(&s.isRunning, 1) return nil } -func (s *Server) waitAPIServiceReady(ctx context.Context) error { - for i := 0; i < maxRetryTimes; i++ { - ready, err := s.isAPIServiceReady(ctx) - if err != nil { - log.Warn("failed to check api server ready", errs.ZapError(err)) - } - if ready { +func (s *Server) waitAPIServiceReady() error { + var ( + ready bool + err error + ) + for i := 0; i < maxRetryTimesWaitAPIService; i++ { + ready, err = s.isAPIServiceReady() + if err == nil && ready { return nil } + log.Debug("api server is not ready, retrying", errs.ZapError(err), zap.Bool("ready", ready)) select { - case <-ctx.Done(): + case <-s.ctx.Done(): return errors.New("context canceled while waiting api server ready") - case <-time.After(retryInterval): - log.Debug("api server is not ready, retrying") + case <-time.After(retryIntervalWaitAPIService): } } - return errors.Errorf("failed to wait api server ready after retrying %d times", maxRetryTimes) + if err != nil { + log.Warn("failed to check api server ready", errs.ZapError(err)) + } + return errors.Errorf("failed to wait api server ready after retrying %d times", maxRetryTimesWaitAPIService) } -func (s *Server) isAPIServiceReady(ctx context.Context) (bool, error) { +func (s *Server) isAPIServiceReady() (bool, error) { urls := strings.Split(s.cfg.BackendEndpoints, ",") if len(urls) == 0 { return false, errors.New("no backend endpoints") } - cc, err := s.GetDelegateClient(s.ctx, urls[0]) // initClient has checked the backendUrls. + cc, err := s.GetDelegateClient(s.ctx, urls[0]) if err != nil { return false, err } @@ -686,7 +575,11 @@ func (s *Server) isAPIServiceReady(ctx context.Context) (bool, error) { if clusterInfo.GetHeader().GetError() != nil { return false, errors.Errorf(clusterInfo.GetHeader().GetError().String()) } - if clusterInfo.ServiceModes[0] == pdpb.ServiceMode_API_SVC_MODE { + modes := clusterInfo.ServiceModes + if len(modes) == 0 { + return false, errors.New("no service mode") + } + if modes[0] == pdpb.ServiceMode_API_SVC_MODE { return true, nil } return false, nil diff --git a/pkg/mcs/tso/server/testutil.go b/pkg/mcs/tso/server/testutil.go index 741dd0325b7..626d1474673 100644 --- a/pkg/mcs/tso/server/testutil.go +++ b/pkg/mcs/tso/server/testutil.go @@ -15,42 +15,14 @@ package server import ( - "context" - "os" "strings" "github.com/pingcap/kvproto/pkg/tsopb" - "github.com/pingcap/log" "github.com/spf13/pflag" "github.com/stretchr/testify/require" - "github.com/tikv/pd/pkg/utils/logutil" "google.golang.org/grpc" ) -// CleanupFunc closes test pd server(s) and deletes any files left behind. -type CleanupFunc func() - -// NewTSOTestServer creates a tso server for testing. -func NewTSOTestServer(ctx context.Context, re *require.Assertions, cfg *Config) (*Server, CleanupFunc, error) { - // New zap logger - err := logutil.SetupLogger(cfg.Log, &cfg.Logger, &cfg.LogProps, cfg.Security.RedactInfoLog) - re.NoError(err) - log.ReplaceGlobals(cfg.Logger, cfg.LogProps) - // Flushing any buffered log entries - defer log.Sync() - - s := CreateServer(ctx, cfg) - if err = s.Run(); err != nil { - return nil, nil, err - } - - cleanup := func() { - s.Close() - os.RemoveAll(cfg.DataDir) - } - return s, cleanup, nil -} - // MustNewGrpcClient must create a new TSO grpc client. func MustNewGrpcClient(re *require.Assertions, addr string) (*grpc.ClientConn, tsopb.TSOClient) { conn, err := grpc.Dial(strings.TrimPrefix(addr, "http://"), grpc.WithInsecure()) diff --git a/pkg/mcs/utils/constant.go b/pkg/mcs/utils/constant.go index 30581df661b..9c9eea77732 100644 --- a/pkg/mcs/utils/constant.go +++ b/pkg/mcs/utils/constant.go @@ -31,18 +31,23 @@ const ( // DefaultDisableErrorVerbose is the default value of DisableErrorVerbose DefaultDisableErrorVerbose = true // DefaultLeaderLease is the default value of LeaderLease - DefaultLeaderLease = 3 + DefaultLeaderLease = int64(3) // LeaderTickInterval is the interval to check leader LeaderTickInterval = 50 * time.Millisecond + // DefaultKeyspaceName is the name reserved for default keyspace. + DefaultKeyspaceName = "DEFAULT" + // DefaultKeyspaceID is the default key space id. // Valid keyspace id range is [0, 0xFFFFFF](uint24max, or 16777215) - // ​0 is reserved for default keyspace with the name "DEFAULT", It's initialized when PD bootstrap and reserved for users who haven't been assigned keyspace. + // ​0 is reserved for default keyspace with the name "DEFAULT", It's initialized when PD bootstrap + // and reserved for users who haven't been assigned keyspace. DefaultKeyspaceID = uint32(0) - - // DefaultKeySpaceGroupID is the default key space group id. + // NullKeyspaceID is used for api v1 or legacy path where is keyspace agnostic. + NullKeyspaceID = uint32(0xFFFFFFFF) + // DefaultKeyspaceGroupID is the default key space group id. // We also reserved 0 for the keyspace group for the same purpose. - DefaultKeySpaceGroupID = 0 + DefaultKeyspaceGroupID = uint32(0) // APIServiceName is the name of api server. APIServiceName = "api" @@ -50,4 +55,19 @@ const ( TSOServiceName = "tso" // ResourceManagerServiceName is the name of resource manager server. ResourceManagerServiceName = "resource_manager" + + // MaxKeyspaceGroupCount is the max count of keyspace groups. keyspace group in tso + // is the sharding unit, i.e., by the definition here, the max count of the shards + // that we support is MaxKeyspaceGroupCount. The keyspace group id is in the range + // [0, 99999], which explains we use five-digits number (%05d) to render the keyspace + // group id in the storage endpoint path. + MaxKeyspaceGroupCount = uint32(100000) + // MaxKeyspaceGroupCountInUse is the max count of keyspace groups in use, which should + // never exceed MaxKeyspaceGroupCount defined above. Compared to MaxKeyspaceGroupCount, + // MaxKeyspaceGroupCountInUse is a much more reasonable value of the max count in the + // foreseen future, and the former is just for extensibility in theory. + MaxKeyspaceGroupCountInUse = uint32(4096) + + // KeyspaceGroupDefaultReplicaCount is the default replica count of keyspace group. + KeyspaceGroupDefaultReplicaCount = 2 ) diff --git a/pkg/member/election_leader.go b/pkg/member/election_leader.go new file mode 100644 index 00000000000..8c0496f670e --- /dev/null +++ b/pkg/member/election_leader.go @@ -0,0 +1,90 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package member + +import ( + "context" + + "github.com/pingcap/kvproto/pkg/pdpb" + "github.com/pingcap/kvproto/pkg/tsopb" +) + +// ElectionLeader defines the common interface of the leader, which is the pdpb.Member +// for in PD/API service or the tsopb.Participant in the microserives. +type ElectionLeader interface { + // GetListenUrls returns the listen urls + GetListenUrls() []string + // GetRevision the revision of the leader in etcd + GetRevision() int64 + // String declares fmt.Stringer + String() string + // Watch on itself, the leader in the election group + Watch(context.Context) +} + +// EmbeddedEtcdLeader is the leader in the election group backed by the embedded etcd. +type EmbeddedEtcdLeader struct { + wrapper *EmbeddedEtcdMember + member *pdpb.Member + revision int64 +} + +// GetListenUrls returns current leader's client urls +func (l *EmbeddedEtcdLeader) GetListenUrls() []string { + return l.member.GetClientUrls() +} + +// GetRevision the revision of the leader in etcd +func (l *EmbeddedEtcdLeader) GetRevision() int64 { + return l.revision +} + +// String declares fmt.Stringer +func (l *EmbeddedEtcdLeader) String() string { + return l.member.String() +} + +// Watch on the leader +func (l *EmbeddedEtcdLeader) Watch(ctx context.Context) { + l.wrapper.WatchLeader(ctx, l.member, l.revision) +} + +// EtcdLeader is the leader in the election group backed by the etcd, but it's +// decoupled from the embedded etcd. +type EtcdLeader struct { + wrapper *Participant + pariticipant *tsopb.Participant + revision int64 +} + +// GetListenUrls returns current leader's client urls +func (l *EtcdLeader) GetListenUrls() []string { + return l.pariticipant.GetListenUrls() +} + +// GetRevision the revision of the leader in etcd +func (l *EtcdLeader) GetRevision() int64 { + return l.revision +} + +// String declares fmt.Stringer +func (l *EtcdLeader) String() string { + return l.pariticipant.String() +} + +// Watch on the leader +func (l *EtcdLeader) Watch(ctx context.Context) { + l.wrapper.WatchLeader(ctx, l.pariticipant, l.revision) +} diff --git a/pkg/member/member.go b/pkg/member/member.go index 0c96424cb53..8bc020dc3ef 100644 --- a/pkg/member/member.go +++ b/pkg/member/member.go @@ -173,8 +173,8 @@ func (m *EmbeddedEtcdMember) KeepLeader(ctx context.Context) { m.leadership.Keep(ctx) } -// PrecheckLeader does some pre-check before checking whether or not it's the leader. -func (m *EmbeddedEtcdMember) PrecheckLeader() error { +// PreCheckLeader does some pre-check before checking whether or not it's the leader. +func (m *EmbeddedEtcdMember) PreCheckLeader() error { if m.GetEtcdLeader() == 0 { return errs.ErrEtcdLeaderNotFound } @@ -195,42 +195,51 @@ func (m *EmbeddedEtcdMember) getPersistentLeader() (*pdpb.Member, int64, error) return leader, rev, nil } -// CheckLeader checks returns true if it is needed to check later. -func (m *EmbeddedEtcdMember) CheckLeader() (*pdpb.Member, int64, bool) { - if err := m.PrecheckLeader(); err != nil { +// CheckLeader checks if someone else is taking the leadership. If yes, returns the leader; +// otherwise returns a bool which indicates if it is needed to check later. +func (m *EmbeddedEtcdMember) CheckLeader() (ElectionLeader, bool) { + if err := m.PreCheckLeader(); err != nil { log.Error("failed to pass pre-check, check pd leader later", errs.ZapError(err)) time.Sleep(200 * time.Millisecond) - return nil, 0, true + return nil, true } - leader, rev, err := m.getPersistentLeader() + leader, revision, err := m.getPersistentLeader() if err != nil { log.Error("getting pd leader meets error", errs.ZapError(err)) time.Sleep(200 * time.Millisecond) - return nil, 0, true - } - if leader != nil { - if m.IsSameLeader(leader) { - // oh, we are already a PD leader, which indicates we may meet something wrong - // in previous CampaignLeader. We should delete the leadership and campaign again. - log.Warn("the pd leader has not changed, delete and campaign again", zap.Stringer("old-pd-leader", leader)) - // Delete the leader itself and let others start a new election again. - if err = m.leadership.DeleteLeaderKey(); err != nil { - log.Error("deleting pd leader key meets error", errs.ZapError(err)) - time.Sleep(200 * time.Millisecond) - return nil, 0, true - } - // Return nil and false to make sure the campaign will start immediately. - return nil, 0, false + return nil, true + } + if leader == nil { + // no leader yet + return nil, false + } + + if m.IsSameLeader(leader) { + // oh, we are already a PD leader, which indicates we may meet something wrong + // in previous CampaignLeader. We should delete the leadership and campaign again. + log.Warn("the pd leader has not changed, delete and campaign again", zap.Stringer("old-pd-leader", leader)) + // Delete the leader itself and let others start a new election again. + if err = m.leadership.DeleteLeaderKey(); err != nil { + log.Error("deleting pd leader key meets error", errs.ZapError(err)) + time.Sleep(200 * time.Millisecond) + return nil, true } + // Return nil and false to make sure the campaign will start immediately. + return nil, false } - return leader, rev, false + + return &EmbeddedEtcdLeader{ + wrapper: m, + member: leader, + revision: revision, + }, false } // WatchLeader is used to watch the changes of the leader. -func (m *EmbeddedEtcdMember) WatchLeader(serverCtx context.Context, leader *pdpb.Member, revision int64) { +func (m *EmbeddedEtcdMember) WatchLeader(ctx context.Context, leader *pdpb.Member, revision int64) { m.setLeader(leader) - m.leadership.Watch(serverCtx, revision) + m.leadership.Watch(ctx, revision) m.unsetLeader() } @@ -308,6 +317,7 @@ func (m *EmbeddedEtcdMember) InitMemberInfo(advertiseClientUrls, advertisePeerUr m.memberValue = string(data) m.rootPath = rootPath m.leadership = election.NewLeadership(m.client, m.GetLeaderPath(), "leader election") + log.Info("member joining election", zap.Stringer("member-info", m.member), zap.String("root-path", m.rootPath)) } // ResignEtcdLeader resigns current PD's etcd leadership. If nextLeader is empty, all diff --git a/pkg/member/participant.go b/pkg/member/participant.go index dd4ec107360..41cdef77004 100644 --- a/pkg/member/participant.go +++ b/pkg/member/participant.go @@ -31,9 +31,11 @@ import ( "go.uber.org/zap" ) +type leadershipCheckFunc func(*election.Leadership) bool + // Participant is used for the election related logic. Compared to its counterpart // EmbeddedEtcdMember, Participant relies on etcd for election, but it's decoupled -// with the embedded etcd. It implements Member interface. +// from the embedded etcd. It implements Member interface. type Participant struct { leadership *election.Leadership // stored as member type @@ -46,6 +48,9 @@ type Participant struct { // leader key when this participant is successfully elected as the leader of // the group. Every write will use it to check the leadership. memberValue string + // campaignChecker is used to check whether the additional constraints for a + // campaign are satisfied. If it returns false, the campaign will fail. + campaignChecker atomic.Value // Store as leadershipCheckFunc } // NewParticipant create a new Participant. @@ -73,7 +78,7 @@ func (m *Participant) InitInfo(name string, id uint64, rootPath string, leaderNa m.rootPath = rootPath m.leaderPath = path.Join(rootPath, leaderName) m.leadership = election.NewLeadership(m.client, m.GetLeaderPath(), purpose) - log.Info("Participant initialized", zap.String("leader-path", m.leaderPath)) + log.Info("participant joining election", zap.Stringer("participant-info", m.member), zap.String("leader-path", m.leaderPath)) } // ID returns the unique ID for this participant in the election group @@ -104,7 +109,7 @@ func (m *Participant) Client() *clientv3.Client { // IsLeader returns whether the participant is the leader or not by checking its leadership's // lease and leader info. func (m *Participant) IsLeader() bool { - return m.leadership.Check() && m.GetLeader().GetId() == m.member.GetId() + return m.leadership.Check() && m.GetLeader().GetId() == m.member.GetId() && m.campaignCheck() } // IsLeaderElected returns true if the leader exists; otherwise false @@ -162,6 +167,9 @@ func (m *Participant) GetLeadership() *election.Leadership { // CampaignLeader is used to campaign the leadership and make it become a leader. func (m *Participant) CampaignLeader(leaseTimeout int64) error { + if !m.campaignCheck() { + return errs.ErrCheckCampaign + } return m.leadership.Campaign(leaseTimeout, m.MemberValue()) } @@ -170,9 +178,9 @@ func (m *Participant) KeepLeader(ctx context.Context) { m.leadership.Keep(ctx) } -// PrecheckLeader does some pre-check before checking whether or not it's the leader. +// PreCheckLeader does some pre-check before checking whether or not it's the leader. // It returns true if it passes the pre-check, false otherwise. -func (m *Participant) PrecheckLeader() error { +func (m *Participant) PreCheckLeader() error { // No specific thing to check. Returns no error. return nil } @@ -191,42 +199,51 @@ func (m *Participant) getPersistentLeader() (*tsopb.Participant, int64, error) { return leader, rev, nil } -// CheckLeader checks returns true if it is needed to check later. -func (m *Participant) CheckLeader() (*tsopb.Participant, int64, bool) { - if err := m.PrecheckLeader(); err != nil { +// CheckLeader checks if someone else is taking the leadership. If yes, returns the leader; +// otherwise returns a bool which indicates if it is needed to check later. +func (m *Participant) CheckLeader() (ElectionLeader, bool) { + if err := m.PreCheckLeader(); err != nil { log.Error("failed to pass pre-check, check the leader later", errs.ZapError(errs.ErrEtcdLeaderNotFound)) time.Sleep(200 * time.Millisecond) - return nil, 0, true + return nil, true } - leader, rev, err := m.getPersistentLeader() + leader, revision, err := m.getPersistentLeader() if err != nil { log.Error("getting the leader meets error", errs.ZapError(err)) time.Sleep(200 * time.Millisecond) - return nil, 0, true + return nil, true + } + if leader == nil { + // no leader yet + return nil, false } - if leader != nil { - if m.IsSameLeader(leader) { - // oh, we are already the leader, which indicates we may meet something wrong - // in previous CampaignLeader. We should delete the leadership and campaign again. - log.Warn("the leader has not changed, delete and campaign again", zap.Stringer("old-leader", leader)) - // Delete the leader itself and let others start a new election again. - if err = m.leadership.DeleteLeaderKey(); err != nil { - log.Error("deleting the leader key meets error", errs.ZapError(err)) - time.Sleep(200 * time.Millisecond) - return nil, 0, true - } - // Return nil and false to make sure the campaign will start immediately. - return nil, 0, false + + if m.IsSameLeader(leader) { + // oh, we are already the leader, which indicates we may meet something wrong + // in previous CampaignLeader. We should delete the leadership and campaign again. + log.Warn("the leader has not changed, delete and campaign again", zap.Stringer("old-leader", leader)) + // Delete the leader itself and let others start a new election again. + if err = m.leadership.DeleteLeaderKey(); err != nil { + log.Error("deleting the leader key meets error", errs.ZapError(err)) + time.Sleep(200 * time.Millisecond) + return nil, true } + // Return nil and false to make sure the campaign will start immediately. + return nil, false } - return leader, rev, false + + return &EtcdLeader{ + wrapper: m, + pariticipant: leader, + revision: revision, + }, false } // WatchLeader is used to watch the changes of the leader. -func (m *Participant) WatchLeader(serverCtx context.Context, leader *tsopb.Participant, revision int64) { +func (m *Participant) WatchLeader(ctx context.Context, leader *tsopb.Participant, revision int64) { m.setLeader(leader) - m.leadership.Watch(serverCtx, revision) + m.leadership.Watch(ctx, revision) m.unsetLeader() } @@ -319,3 +336,20 @@ func (m *Participant) GetLeaderPriority(id uint64) (int, error) { } return int(priority), nil } + +func (m *Participant) campaignCheck() bool { + checker := m.campaignChecker.Load() + if checker == nil { + return true + } + checkerFunc, ok := checker.(leadershipCheckFunc) + if !ok || checkerFunc == nil { + return true + } + return checkerFunc(m.leadership) +} + +// SetCampaignChecker sets the pre-campaign checker. +func (m *Participant) SetCampaignChecker(checker leadershipCheckFunc) { + m.campaignChecker.Store(checker) +} diff --git a/pkg/slice/slice.go b/pkg/slice/slice.go index f26f6ca6459..b3741593670 100644 --- a/pkg/slice/slice.go +++ b/pkg/slice/slice.go @@ -46,3 +46,15 @@ func Contains[T comparable](slice []T, value T) bool { } return false } + +// Remove removes the value from the slice. +func Remove[T comparable](slice []T, value T) []T { + i, j := 0, 0 + for ; i < len(slice); i++ { + if slice[i] != value { + slice[j] = slice[i] + j++ + } + } + return slice[:j] +} diff --git a/pkg/slice/slice_test.go b/pkg/slice/slice_test.go index d8ba709eb66..1fe3fe79dcf 100644 --- a/pkg/slice/slice_test.go +++ b/pkg/slice/slice_test.go @@ -48,14 +48,53 @@ func TestSliceContains(t *testing.T) { t.Parallel() re := require.New(t) ss := []string{"a", "b", "c"} - re.Contains(ss, "a") - re.NotContains(ss, "d") + re.True(slice.Contains(ss, "a")) + re.False(slice.Contains(ss, "d")) us := []uint64{1, 2, 3} - re.Contains(us, uint64(1)) - re.NotContains(us, uint64(4)) + re.True(slice.Contains(us, uint64(1))) + re.False(slice.Contains(us, uint64(4))) is := []int64{1, 2, 3} - re.Contains(is, int64(1)) - re.NotContains(is, int64(4)) + re.True(slice.Contains(is, int64(1))) + re.False(slice.Contains(is, int64(4))) +} + +func TestSliceRemoveGenericTypes(t *testing.T) { + t.Parallel() + re := require.New(t) + ss := []string{"a", "b", "c"} + ss = slice.Remove(ss, "a") + re.Equal([]string{"b", "c"}, ss) + + us := []uint64{1, 2, 3} + us = slice.Remove(us, 1) + re.Equal([]uint64{2, 3}, us) + + is := []int64{1, 2, 3} + is = slice.Remove(is, 1) + re.Equal([]int64{2, 3}, is) +} + +func TestSliceRemove(t *testing.T) { + t.Parallel() + re := require.New(t) + + is := []int64{} + is = slice.Remove(is, 1) + re.Equal([]int64{}, is) + + is = []int64{1} + is = slice.Remove(is, 2) + re.Equal([]int64{1}, is) + is = slice.Remove(is, 1) + re.Equal([]int64{}, is) + + is = []int64{1, 2, 3} + is = slice.Remove(is, 1) + re.Equal([]int64{2, 3}, is) + + is = []int64{1, 1, 1} + is = slice.Remove(is, 1) + re.Equal([]int64{}, is) } diff --git a/pkg/storage/endpoint/key_path.go b/pkg/storage/endpoint/key_path.go index fc719f7872c..80e523f63a7 100644 --- a/pkg/storage/endpoint/key_path.go +++ b/pkg/storage/endpoint/key_path.go @@ -17,6 +17,7 @@ package endpoint import ( "fmt" "path" + "regexp" "strconv" "strings" @@ -49,10 +50,13 @@ const ( resourceGroupStatesPath = "states" requestUnitConfigPath = "ru_config" // tso storage endpoint has prefix `tso` - microserviceKey = "microservice" + microserviceKey = "ms" tsoServiceKey = utils.TSOServiceName timestampKey = "timestamp" + tsoKeyspaceGroupPrefix = "tso/keyspace_groups" + keyspaceGroupMembershipKey = "membership" + // we use uint64 to represent ID, the max length of uint64 is 20. keyLen = 20 ) @@ -223,3 +227,36 @@ func KeyspaceIDAlloc() string { func encodeKeyspaceID(spaceID uint32) string { return fmt.Sprintf("%08d", spaceID) } + +// KeyspaceGroupIDPrefix returns the prefix of keyspace group id. +// Path: tso/keyspace_groups/membership +func KeyspaceGroupIDPrefix() string { + return path.Join(tsoKeyspaceGroupPrefix, keyspaceGroupMembershipKey) +} + +// KeyspaceGroupIDPath returns the path to keyspace id from the given name. +// Path: tso/keyspace_groups/membership/{id} +func KeyspaceGroupIDPath(id uint32) string { + return path.Join(tsoKeyspaceGroupPrefix, keyspaceGroupMembershipKey, encodeKeyspaceGroupID(id)) +} + +// ExtractKeyspaceGroupIDFromPath extracts keyspace group id from the given path, which contains +// the pattern of `tso/keyspace_groups/membership/(\d{5})$`. +func ExtractKeyspaceGroupIDFromPath(path string) (uint32, error) { + pattern := strings.Join([]string{KeyspaceGroupIDPrefix(), `(\d{5})$`}, "/") + re := regexp.MustCompile(pattern) + match := re.FindStringSubmatch(path) + if match == nil { + return 0, fmt.Errorf("invalid keyspace group id path: %s", path) + } + id, err := strconv.ParseUint(match[1], 10, 32) + if err != nil { + return 0, fmt.Errorf("failed to parse keyspace group ID: %v", err) + } + return uint32(id), nil +} + +// encodeKeyspaceGroupID from uint32 to string. +func encodeKeyspaceGroupID(groupID uint32) string { + return fmt.Sprintf("%05d", groupID) +} diff --git a/pkg/storage/endpoint/key_path_test.go b/pkg/storage/endpoint/key_path_test.go index 57313f3a7b2..02547c55144 100644 --- a/pkg/storage/endpoint/key_path_test.go +++ b/pkg/storage/endpoint/key_path_test.go @@ -27,3 +27,51 @@ func BenchmarkRegionPath(b *testing.B) { _ = RegionPath(uint64(i)) } } + +func TestExtractKeyspaceGroupIDFromPath(t *testing.T) { + re := require.New(t) + + rightCases := []struct { + path string + id uint32 + }{ + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/00000", id: 0}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/00001", id: 1}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/12345", id: 12345}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/99999", id: 99999}, + {path: "tso/keyspace_groups/membership/00000", id: 0}, + {path: "tso/keyspace_groups/membership/00001", id: 1}, + {path: "tso/keyspace_groups/membership/12345", id: 12345}, + {path: "tso/keyspace_groups/membership/99999", id: 99999}, + } + + for _, tt := range rightCases { + id, err := ExtractKeyspaceGroupIDFromPath(tt.path) + re.Equal(tt.id, id) + re.NoError(err) + } + + wrongCases := []struct { + path string + }{ + {path: ""}, + {path: "00001"}, + {path: "xxx/keyspace_groups/membership/00001"}, + {path: "tso/xxxxxxxxxxxxxxx/membership/00001"}, + {path: "tso/keyspace_groups/xxxxxxxxxx/00001"}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/xxxxxxxxxx/00001"}, + {path: "/pd/{cluster_id}/xxx/keyspace_groups/membership/00001"}, + {path: "/pd/{cluster_id}/tso/xxxxxxxxxxxxxxx/membership/00001"}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/"}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/0"}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/0001"}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/123456"}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/1234a"}, + {path: "/pd/{cluster_id}/tso/keyspace_groups/membership/12345a"}, + } + + for _, tt := range wrongCases { + _, err := ExtractKeyspaceGroupIDFromPath(tt.path) + re.Error(err) + } +} diff --git a/pkg/storage/endpoint/keyspace.go b/pkg/storage/endpoint/keyspace.go index 7aa82e8985b..09733ad59c1 100644 --- a/pkg/storage/endpoint/keyspace.go +++ b/pkg/storage/endpoint/keyspace.go @@ -41,7 +41,7 @@ type KeyspaceStorage interface { SaveKeyspaceID(txn kv.Txn, id uint32, name string) error LoadKeyspaceID(txn kv.Txn, name string) (bool, uint32, error) // LoadRangeKeyspace loads no more than limit keyspaces starting at startID. - LoadRangeKeyspace(startID uint32, limit int) ([]*keyspacepb.KeyspaceMeta, error) + LoadRangeKeyspace(txn kv.Txn, startID uint32, limit int) ([]*keyspacepb.KeyspaceMeta, error) RunInTxn(ctx context.Context, f func(txn kv.Txn) error) error } @@ -104,10 +104,10 @@ func (se *StorageEndpoint) RunInTxn(ctx context.Context, f func(txn kv.Txn) erro // LoadRangeKeyspace loads keyspaces starting at startID. // limit specifies the limit of loaded keyspaces. -func (se *StorageEndpoint) LoadRangeKeyspace(startID uint32, limit int) ([]*keyspacepb.KeyspaceMeta, error) { +func (se *StorageEndpoint) LoadRangeKeyspace(txn kv.Txn, startID uint32, limit int) ([]*keyspacepb.KeyspaceMeta, error) { startKey := KeyspaceMetaPath(startID) endKey := clientv3.GetPrefixRangeEnd(KeyspaceMetaPrefix()) - keys, values, err := se.LoadRange(startKey, endKey, limit) + keys, values, err := txn.LoadRange(startKey, endKey, limit) if err != nil { return nil, err } diff --git a/pkg/storage/endpoint/tso_keyspace_group.go b/pkg/storage/endpoint/tso_keyspace_group.go new file mode 100644 index 00000000000..f63922ea64b --- /dev/null +++ b/pkg/storage/endpoint/tso_keyspace_group.go @@ -0,0 +1,186 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package endpoint + +import ( + "context" + "encoding/json" + + "github.com/tikv/pd/pkg/storage/kv" + "go.etcd.io/etcd/clientv3" +) + +// UserKind represents the user kind. +type UserKind int + +// Different user kinds. +const ( + Basic UserKind = iota + Standard + Enterprise + + UserKindCount +) + +// StringUserKind creates a UserKind with string. +func StringUserKind(input string) UserKind { + switch input { + case Basic.String(): + return Basic + case Standard.String(): + return Standard + case Enterprise.String(): + return Enterprise + default: + return Basic + } +} + +func (k UserKind) String() string { + switch k { + case Basic: + return "basic" + case Standard: + return "standard" + case Enterprise: + return "enterprise" + } + return "unknown UserKind" +} + +// IsUserKindValid checks if the user kind is valid. +func IsUserKindValid(kind string) bool { + switch kind { + case Basic.String(), Standard.String(), Enterprise.String(): + return true + default: + return false + } +} + +// KeyspaceGroupMember defines an election member which campaigns for the primary of the keyspace group. +type KeyspaceGroupMember struct { + Address string `json:"address"` +} + +// SplitState defines the split state of a keyspace group. +type SplitState struct { + // SplitSource is the current keyspace group ID from which the keyspace group is split. + // When the keyspace group is being split to another keyspace group, the split-source will + // be set to its own ID. + SplitSource uint32 `json:"split-source"` +} + +// KeyspaceGroup is the keyspace group. +type KeyspaceGroup struct { + ID uint32 `json:"id"` + UserKind string `json:"user-kind"` + // SplitState is the current split state of the keyspace group. + SplitState *SplitState `json:"split-state,omitempty"` + // Members are the election members which campaign for the primary of the keyspace group. + Members []KeyspaceGroupMember `json:"members"` + // Keyspaces are the keyspace IDs which belong to the keyspace group. + Keyspaces []uint32 `json:"keyspaces"` + // KeyspaceLookupTable is for fast lookup if a given keyspace belongs to this keyspace group. + // It's not persisted and will be built when loading from storage. + KeyspaceLookupTable map[uint32]struct{} `json:"-"` +} + +// IsSplitting checks if the keyspace group is in split state. +func (kg *KeyspaceGroup) IsSplitting() bool { + return kg != nil && kg.SplitState != nil +} + +// IsSplitTarget checks if the keyspace group is in split state and is the split target. +func (kg *KeyspaceGroup) IsSplitTarget() bool { + return kg.IsSplitting() && kg.SplitState.SplitSource != kg.ID +} + +// IsSplitSource checks if the keyspace group is in split state and is the split source. +func (kg *KeyspaceGroup) IsSplitSource() bool { + return kg.IsSplitting() && kg.SplitState.SplitSource == kg.ID +} + +// SplitSource returns the keyspace group split source ID. When the keyspace group is the split source +// itself, it will return its own ID. +func (kg *KeyspaceGroup) SplitSource() uint32 { + if kg.IsSplitting() { + return kg.SplitState.SplitSource + } + return 0 +} + +// KeyspaceGroupStorage is the interface for keyspace group storage. +type KeyspaceGroupStorage interface { + LoadKeyspaceGroups(startID uint32, limit int) ([]*KeyspaceGroup, error) + LoadKeyspaceGroup(txn kv.Txn, id uint32) (*KeyspaceGroup, error) + SaveKeyspaceGroup(txn kv.Txn, kg *KeyspaceGroup) error + DeleteKeyspaceGroup(txn kv.Txn, id uint32) error + // TODO: add more interfaces. + RunInTxn(ctx context.Context, f func(txn kv.Txn) error) error +} + +var _ KeyspaceGroupStorage = (*StorageEndpoint)(nil) + +// LoadKeyspaceGroup loads the keyspace group by ID. +func (se *StorageEndpoint) LoadKeyspaceGroup(txn kv.Txn, id uint32) (*KeyspaceGroup, error) { + value, err := txn.Load(KeyspaceGroupIDPath(id)) + if err != nil || value == "" { + return nil, err + } + kg := &KeyspaceGroup{} + if err := json.Unmarshal([]byte(value), kg); err != nil { + return nil, err + } + return kg, nil +} + +// SaveKeyspaceGroup saves the keyspace group. +func (se *StorageEndpoint) SaveKeyspaceGroup(txn kv.Txn, kg *KeyspaceGroup) error { + key := KeyspaceGroupIDPath(kg.ID) + value, err := json.Marshal(kg) + if err != nil { + return err + } + return txn.Save(key, string(value)) +} + +// DeleteKeyspaceGroup deletes the keyspace group. +func (se *StorageEndpoint) DeleteKeyspaceGroup(txn kv.Txn, id uint32) error { + return txn.Remove(KeyspaceGroupIDPath(id)) +} + +// LoadKeyspaceGroups loads keyspace groups from the start ID with limit. +// If limit is 0, it will load all keyspace groups from the start ID. +func (se *StorageEndpoint) LoadKeyspaceGroups(startID uint32, limit int) ([]*KeyspaceGroup, error) { + prefix := KeyspaceGroupIDPath(startID) + prefixEnd := clientv3.GetPrefixRangeEnd(KeyspaceGroupIDPrefix()) + keys, values, err := se.LoadRange(prefix, prefixEnd, limit) + if err != nil { + return nil, err + } + if len(keys) == 0 { + return []*KeyspaceGroup{}, nil + } + kgs := make([]*KeyspaceGroup, 0, len(keys)) + for _, value := range values { + kg := &KeyspaceGroup{} + if err = json.Unmarshal([]byte(value), kg); err != nil { + return nil, err + } + kgs = append(kgs, kg) + } + return kgs, nil +} diff --git a/pkg/storage/keyspace_test.go b/pkg/storage/keyspace_test.go index 152d71afb5c..1236d1b34bd 100644 --- a/pkg/storage/keyspace_test.go +++ b/pkg/storage/keyspace_test.go @@ -72,9 +72,8 @@ func TestSaveLoadKeyspace(t *testing.T) { func TestLoadRangeKeyspaces(t *testing.T) { re := require.New(t) storage := NewStorageWithMemoryBackend() - - // Store test keyspace meta. keyspaces := makeTestKeyspaces() + // Store test keyspace meta. err := storage.RunInTxn(context.TODO(), func(txn kv.Txn) error { for _, keyspace := range keyspaces { re.NoError(storage.SaveKeyspaceMeta(txn, keyspace)) @@ -82,21 +81,26 @@ func TestLoadRangeKeyspaces(t *testing.T) { return nil }) re.NoError(err) + // Test load range keyspaces. + err = storage.RunInTxn(context.TODO(), func(txn kv.Txn) error { + // Load all keyspaces. + loadedKeyspaces, err := storage.LoadRangeKeyspace(txn, keyspaces[0].GetId(), 0) + re.NoError(err) + re.ElementsMatch(keyspaces, loadedKeyspaces) - // Load all keyspaces. - loadedKeyspaces, err := storage.LoadRangeKeyspace(keyspaces[0].GetId(), 0) - re.NoError(err) - re.ElementsMatch(keyspaces, loadedKeyspaces) + // Load keyspaces with id >= second test keyspace's id. + loadedKeyspaces2, err := storage.LoadRangeKeyspace(txn, keyspaces[1].GetId(), 0) + re.NoError(err) + re.ElementsMatch(keyspaces[1:], loadedKeyspaces2) - // Load keyspaces with id >= second test keyspace's id. - loadedKeyspaces2, err := storage.LoadRangeKeyspace(keyspaces[1].GetId(), 0) - re.NoError(err) - re.ElementsMatch(keyspaces[1:], loadedKeyspaces2) + // Load keyspace with the smallest id. + loadedKeyspace3, err := storage.LoadRangeKeyspace(txn, 1, 1) + re.NoError(err) + re.ElementsMatch(keyspaces[:1], loadedKeyspace3) - // Load keyspace with the smallest id. - loadedKeyspace3, err := storage.LoadRangeKeyspace(1, 1) + return nil + }) re.NoError(err) - re.ElementsMatch(keyspaces[:1], loadedKeyspace3) } func makeTestKeyspaces() []*keyspacepb.KeyspaceMeta { diff --git a/pkg/storage/storage.go b/pkg/storage/storage.go index f2bfe1f74fb..da6a06c588a 100644 --- a/pkg/storage/storage.go +++ b/pkg/storage/storage.go @@ -44,6 +44,7 @@ type Storage interface { endpoint.KeyspaceStorage endpoint.ResourceGroupStorage endpoint.TSOStorage + endpoint.KeyspaceGroupStorage } // NewStorageWithMemoryBackend creates a new storage with memory backend. diff --git a/pkg/tso/admin.go b/pkg/tso/admin.go index 337d09e677a..7d510cdef65 100644 --- a/pkg/tso/admin.go +++ b/pkg/tso/admin.go @@ -25,7 +25,7 @@ import ( // Handler defines the common behaviors of a basic tso handler. type Handler interface { - ResetTS(ts uint64, ignoreSmaller, skipUpperBoundCheck bool) error + ResetTS(ts uint64, ignoreSmaller, skipUpperBoundCheck bool, keyspaceGroupID uint32) error } // AdminHandler wrap the basic tso handler to provide http service. @@ -93,7 +93,7 @@ func (h *AdminHandler) ResetTS(w http.ResponseWriter, r *http.Request) { ignoreSmaller, skipUpperBoundCheck = true, true } - if err = handler.ResetTS(ts, ignoreSmaller, skipUpperBoundCheck); err != nil { + if err = handler.ResetTS(ts, ignoreSmaller, skipUpperBoundCheck, 0); err != nil { if err == errs.ErrServerNotStarted { h.rd.JSON(w, http.StatusInternalServerError, err.Error()) } else { diff --git a/pkg/tso/allocator_manager.go b/pkg/tso/allocator_manager.go index 473b39c9ad5..35c671b2236 100644 --- a/pkg/tso/allocator_manager.go +++ b/pkg/tso/allocator_manager.go @@ -30,6 +30,8 @@ import ( "github.com/pingcap/log" "github.com/tikv/pd/pkg/election" "github.com/tikv/pd/pkg/errs" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/member" "github.com/tikv/pd/pkg/slice" "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/pkg/storage/kv" @@ -44,13 +46,13 @@ import ( const ( // GlobalDCLocation is the Global TSO Allocator's DC location label. - GlobalDCLocation = "global" - checkStep = time.Minute - patrolStep = time.Second - defaultAllocatorLeaderLease = 3 - leaderTickInterval = 50 * time.Millisecond - localTSOAllocatorEtcdPrefix = "lta" - localTSOSuffixEtcdPrefix = "lts" + GlobalDCLocation = "global" + checkStep = time.Minute + patrolStep = time.Second + defaultAllocatorLeaderLease = 3 + globalTSOAllocatorEtcdPrefix = "gta" + localTSOAllocatorEtcdPrefix = "lta" + localTSOSuffixEtcdPrefix = "lts" ) var ( @@ -95,8 +97,8 @@ func (info *DCLocationInfo) clone() DCLocationInfo { return copiedInfo } -// Member defines the interface for the election related logic. -type Member interface { +// ElectionMember defines the interface for the election related logic. +type ElectionMember interface { // ID returns the unique ID in the election group. For example, it can be unique // server id of a cluster or the unique keyspace group replica id of the election // group comprised of the replicas of a keyspace group. @@ -112,32 +114,42 @@ type Member interface { // IsLeader returns whether the participant is the leader or not by checking its // leadership's lease and leader info. IsLeader() bool - // IsLeaderElected returns true if the leader exists; otherwise false + // IsLeaderElected returns true if the leader exists; otherwise false. IsLeaderElected() bool + // CheckLeader checks if someone else is taking the leadership. If yes, returns the leader; + // otherwise returns a bool which indicates if it is needed to check later. + CheckLeader() (leader member.ElectionLeader, checkAgain bool) + // EnableLeader declares the member itself to be the leader. + EnableLeader() + // KeepLeader is used to keep the leader's leadership. + KeepLeader(ctx context.Context) + // CampaignLeader is used to campaign the leadership and make it become a leader in an election group. + CampaignLeader(leaseTimeout int64) error + // ResetLeader is used to reset the member's current leadership. + // Basically it will reset the leader lease and unset leader info. + ResetLeader() // GetLeaderListenUrls returns current leader's listen urls + // The first element is the leader/primary url GetLeaderListenUrls() []string // GetLeaderID returns current leader's member ID. GetLeaderID() uint64 - // EnableLeader declares the member itself to be the leader. - EnableLeader() // GetLeaderPath returns the path of the leader. GetLeaderPath() string - // GetLeadership returns the leadership of the PD member. + // GetLeadership returns the leadership of the election member. GetLeadership() *election.Leadership // GetDCLocationPathPrefix returns the dc-location path prefix of the cluster. GetDCLocationPathPrefix() string // GetDCLocationPath returns the dc-location path of a member with the given member ID. GetDCLocationPath(id uint64) string - // PrecheckLeader does some pre-check before checking whether it's the leader. - PrecheckLeader() error + // PreCheckLeader does some pre-check before checking whether it's the leader. + PreCheckLeader() error } // AllocatorManager is used to manage the TSO Allocators a PD server holds. // It is in charge of maintaining TSO allocators' leadership, checking election // priority, and forwarding TSO allocation requests to correct TSO Allocators. type AllocatorManager struct { - enableLocalTSO bool - mu struct { + mu struct { syncutil.RWMutex // There are two kinds of TSO Allocators: // 1. Global TSO Allocator, as a global single point to allocate @@ -150,16 +162,29 @@ type AllocatorManager struct { // the number of suffix bits we need in the TSO logical part. maxSuffix int32 } + // for the synchronization purpose of the allocator update checks wg sync.WaitGroup - // for election use - member Member + // for the synchronization purpose of the service loops + svcLoopWG sync.WaitGroup + + ctx context.Context + cancel context.CancelFunc + // kgID is the keyspace group ID + kgID uint32 + // member is for election use + member ElectionMember // TSO config rootPath string storage endpoint.TSOStorage + enableLocalTSO bool saveInterval time.Duration updatePhysicalInterval time.Duration - maxResetTSGap func() time.Duration - securityConfig *grpcutil.TLSConfig + // leaderLease defines the time within which a TSO primary/leader must update its TTL + // in etcd, otherwise etcd will expire the leader key and other servers can campaign + // the primary/leader again. Etcd only supports seconds TTL, so here is second too. + leaderLease int64 + maxResetTSGap func() time.Duration + securityConfig *grpcutil.TLSConfig // for gRPC use localAllocatorConn struct { syncutil.RWMutex @@ -169,29 +194,128 @@ type AllocatorManager struct { // NewAllocatorManager creates a new TSO Allocator Manager. func NewAllocatorManager( - m Member, + ctx context.Context, + keyspaceGroupID uint32, + member ElectionMember, rootPath string, storage endpoint.TSOStorage, - enableLocalTSO bool, - saveInterval time.Duration, - updatePhysicalInterval time.Duration, - tlsConfig *grpcutil.TLSConfig, - maxResetTSGap func() time.Duration, + cfg Config, + startGlobalLeaderLoop bool, ) *AllocatorManager { - allocatorManager := &AllocatorManager{ - enableLocalTSO: enableLocalTSO, - member: m, + ctx, cancel := context.WithCancel(ctx) + am := &AllocatorManager{ + ctx: ctx, + cancel: cancel, + kgID: keyspaceGroupID, + member: member, rootPath: rootPath, storage: storage, - saveInterval: saveInterval, - updatePhysicalInterval: updatePhysicalInterval, - maxResetTSGap: maxResetTSGap, - securityConfig: tlsConfig, - } - allocatorManager.mu.allocatorGroups = make(map[string]*allocatorGroup) - allocatorManager.mu.clusterDCLocations = make(map[string]*DCLocationInfo) - allocatorManager.localAllocatorConn.clientConns = make(map[string]*grpc.ClientConn) - return allocatorManager + enableLocalTSO: cfg.IsLocalTSOEnabled(), + saveInterval: cfg.GetTSOSaveInterval(), + updatePhysicalInterval: cfg.GetTSOUpdatePhysicalInterval(), + leaderLease: cfg.GetLeaderLease(), + maxResetTSGap: cfg.GetMaxResetTSGap, + securityConfig: cfg.GetTLSConfig(), + } + am.mu.allocatorGroups = make(map[string]*allocatorGroup) + am.mu.clusterDCLocations = make(map[string]*DCLocationInfo) + am.localAllocatorConn.clientConns = make(map[string]*grpc.ClientConn) + + // Set up the Global TSO Allocator here, it will be initialized once the member campaigns leader successfully. + am.SetUpGlobalAllocator(am.ctx, am.member.GetLeadership(), startGlobalLeaderLoop) + am.svcLoopWG.Add(1) + go am.tsoAllocatorLoop() + + return am +} + +// SetUpGlobalAllocator is used to set up the global allocator, which will initialize the allocator and put it into +// an allocator daemon. An TSO Allocator should only be set once, and may be initialized and reset multiple times +// depending on the election. +func (am *AllocatorManager) SetUpGlobalAllocator(ctx context.Context, leadership *election.Leadership, startGlobalLeaderLoop bool) { + am.mu.Lock() + defer am.mu.Unlock() + + allocator := NewGlobalTSOAllocator(ctx, am, startGlobalLeaderLoop) + // Create a new allocatorGroup + ctx, cancel := context.WithCancel(ctx) + am.mu.allocatorGroups[GlobalDCLocation] = &allocatorGroup{ + dcLocation: GlobalDCLocation, + ctx: ctx, + cancel: cancel, + leadership: leadership, + allocator: allocator, + } +} + +// setUpLocalAllocator is used to set up an allocator, which will initialize the allocator and put it into allocator daemon. +// One TSO Allocator should only be set once, and may be initialized and reset multiple times depending on the election. +func (am *AllocatorManager) setUpLocalAllocator(parentCtx context.Context, dcLocation string, leadership *election.Leadership) { + am.mu.Lock() + defer am.mu.Unlock() + + if _, exist := am.mu.allocatorGroups[dcLocation]; exist { + return + } + allocator := NewLocalTSOAllocator(am, leadership, dcLocation) + // Create a new allocatorGroup + ctx, cancel := context.WithCancel(parentCtx) + am.mu.allocatorGroups[dcLocation] = &allocatorGroup{ + dcLocation: dcLocation, + ctx: ctx, + cancel: cancel, + leadership: leadership, + allocator: allocator, + } + // Start election of the Local TSO Allocator here + localTSOAllocator, _ := allocator.(*LocalTSOAllocator) + go am.allocatorLeaderLoop(parentCtx, localTSOAllocator) +} + +// GetTimestampPath returns the timestamp path in etcd for the given DCLocation. +func (am *AllocatorManager) GetTimestampPath(dcLocation string) string { + if am == nil { + return "" + } + if len(dcLocation) == 0 { + dcLocation = GlobalDCLocation + } + + am.mu.RLock() + defer am.mu.RUnlock() + if allocatorGroup, exist := am.mu.allocatorGroups[dcLocation]; exist { + return path.Join(am.rootPath, allocatorGroup.allocator.GetTimestampPath()) + } + return "" +} + +// tsoAllocatorLoop is used to run the TSO Allocator updating daemon. +func (am *AllocatorManager) tsoAllocatorLoop() { + defer logutil.LogPanic() + defer am.svcLoopWG.Done() + + am.AllocatorDaemon(am.ctx) + log.Info("exit allocator loop") +} + +// close is used to shutdown TSO Allocator updating daemon. +// tso service call this function to shutdown the loop here, but pd manages its own loop. +func (am *AllocatorManager) close() { + log.Info("closing the allocator manager") + + if allocatorGroup, exist := am.getAllocatorGroup(GlobalDCLocation); exist { + allocatorGroup.allocator.(*GlobalTSOAllocator).close() + } + + am.cancel() + am.svcLoopWG.Wait() + + log.Info("closed the allocator manager") +} + +// GetMember returns the ElectionMember of this AllocatorManager. +func (am *AllocatorManager) GetMember() ElectionMember { + return am.member } // SetLocalTSOConfig receives the zone label of this PD server and write it into etcd as dc-location @@ -301,8 +425,7 @@ func (am *AllocatorManager) CleanUpDCLocation() error { } else if !resp.Succeeded { return errs.ErrEtcdTxnConflict.FastGenByArgs() } - log.Info("delete the dc-location key previously written in etcd", - zap.Uint64("server-id", serverID)) + log.Info("delete the dc-location key previously written in etcd", zap.Uint64("server-id", serverID)) go am.ClusterDCLocationChecker() return nil } @@ -350,40 +473,6 @@ func CalSuffixBits(maxSuffix int32) int { return int(math.Ceil(math.Log2(float64(maxSuffix + 1)))) } -// SetUpAllocator is used to set up an allocator, which will initialize the allocator and put it into allocator daemon. -// One TSO Allocator should only be set once, and may be initialized and reset multiple times depending on the election. -func (am *AllocatorManager) SetUpAllocator(parentCtx context.Context, dcLocation string, leadership *election.Leadership) { - am.mu.Lock() - defer am.mu.Unlock() - if _, exist := am.mu.allocatorGroups[dcLocation]; exist { - return - } - var allocator Allocator - if dcLocation == GlobalDCLocation { - allocator = NewGlobalTSOAllocator(am, leadership) - } else { - allocator = NewLocalTSOAllocator(am, leadership, dcLocation) - } - // Create a new allocatorGroup - ctx, cancel := context.WithCancel(parentCtx) - am.mu.allocatorGroups[dcLocation] = &allocatorGroup{ - dcLocation: dcLocation, - ctx: ctx, - cancel: cancel, - leadership: leadership, - allocator: allocator, - } - // Because the Global TSO Allocator only depends on PD leader's leadership, - // so we can directly return here. The election and initialization process - // will happen in server.campaignLeader(). - if dcLocation == GlobalDCLocation { - return - } - // Start election of the Local TSO Allocator here - localTSOAllocator, _ := allocator.(*LocalTSOAllocator) - go am.allocatorLeaderLoop(parentCtx, localTSOAllocator) -} - func (am *AllocatorManager) getAllocatorPath(dcLocation string) string { // For backward compatibility, the global timestamp's store path will still use the old one if dcLocation == GlobalDCLocation { @@ -519,7 +608,7 @@ func (am *AllocatorManager) campaignAllocatorLeader( } } }) - if err := allocator.CampaignAllocatorLeader(defaultAllocatorLeaderLease, cmps...); err != nil { + if err := allocator.CampaignAllocatorLeader(am.leaderLease, cmps...); err != nil { if err.Error() == errs.ErrEtcdTxnConflict.Error() { log.Info("failed to campaign local tso allocator leader due to txn conflict, another allocator may campaign successfully", zap.String("dc-location", allocator.GetDCLocation()), @@ -575,7 +664,7 @@ func (am *AllocatorManager) campaignAllocatorLeader( zap.Any("dc-location-info", dcLocationInfo), zap.String("name", am.member.Name())) - leaderTicker := time.NewTicker(leaderTickInterval) + leaderTicker := time.NewTicker(mcsutils.LeaderTickInterval) defer leaderTicker.Stop() for { @@ -601,7 +690,9 @@ func (am *AllocatorManager) campaignAllocatorLeader( // AllocatorDaemon is used to update every allocator's TSO and check whether we have // any new local allocator that needs to be set up. -func (am *AllocatorManager) AllocatorDaemon(serverCtx context.Context) { +func (am *AllocatorManager) AllocatorDaemon(ctx context.Context) { + log.Info("entering into allocator daemon") + // allocatorPatroller should only work when enableLocalTSO is true to // set up the new Local TSO Allocator in time. var patrolTicker = &time.Ticker{} @@ -610,6 +701,10 @@ func (am *AllocatorManager) AllocatorDaemon(serverCtx context.Context) { defer patrolTicker.Stop() } tsTicker := time.NewTicker(am.updatePhysicalInterval) + failpoint.Inject("fastUpdatePhysicalInterval", func() { + tsTicker.Stop() + tsTicker = time.NewTicker(time.Millisecond) + }) defer tsTicker.Stop() checkerTicker := time.NewTicker(PriorityCheck) defer checkerTicker.Stop() @@ -618,7 +713,7 @@ func (am *AllocatorManager) AllocatorDaemon(serverCtx context.Context) { select { case <-patrolTicker.C: // Inspect the cluster dc-location info and set up the new Local TSO Allocator in time. - am.allocatorPatroller(serverCtx) + am.allocatorPatroller(ctx) case <-tsTicker.C: // Update the initialized TSO Allocator to advance TSO. am.allocatorUpdater() @@ -632,7 +727,8 @@ func (am *AllocatorManager) AllocatorDaemon(serverCtx context.Context) { } // PS: ClusterDCLocationChecker and PriorityChecker are time consuming and low frequent to run, // we should run them concurrently to speed up the progress. - case <-serverCtx.Done(): + case <-ctx.Done(): + log.Info("exit allocator daemon") return } } @@ -654,10 +750,12 @@ func (am *AllocatorManager) allocatorUpdater() { func (am *AllocatorManager) updateAllocator(ag *allocatorGroup) { defer logutil.LogPanic() defer am.wg.Done() + select { case <-ag.ctx.Done(): // Resetting the allocator will clear TSO in memory ag.allocator.Reset() + log.Info("exit the allocator update loop") return default: } @@ -667,7 +765,10 @@ func (am *AllocatorManager) updateAllocator(ag *allocatorGroup) { return } if err := ag.allocator.UpdateTSO(); err != nil { - log.Warn("failed to update allocator's timestamp", zap.String("dc-location", ag.dcLocation), errs.ZapError(err)) + log.Warn("failed to update allocator's timestamp", + zap.String("dc-location", ag.dcLocation), + zap.String("name", am.member.Name()), + errs.ZapError(err)) am.ResetAllocatorGroup(ag.dcLocation) return } @@ -685,7 +786,7 @@ func (am *AllocatorManager) allocatorPatroller(serverCtx context.Context) { if slice.NoneOf(allocatorGroups, func(i int) bool { return allocatorGroups[i].dcLocation == dcLocation }) { - am.SetUpAllocator(serverCtx, dcLocation, election.NewLeadership( + am.setUpLocalAllocator(serverCtx, dcLocation, election.NewLeadership( am.member.Client(), am.getAllocatorPath(dcLocation), fmt.Sprintf("%s local allocator leader election", dcLocation), @@ -733,14 +834,15 @@ func (am *AllocatorManager) ClusterDCLocationChecker() { } } // Only leader can write the TSO suffix to etcd in order to make it consistent in the cluster - if am.member.IsLeader() { + if am.IsLeader() { for dcLocation, info := range am.mu.clusterDCLocations { if info.Suffix > 0 { continue } suffix, err := am.getOrCreateLocalTSOSuffix(dcLocation) if err != nil { - log.Warn("get or create the local tso suffix failed", zap.String("dc-location", dcLocation), errs.ZapError(err)) + log.Warn("get or create the local tso suffix failed", + zap.String("dc-location", dcLocation), errs.ZapError(err)) continue } if suffix > am.mu.maxSuffix { @@ -901,8 +1003,7 @@ func (am *AllocatorManager) PriorityChecker() { nextLeader, err := am.getNextLeaderID(allocatorGroup.dcLocation) if err != nil { log.Error("get next leader from etcd failed", - zap.String("dc-location", allocatorGroup.dcLocation), - errs.ZapError(err)) + zap.String("dc-location", allocatorGroup.dcLocation), errs.ZapError(err)) continue } // nextLeader is not empty and isn't same with the server ID, resign the leader @@ -985,9 +1086,9 @@ func (am *AllocatorManager) deleteAllocatorGroup(dcLocation string) { } } -// HandleTSORequest forwards TSO allocation requests to correct TSO Allocators. -func (am *AllocatorManager) HandleTSORequest(dcLocation string, count uint32) (pdpb.Timestamp, error) { - if dcLocation == "" { +// HandleRequest forwards TSO allocation requests to correct TSO Allocators. +func (am *AllocatorManager) HandleRequest(dcLocation string, count uint32) (pdpb.Timestamp, error) { + if len(dcLocation) == 0 { dcLocation = GlobalDCLocation } allocatorGroup, exist := am.getAllocatorGroup(dcLocation) @@ -995,6 +1096,7 @@ func (am *AllocatorManager) HandleTSORequest(dcLocation string, count uint32) (p err := errs.ErrGetAllocator.FastGenByArgs(fmt.Sprintf("%s allocator not found, generate timestamp failed", dcLocation)) return pdpb.Timestamp{}, err } + return allocatorGroup.allocator.GenerateTSO(count) } @@ -1106,7 +1208,7 @@ func (am *AllocatorManager) getOrCreateGRPCConn(ctx context.Context, addr string } func (am *AllocatorManager) getDCLocationInfoFromLeader(ctx context.Context, dcLocation string) (bool, *pdpb.GetDCLocationInfoResponse, error) { - if am.member.IsLeader() { + if am.IsLeader() { info, ok := am.GetDCLocationInfo(dcLocation) if !ok { return false, &pdpb.GetDCLocationInfoResponse{}, nil @@ -1119,11 +1221,11 @@ func (am *AllocatorManager) getDCLocationInfoFromLeader(ctx context.Context, dcL return ok, dcLocationInfo, nil } - leaderAddrs := am.member.GetLeaderListenUrls() - if leaderAddrs == nil || len(leaderAddrs) < 1 { + leaderAddr := am.GetLeaderAddr() + if len(leaderAddr) < 1 { return false, &pdpb.GetDCLocationInfoResponse{}, fmt.Errorf("failed to get leader client url") } - conn, err := am.getOrCreateGRPCConn(ctx, leaderAddrs[0]) + conn, err := am.getOrCreateGRPCConn(ctx, leaderAddr) if err != nil { return false, &pdpb.GetDCLocationInfoResponse{}, err } @@ -1181,7 +1283,9 @@ func (am *AllocatorManager) setGRPCConn(newConn *grpc.ClientConn, addr string) { defer am.localAllocatorConn.Unlock() if _, ok := am.localAllocatorConn.clientConns[addr]; ok { newConn.Close() - log.Debug("use old connection", zap.String("target", newConn.Target()), zap.String("state", newConn.GetState().String())) + log.Debug("use old connection", + zap.String("target", newConn.Target()), + zap.String("state", newConn.GetState().String())) return } am.localAllocatorConn.clientConns[addr] = newConn @@ -1197,7 +1301,8 @@ func (am *AllocatorManager) transferLocalAllocator(dcLocation string, serverID u if err != nil { err = errs.ErrEtcdGrantLease.Wrap(err).GenWithStackByCause() log.Error("failed to grant the lease of the next leader key", - zap.String("dc-location", dcLocation), zap.Uint64("serverID", serverID), + zap.String("dc-location", dcLocation), + zap.Uint64("serverID", serverID), errs.ZapError(err)) return err } @@ -1227,3 +1332,23 @@ func (am *AllocatorManager) nextLeaderKey(dcLocation string) string { func (am *AllocatorManager) EnableLocalTSO() bool { return am.enableLocalTSO } + +// IsLeader returns whether the current member is the leader in the election group. +func (am *AllocatorManager) IsLeader() bool { + if am == nil || am.member == nil || !am.member.IsLeader() { + return false + } + return true +} + +// GetLeaderAddr returns the address of leader in the election group. +func (am *AllocatorManager) GetLeaderAddr() string { + if am == nil || am.member == nil { + return "" + } + leaderAddrs := am.member.GetLeaderListenUrls() + if len(leaderAddrs) < 1 { + return "" + } + return leaderAddrs[0] +} diff --git a/pkg/tso/config.go b/pkg/tso/config.go new file mode 100644 index 00000000000..598f76004b1 --- /dev/null +++ b/pkg/tso/config.go @@ -0,0 +1,51 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tso + +import ( + "time" + + "github.com/tikv/pd/pkg/utils/grpcutil" +) + +// ServiceConfig defines the configuration interface for the TSO service. +type ServiceConfig interface { + // GetName returns the Name + GetName() string + // GeBackendEndpoints returns the BackendEndpoints + GeBackendEndpoints() string + // GetListenAddr returns the ListenAddr + GetListenAddr() string + // GetAdvertiseListenAddr returns the AdvertiseListenAddr + GetAdvertiseListenAddr() string + // TSO-related configuration + Config +} + +// Config is used to provide TSO configuration. +type Config interface { + // GetLeaderLease returns the leader lease. + GetLeaderLease() int64 + // IsLocalTSOEnabled returns if the local TSO is enabled. + IsLocalTSOEnabled() bool + // GetTSOUpdatePhysicalInterval returns TSO update physical interval. + GetTSOUpdatePhysicalInterval() time.Duration + // GetTSOSaveInterval returns TSO save interval. + GetTSOSaveInterval() time.Duration + // GetMaxResetTSGap returns the MaxResetTSGap. + GetMaxResetTSGap() time.Duration + // GetTLSConfig returns the TLS config. + GetTLSConfig() *grpcutil.TLSConfig +} diff --git a/pkg/tso/global_allocator.go b/pkg/tso/global_allocator.go index 7402859c635..1d961fd1b95 100644 --- a/pkg/tso/global_allocator.go +++ b/pkg/tso/global_allocator.go @@ -16,17 +16,18 @@ package tso import ( "context" + "errors" "fmt" + "path" "sync" "sync/atomic" "time" - "github.com/pingcap/errors" "github.com/pingcap/failpoint" "github.com/pingcap/kvproto/pkg/pdpb" "github.com/pingcap/log" - "github.com/tikv/pd/pkg/election" "github.com/tikv/pd/pkg/errs" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/slice" "github.com/tikv/pd/pkg/utils/logutil" "github.com/tikv/pd/pkg/utils/tsoutil" @@ -45,6 +46,14 @@ type Allocator interface { IsInitialize() bool // UpdateTSO is used to update the TSO in memory and the time window in etcd. UpdateTSO() error + // GetTimestampPath returns the timestamp path in etcd, which is: + // 1. for the default keyspace group: + // a. timestamp in /pd/{cluster_id}/timestamp + // b. lta/{dc-location}/timestamp in /pd/{cluster_id}/lta/{dc-location}/timestamp + // 1. for the non-default keyspace groups: + // a. {group}/gts/timestamp in /ms/{cluster_id}/tso/{group}/gta/timestamp + // b. {group}/lts/{dc-location}/timestamp in /ms/{cluster_id}/tso/{group}/lta/{dc-location}/timestamp + GetTimestampPath() string // SetTSO sets the physical part with given TSO. It's mainly used for BR restore. // Cannot set the TSO smaller than now in any case. // if ignoreSmaller=true, if input ts is smaller than current, ignore silently, else return error @@ -59,11 +68,14 @@ type Allocator interface { // GlobalTSOAllocator is the global single point TSO allocator. type GlobalTSOAllocator struct { + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup + // for global TSO synchronization - allocatorManager *AllocatorManager - // leadership is used to check the current PD server's leadership - // to determine whether a TSO request could be processed. - leadership *election.Leadership + am *AllocatorManager + // for election use + member ElectionMember timestampOracle *timestampOracle // syncRTT is the RTT duration a SyncMaxTS RPC call will cost, // which is used to estimate the MaxTS in a Global TSO generation @@ -73,16 +85,30 @@ type GlobalTSOAllocator struct { // NewGlobalTSOAllocator creates a new global TSO allocator. func NewGlobalTSOAllocator( + ctx context.Context, am *AllocatorManager, - leadership *election.Leadership, + startGlobalLeaderLoop bool, ) Allocator { + // Construct the timestampOracle path prefix, which is: + // 1. for the default keyspace group: + // "" in /pd/{cluster_id}/timestamp + // 2. for the non-default keyspace groups: + // {group}/gta in /ms/{cluster_id}/tso/{group}/gta/timestamp + tsPath := "" + if am.kgID != mcsutils.DefaultKeyspaceGroupID { + tsPath = path.Join(fmt.Sprintf("%05d", am.kgID), globalTSOAllocatorEtcdPrefix) + } + + ctx, cancel := context.WithCancel(ctx) gta := &GlobalTSOAllocator{ - allocatorManager: am, - leadership: leadership, + ctx: ctx, + cancel: cancel, + am: am, + member: am.member, timestampOracle: ×tampOracle{ - client: leadership.GetClient(), + client: am.member.GetLeadership().GetClient(), rootPath: am.rootPath, - ltsPath: "", + tsPath: tsPath, storage: am.storage, saveInterval: am.saveInterval, updatePhysicalInterval: am.updatePhysicalInterval, @@ -91,9 +117,22 @@ func NewGlobalTSOAllocator( tsoMux: &tsoObject{}, }, } + + if startGlobalLeaderLoop { + gta.wg.Add(1) + go gta.primaryElectionLoop() + } + return gta } +// close is used to shutdown the primary election loop. +// tso service call this function to shutdown the loop here, but pd manages its own loop. +func (gta *GlobalTSOAllocator) close() { + gta.cancel() + gta.wg.Wait() +} + func (gta *GlobalTSOAllocator) setSyncRTT(rtt int64) { gta.syncRTT.Store(rtt) tsoGauge.WithLabelValues("global_tso_sync_rtt", gta.timestampOracle.dcLocation).Set(float64(rtt)) @@ -107,6 +146,14 @@ func (gta *GlobalTSOAllocator) getSyncRTT() int64 { return syncRTT.(int64) } +// GetTimestampPath returns the timestamp path in etcd. +func (gta *GlobalTSOAllocator) GetTimestampPath() string { + if gta == nil || gta.timestampOracle == nil { + return "" + } + return gta.timestampOracle.GetTimestampPath() +} + func (gta *GlobalTSOAllocator) estimateMaxTS(count uint32, suffixBits int) (*pdpb.Timestamp, bool, error) { physical, logical, lastUpdateTime := gta.timestampOracle.generateTSO(int64(count), 0) if physical == 0 { @@ -130,7 +177,7 @@ func (gta *GlobalTSOAllocator) Initialize(int) error { tsoAllocatorRole.WithLabelValues(gta.timestampOracle.dcLocation).Set(1) // The suffix of a Global TSO should always be 0. gta.timestampOracle.suffix = 0 - return gta.timestampOracle.SyncTimestamp(gta.leadership) + return gta.timestampOracle.SyncTimestamp(gta.member.GetLeadership()) } // IsInitialize is used to indicates whether this allocator is initialized. @@ -140,12 +187,12 @@ func (gta *GlobalTSOAllocator) IsInitialize() bool { // UpdateTSO is used to update the TSO in memory and the time window in etcd. func (gta *GlobalTSOAllocator) UpdateTSO() error { - return gta.timestampOracle.UpdateTimestamp(gta.leadership) + return gta.timestampOracle.UpdateTimestamp(gta.member.GetLeadership()) } // SetTSO sets the physical part with given TSO. func (gta *GlobalTSOAllocator) SetTSO(tso uint64, ignoreSmaller, skipUpperBoundCheck bool) error { - return gta.timestampOracle.resetUserTimestampInner(gta.leadership, tso, ignoreSmaller, skipUpperBoundCheck) + return gta.timestampOracle.resetUserTimestampInner(gta.member.GetLeadership(), tso, ignoreSmaller, skipUpperBoundCheck) } // GenerateTSO is used to generate the given number of TSOs. @@ -159,21 +206,21 @@ func (gta *GlobalTSOAllocator) SetTSO(tso uint64, ignoreSmaller, skipUpperBoundC // 2. Estimate a MaxTS and try to write it to all Local TSO Allocator leaders directly to reduce the RTT. // During the process, if the estimated MaxTS is not accurate, it will fallback to the collecting way. func (gta *GlobalTSOAllocator) GenerateTSO(count uint32) (pdpb.Timestamp, error) { - if !gta.leadership.Check() { + if !gta.member.GetLeadership().Check() { tsoCounter.WithLabelValues("not_leader", gta.timestampOracle.dcLocation).Inc() return pdpb.Timestamp{}, errs.ErrGenerateTimestamp.FastGenByArgs(fmt.Sprintf("requested pd %s of cluster", errs.NotLeaderErr)) } // To check if we have any dc-location configured in the cluster - dcLocationMap := gta.allocatorManager.GetClusterDCLocations() + dcLocationMap := gta.am.GetClusterDCLocations() // No dc-locations configured in the cluster, use the normal Global TSO generation way. // (without synchronization with other Local TSO Allocators) if len(dcLocationMap) == 0 { - return gta.timestampOracle.getTS(gta.leadership, count, 0) + return gta.timestampOracle.getTS(gta.member.GetLeadership(), count, 0) } // Have dc-locations configured in the cluster, use the Global TSO generation way. // (whit synchronization with other Local TSO Allocators) - ctx, cancel := context.WithCancel(context.Background()) + ctx, cancel := context.WithCancel(gta.ctx) defer cancel() for i := 0; i < maxRetryCount; i++ { var ( @@ -181,7 +228,7 @@ func (gta *GlobalTSOAllocator) GenerateTSO(count uint32) (pdpb.Timestamp, error) shouldRetry, skipCheck bool globalTSOResp pdpb.Timestamp estimatedMaxTSO *pdpb.Timestamp - suffixBits = gta.allocatorManager.GetSuffixBits() + suffixBits = gta.am.GetSuffixBits() ) // TODO: add a switch to control whether to enable the MaxTSO estimation. // 1. Estimate a MaxTS among all Local TSO Allocator leaders according to the RTT. @@ -217,7 +264,7 @@ func (gta *GlobalTSOAllocator) GenerateTSO(count uint32) (pdpb.Timestamp, error) skipCheck = true goto SETTING_PHASE } - // Is skipCheck is false and globalTSOResp remains the same, it means the estimatedTSO is valide. + // Is skipCheck is false and globalTSOResp remains the same, it means the estimatedTSO is valid. if !skipCheck && tsoutil.CompareTimestamp(&globalTSOResp, estimatedMaxTSO) == 0 { tsoCounter.WithLabelValues("global_tso_estimate", gta.timestampOracle.dcLocation).Inc() } @@ -230,19 +277,19 @@ func (gta *GlobalTSOAllocator) GenerateTSO(count uint32) (pdpb.Timestamp, error) if tsoutil.CompareTimestamp(currentGlobalTSO, &globalTSOResp) < 0 { tsoCounter.WithLabelValues("global_tso_persist", gta.timestampOracle.dcLocation).Inc() // Update the Global TSO in memory - if err = gta.timestampOracle.resetUserTimestamp(gta.leadership, tsoutil.GenerateTS(&globalTSOResp), true); err != nil { + if err = gta.timestampOracle.resetUserTimestamp(gta.member.GetLeadership(), tsoutil.GenerateTS(&globalTSOResp), true); err != nil { tsoCounter.WithLabelValues("global_tso_persist_err", gta.timestampOracle.dcLocation).Inc() log.Error("global tso allocator update the global tso in memory failed", errs.ZapError(err)) continue } } // 5. Check leadership again before we returning the response. - if !gta.leadership.Check() { + if !gta.member.GetLeadership().Check() { tsoCounter.WithLabelValues("not_leader_anymore", gta.timestampOracle.dcLocation).Inc() - return pdpb.Timestamp{}, errs.ErrGenerateTimestamp.FastGenByArgs("not the pd leader anymore") + return pdpb.Timestamp{}, errs.ErrGenerateTimestamp.FastGenByArgs(fmt.Sprintf("requested %s anymore", errs.NotLeaderErr)) } - // 6. Differentiate the logical part to make the TSO unique globally by giving it a unique suffix in the whole cluster - globalTSOResp.Logical = gta.timestampOracle.differentiateLogical(globalTSOResp.GetLogical(), suffixBits) + // 6. Calibrate the logical part to make the TSO unique globally by giving it a unique suffix in the whole cluster + globalTSOResp.Logical = gta.timestampOracle.calibrateLogical(globalTSOResp.GetLogical(), suffixBits) globalTSOResp.SuffixBits = uint32(suffixBits) return globalTSOResp, nil } @@ -265,7 +312,7 @@ func (gta *GlobalTSOAllocator) precheckLogical(maxTSO *pdpb.Timestamp, suffixBit return false } // Check if the logical part will reach the overflow condition after being differentiated. - if differentiatedLogical := gta.timestampOracle.differentiateLogical(maxTSO.Logical, suffixBits); differentiatedLogical >= maxLogical { + if caliLogical := gta.timestampOracle.calibrateLogical(maxTSO.Logical, suffixBits); caliLogical >= maxLogical { log.Error("estimated logical part outside of max logical interval, please check ntp time", zap.Reflect("max-tso", maxTSO), errs.ZapError(errs.ErrLogicOverflow)) tsoCounter.WithLabelValues("precheck_logical_overflow", gta.timestampOracle.dcLocation).Inc() @@ -302,7 +349,7 @@ func (gta *GlobalTSOAllocator) SyncMaxTS( // Collect all allocator leaders' client URLs allocatorLeaders := make(map[string]*pdpb.Member) for dcLocation := range dcLocationMap { - allocator, err := gta.allocatorManager.GetAllocator(dcLocation) + allocator, err := gta.am.GetAllocator(dcLocation) if err != nil { return err } @@ -328,13 +375,13 @@ func (gta *GlobalTSOAllocator) SyncMaxTS( wg := sync.WaitGroup{} request := &pdpb.SyncMaxTSRequest{ Header: &pdpb.RequestHeader{ - SenderId: gta.allocatorManager.member.ID(), + SenderId: gta.am.member.ID(), }, SkipCheck: skipCheck, MaxTs: maxTSO, } for _, leaderURL := range leaderURLs { - leaderConn, err := gta.allocatorManager.getOrCreateGRPCConn(ctx, leaderURL) + leaderConn, err := gta.am.getOrCreateGRPCConn(ctx, leaderURL) if err != nil { return err } @@ -352,12 +399,15 @@ func (gta *GlobalTSOAllocator) SyncMaxTS( cancel() respCh <- syncMaxTSResp if syncMaxTSResp.err != nil { - log.Error("sync max ts rpc failed, got an error", zap.String("local-allocator-leader-url", leaderConn.Target()), errs.ZapError(err)) + log.Error("sync max ts rpc failed, got an error", + zap.String("local-allocator-leader-url", leaderConn.Target()), + errs.ZapError(err)) return } if syncMaxTSResp.rpcRes.GetHeader().GetError() != nil { - log.Error("sync max ts rpc failed, got an error", zap.String("local-allocator-leader-url", leaderConn.Target()), - errs.ZapError(errors.Errorf("%s", syncMaxTSResp.rpcRes.GetHeader().GetError().String()))) + log.Error("sync max ts rpc failed, got an error", + zap.String("local-allocator-leader-url", leaderConn.Target()), + errs.ZapError(errors.New(syncMaxTSResp.rpcRes.GetHeader().GetError().String()))) return } }(ctx, leaderConn, respCh) @@ -405,15 +455,18 @@ func (gta *GlobalTSOAllocator) SyncMaxTS( } // Check whether all dc-locations have been considered during the synchronization and retry once if any dc-location missed. if ok, unsyncedDCs := gta.checkSyncedDCs(dcLocationMap, syncedDCs); !ok { - log.Info("unsynced dc-locations found, will retry", zap.Bool("skip-check", skipCheck), zap.Strings("synced-DCs", syncedDCs), zap.Strings("unsynced-DCs", unsyncedDCs)) + log.Info("unsynced dc-locations found, will retry", + zap.Bool("skip-check", skipCheck), zap.Strings("synced-DCs", syncedDCs), zap.Strings("unsynced-DCs", unsyncedDCs)) if i < syncMaxRetryCount-1 { // maxTSO should remain the same. *maxTSO = originalMaxTSO // To make sure we have the latest dc-location info - gta.allocatorManager.ClusterDCLocationChecker() + gta.am.ClusterDCLocationChecker() continue } - return errs.ErrSyncMaxTS.FastGenByArgs(fmt.Sprintf("unsynced dc-locations found, skip-check: %t, synced dc-locations: %+v, unsynced dc-locations: %+v", skipCheck, syncedDCs, unsyncedDCs)) + return errs.ErrSyncMaxTS.FastGenByArgs( + fmt.Sprintf("unsynced dc-locations found, skip-check: %t, synced dc-locations: %+v, unsynced dc-locations: %+v", + skipCheck, syncedDCs, unsyncedDCs)) } // Update the sync RTT to help estimate MaxTS later. if maxTSORtt != 0 { @@ -447,3 +500,105 @@ func (gta *GlobalTSOAllocator) Reset() { tsoAllocatorRole.WithLabelValues(gta.timestampOracle.dcLocation).Set(0) gta.timestampOracle.ResetTimestamp() } + +func (gta *GlobalTSOAllocator) primaryElectionLoop() { + defer logutil.LogPanic() + defer gta.wg.Done() + + for { + select { + case <-gta.ctx.Done(): + log.Info("exit the global tso primary election loop") + return + default: + } + + primary, checkAgain := gta.member.CheckLeader() + if checkAgain { + continue + } + if primary != nil { + log.Info("start to watch the primary", + zap.String("campaign-tso-primary-name", gta.member.Name()), + zap.Stringer("tso-primary", primary)) + // Watch will keep looping and never return unless the primary has changed. + primary.Watch(gta.ctx) + log.Info("the tso primary has changed, try to re-campaign a primary") + } + + gta.campaignLeader() + } +} + +func (gta *GlobalTSOAllocator) campaignLeader() { + log.Info("start to campaign the primary", zap.String("campaign-tso-primary-name", gta.member.Name())) + if err := gta.am.member.CampaignLeader(gta.am.leaderLease); err != nil { + if errors.Is(err, errs.ErrEtcdTxnConflict) { + log.Info("campaign tso primary meets error due to txn conflict, another tso server may campaign successfully", + zap.String("campaign-tso-primary-name", gta.member.Name())) + } else if errors.Is(err, errs.ErrCheckCampaign) { + log.Info("campaign tso primary meets error due to pre-check campaign failed, the tso keyspace group may be in split", + zap.String("campaign-tso-primary-name", gta.member.Name())) + } else { + log.Error("campaign tso primary meets error due to etcd error", + zap.String("campaign-tso-primary-name", gta.member.Name()), errs.ZapError(err)) + } + return + } + + // Start keepalive the leadership and enable TSO service. + // TSO service is strictly enabled/disabled by the leader lease for 2 reasons: + // 1. lease based approach is not affected by thread pause, slow runtime schedule, etc. + // 2. load region could be slow. Based on lease we can recover TSO service faster. + ctx, cancel := context.WithCancel(gta.ctx) + var resetLeaderOnce sync.Once + defer resetLeaderOnce.Do(func() { + cancel() + gta.member.ResetLeader() + }) + + // maintain the the leadership, after this, TSO can be service. + gta.member.KeepLeader(ctx) + log.Info("campaign tso primary ok", zap.String("campaign-tso-primary-name", gta.member.Name())) + + allocator, err := gta.am.GetAllocator(GlobalDCLocation) + if err != nil { + log.Error("failed to get the global tso allocator", errs.ZapError(err)) + return + } + log.Info("initializing the global tso allocator") + if err := allocator.Initialize(0); err != nil { + log.Error("failed to initialize the global tso allocator", errs.ZapError(err)) + return + } + defer func() { + gta.am.ResetAllocatorGroup(GlobalDCLocation) + }() + + gta.member.EnableLeader() + defer resetLeaderOnce.Do(func() { + cancel() + gta.member.ResetLeader() + }) + + // TODO: if enable-local-tso is true, check the cluster dc-location after the primary is elected + // go gta.tsoAllocatorManager.ClusterDCLocationChecker() + log.Info("tso primary is ready to serve", zap.String("tso-primary-name", gta.member.Name())) + + leaderTicker := time.NewTicker(mcsutils.LeaderTickInterval) + defer leaderTicker.Stop() + + for { + select { + case <-leaderTicker.C: + if !gta.member.IsLeader() { + log.Info("no longer a primary because lease has expired, the tso primary will step down") + return + } + case <-ctx.Done(): + // Server is closed and it should return nil. + log.Info("exit leader campaign") + return + } + } +} diff --git a/pkg/tso/keyspace_group_manager.go b/pkg/tso/keyspace_group_manager.go new file mode 100644 index 00000000000..4d32ae92c80 --- /dev/null +++ b/pkg/tso/keyspace_group_manager.go @@ -0,0 +1,827 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tso + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "path" + "sort" + "strings" + "sync" + "time" + + perrors "github.com/pingcap/errors" + "github.com/pingcap/kvproto/pkg/pdpb" + "github.com/pingcap/log" + "github.com/tikv/pd/pkg/election" + "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/mcs/discovery" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/member" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/storage/kv" + "github.com/tikv/pd/pkg/utils/apiutil" + "github.com/tikv/pd/pkg/utils/etcdutil" + "github.com/tikv/pd/pkg/utils/logutil" + "github.com/tikv/pd/pkg/utils/memberutil" + "github.com/tikv/pd/pkg/utils/tsoutil" + "go.etcd.io/etcd/clientv3" + "go.etcd.io/etcd/mvcc/mvccpb" + "go.uber.org/zap" +) + +const ( + // primaryElectionSuffix is the suffix of the key for keyspace group primary election + primaryElectionSuffix = "primary" + defaultRetryInterval = 500 * time.Millisecond +) + +type state struct { + sync.RWMutex + // ams stores the allocator managers of the keyspace groups. Each keyspace group is + // assigned with an allocator manager managing its global/local tso allocators. + // Use a fixed size array to maximize the efficiency of concurrent access to + // different keyspace groups for tso service. + ams [mcsutils.MaxKeyspaceGroupCountInUse]*AllocatorManager + // kgs stores the keyspace groups' membership/distribution meta. + kgs [mcsutils.MaxKeyspaceGroupCountInUse]*endpoint.KeyspaceGroup + // keyspaceLookupTable is a map from keyspace to the keyspace group to which it belongs. + keyspaceLookupTable map[uint32]uint32 +} + +func (s *state) initialize() { + s.keyspaceLookupTable = make(map[uint32]uint32) +} + +func (s *state) deinitialize() { + log.Info("closing all keyspace groups") + + s.Lock() + defer s.Unlock() + + wg := sync.WaitGroup{} + for _, am := range s.ams { + if am != nil { + wg.Add(1) + go func(am *AllocatorManager) { + defer logutil.LogPanic() + defer wg.Done() + am.close() + log.Info("keyspace group closed", zap.Uint32("keyspace-group-id", am.kgID)) + }(am) + } + } + wg.Wait() + + log.Info("all keyspace groups closed") +} + +// getKeyspaceGroupMeta returns the meta of the given keyspace group +func (s *state) getKeyspaceGroupMeta( + groupID uint32, +) (*AllocatorManager, *endpoint.KeyspaceGroup) { + s.RLock() + defer s.RUnlock() + return s.ams[groupID], s.kgs[groupID] +} + +// getKeyspaceGroupMetaWithCheck returns the keyspace group meta of the given keyspace. +// It also checks if the keyspace is served by the given keyspace group. If not, it returns the meta +// of the keyspace group to which the keyspace currently belongs and returns NotServed (by the given +// keyspace group) error. If the keyspace doesn't belong to any keyspace group, it returns the +// NotAssigned error, which could happen because loading keyspace group meta isn't atomic when there is +// keyspace movement between keyspace groups. +func (s *state) getKeyspaceGroupMetaWithCheck( + keyspaceID, keyspaceGroupID uint32, +) (*AllocatorManager, *endpoint.KeyspaceGroup, uint32, error) { + s.RLock() + defer s.RUnlock() + + if am := s.ams[keyspaceGroupID]; am != nil { + kg := s.kgs[keyspaceGroupID] + if kg != nil { + if _, ok := kg.KeyspaceLookupTable[keyspaceID]; ok { + return am, kg, keyspaceGroupID, nil + } + } + } + + // The keyspace doesn't belong to this keyspace group, we should check if it belongs to any other + // keyspace groups, and return the correct keyspace group meta to the client. + if kgid, ok := s.keyspaceLookupTable[keyspaceID]; ok { + if s.ams[kgid] != nil { + return s.ams[kgid], s.kgs[kgid], kgid, nil + } + return nil, s.kgs[kgid], kgid, genNotServedErr(errs.ErrGetAllocatorManager, keyspaceGroupID) + } + + // The keyspace doesn't belong to any keyspace group but the keyspace has been assigned to a + // keyspace group before, which means the keyspace group hasn't initialized yet. + if keyspaceGroupID != mcsutils.DefaultKeyspaceGroupID { + return nil, nil, keyspaceGroupID, errs.ErrKeyspaceNotAssigned.FastGenByArgs(keyspaceID) + } + + // For migrating the existing keyspaces which have no keyspace group assigned as configured + // in the keyspace meta. All these keyspaces will be served by the default keyspace group. + if s.ams[mcsutils.DefaultKeyspaceGroupID] == nil { + return nil, nil, mcsutils.DefaultKeyspaceGroupID, + errs.ErrKeyspaceNotAssigned.FastGenByArgs(keyspaceID) + } + return s.ams[mcsutils.DefaultKeyspaceGroupID], + s.kgs[mcsutils.DefaultKeyspaceGroupID], + mcsutils.DefaultKeyspaceGroupID, nil +} + +// KeyspaceGroupManager manages the members of the keyspace groups assigned to this host. +// The replicas campaign for the leaders which provide the tso service for the corresponding +// keyspace groups. +type KeyspaceGroupManager struct { + // state is the in-memory state of the keyspace groups + state + + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup + + // tsoServiceID is the service ID of the TSO service, registered in the service discovery + tsoServiceID *discovery.ServiceRegistryEntry + etcdClient *clientv3.Client + httpClient *http.Client + // electionNamePrefix is the name prefix to generate the unique name of a participant, + // which participate in the election of its keyspace group's primary, in the format of + // "electionNamePrefix:keyspace-group-id" + electionNamePrefix string + // legacySvcRootPath defines the legacy root path for all etcd paths which derives from + // the PD/API service. It's in the format of "/pd/{cluster_id}". + // The main paths for different usages include: + // 1. The path, used by the default keyspace group, for LoadTimestamp/SaveTimestamp in the + // storage endpoint. + // Key: /pd/{cluster_id}/timestamp + // Value: ts(time.Time) + // Key: /pd/{cluster_id}/lta/{dc-location}/timestamp + // Value: ts(time.Time) + // 2. The path for storing keyspace group membership/distribution metadata. + // Key: /pd/{cluster_id}/tso/keyspace_groups/membership/{group} + // Value: endpoint.KeyspaceGroup + // Note: The {group} is 5 digits integer with leading zeros. + legacySvcRootPath string + // tsoSvcRootPath defines the root path for all etcd paths used in the tso microservices. + // It is in the format of "/ms//tso". + // The main paths for different usages include: + // 1. The path for keyspace group primary election. Format: "/ms/{cluster_id}/tso/{group}/primary" + // 2. The path for LoadTimestamp/SaveTimestamp in the storage endpoint for all the non-default + // keyspace groups. + // Key: /ms/{cluster_id}/tso/{group}/gta/timestamp + // Value: ts(time.Time) + // Key: /ms/{cluster_id}/tso/{group}/lta/{dc-location}/timestamp + // Value: ts(time.Time) + // Note: The {group} is 5 digits integer with leading zeros. + tsoSvcRootPath string + // legacySvcStorage is storage with legacySvcRootPath. + legacySvcStorage *endpoint.StorageEndpoint + // tsoSvcStorage is storage with tsoSvcRootPath. + tsoSvcStorage *endpoint.StorageEndpoint + // cfg is the TSO config + cfg ServiceConfig + + // loadKeyspaceGroupsTimeout is the timeout for loading the initial keyspace group assignment. + loadKeyspaceGroupsTimeout time.Duration + loadKeyspaceGroupsBatchSize int64 + loadFromEtcdMaxRetryTimes int + + // groupUpdateRetryList is the list of keyspace groups which failed to update and need to retry. + groupUpdateRetryList map[uint32]*endpoint.KeyspaceGroup + + groupWatcher *etcdutil.LoopWatcher +} + +// NewKeyspaceGroupManager creates a new Keyspace Group Manager. +func NewKeyspaceGroupManager( + ctx context.Context, + tsoServiceID *discovery.ServiceRegistryEntry, + etcdClient *clientv3.Client, + httpClient *http.Client, + electionNamePrefix string, + legacySvcRootPath string, + tsoSvcRootPath string, + cfg ServiceConfig, +) *KeyspaceGroupManager { + if mcsutils.MaxKeyspaceGroupCountInUse > mcsutils.MaxKeyspaceGroupCount { + log.Fatal("MaxKeyspaceGroupCountInUse is larger than MaxKeyspaceGroupCount", + zap.Uint32("max-keyspace-group-count-in-use", mcsutils.MaxKeyspaceGroupCountInUse), + zap.Uint32("max-keyspace-group-count", mcsutils.MaxKeyspaceGroupCount)) + } + + ctx, cancel := context.WithCancel(ctx) + kgm := &KeyspaceGroupManager{ + ctx: ctx, + cancel: cancel, + tsoServiceID: tsoServiceID, + etcdClient: etcdClient, + httpClient: httpClient, + electionNamePrefix: electionNamePrefix, + legacySvcRootPath: legacySvcRootPath, + tsoSvcRootPath: tsoSvcRootPath, + cfg: cfg, + groupUpdateRetryList: make(map[uint32]*endpoint.KeyspaceGroup), + } + kgm.legacySvcStorage = endpoint.NewStorageEndpoint( + kv.NewEtcdKVBase(kgm.etcdClient, kgm.legacySvcRootPath), nil) + kgm.tsoSvcStorage = endpoint.NewStorageEndpoint( + kv.NewEtcdKVBase(kgm.etcdClient, kgm.tsoSvcRootPath), nil) + kgm.state.initialize() + return kgm +} + +// Initialize this KeyspaceGroupManager +func (kgm *KeyspaceGroupManager) Initialize() error { + rootPath := kgm.legacySvcRootPath + startKey := strings.Join([]string{rootPath, endpoint.KeyspaceGroupIDPath(mcsutils.DefaultKeyspaceGroupID)}, "/") + endKey := strings.Join( + []string{rootPath, clientv3.GetPrefixRangeEnd(endpoint.KeyspaceGroupIDPrefix())}, "/") + + defaultKGConfigured := false + putFn := func(kv *mvccpb.KeyValue) error { + group := &endpoint.KeyspaceGroup{} + if err := json.Unmarshal(kv.Value, group); err != nil { + return errs.ErrJSONUnmarshal.Wrap(err).FastGenWithCause() + } + kgm.updateKeyspaceGroup(group) + if group.ID == mcsutils.DefaultKeyspaceGroupID { + defaultKGConfigured = true + } + return nil + } + deleteFn := func(kv *mvccpb.KeyValue) error { + groupID, err := endpoint.ExtractKeyspaceGroupIDFromPath(string(kv.Key)) + if err != nil { + return err + } + kgm.deleteKeyspaceGroup(groupID) + return nil + } + postEventFn := func() error { + // Retry the groups that are not initialized successfully before. + for id, group := range kgm.groupUpdateRetryList { + delete(kgm.groupUpdateRetryList, id) + kgm.updateKeyspaceGroup(group) + } + return nil + } + kgm.groupWatcher = etcdutil.NewLoopWatcher( + kgm.ctx, + &kgm.wg, + kgm.etcdClient, + "keyspace-watcher", + startKey, + putFn, + deleteFn, + postEventFn, + clientv3.WithRange(endKey), + ) + if kgm.loadKeyspaceGroupsTimeout > 0 { + kgm.groupWatcher.SetLoadTimeout(kgm.loadKeyspaceGroupsTimeout) + } + if kgm.loadFromEtcdMaxRetryTimes > 0 { + kgm.groupWatcher.SetLoadRetryTimes(kgm.loadFromEtcdMaxRetryTimes) + } + if kgm.loadKeyspaceGroupsBatchSize > 0 { + kgm.groupWatcher.SetLoadBatchSize(kgm.loadKeyspaceGroupsBatchSize) + } + kgm.wg.Add(1) + go kgm.groupWatcher.StartWatchLoop() + + if err := kgm.groupWatcher.WaitLoad(); err != nil { + log.Error("failed to initialize keyspace group manager", errs.ZapError(err)) + // We might have partially loaded/initialized the keyspace groups. Close the manager to clean up. + kgm.Close() + return errs.ErrLoadKeyspaceGroupsTerminated + } + + if !defaultKGConfigured { + log.Info("initializing default keyspace group") + group := &endpoint.KeyspaceGroup{ + ID: mcsutils.DefaultKeyspaceGroupID, + Members: []endpoint.KeyspaceGroupMember{{Address: kgm.tsoServiceID.ServiceAddr}}, + Keyspaces: []uint32{mcsutils.DefaultKeyspaceID}, + } + kgm.updateKeyspaceGroup(group) + } + return nil +} + +// Close this KeyspaceGroupManager +func (kgm *KeyspaceGroupManager) Close() { + log.Info("closing keyspace group manager") + + // Note: don't change the order. We need to cancel all service loops in the keyspace group manager + // before closing all keyspace groups. It's to prevent concurrent addition/removal of keyspace groups + // during critical periods such as service shutdown and online keyspace group, while the former requires + // snapshot isolation to ensure all keyspace groups are properly closed and no new keyspace group is + // added/initialized after that. + kgm.cancel() + kgm.wg.Wait() + kgm.state.deinitialize() + + log.Info("keyspace group manager closed") +} + +func (kgm *KeyspaceGroupManager) isAssignedToMe(group *endpoint.KeyspaceGroup) bool { + for _, member := range group.Members { + if member.Address == kgm.tsoServiceID.ServiceAddr { + return true + } + } + return false +} + +// updateKeyspaceGroup applies the given keyspace group. If the keyspace group is just assigned to +// this host/pod, it will join the primary election. +func (kgm *KeyspaceGroupManager) updateKeyspaceGroup(group *endpoint.KeyspaceGroup) { + if err := kgm.checkKeySpaceGroupID(group.ID); err != nil { + log.Warn("keyspace group ID is invalid, ignore it", zap.Error(err)) + return + } + + // If the default keyspace group isn't assigned to any tso node/pod, assign it to everyone. + if group.ID == mcsutils.DefaultKeyspaceGroupID && len(group.Members) == 0 { + log.Warn("configured the default keyspace group but no members/distribution specified. " + + "ignore it for now and fallback to the way of every tso node/pod owning a replica") + // TODO: fill members with all tso nodes/pods. + group.Members = []endpoint.KeyspaceGroupMember{{Address: kgm.tsoServiceID.ServiceAddr}} + } + + if !kgm.isAssignedToMe(group) { + // Not assigned to me. If this host/pod owns a replica of this keyspace group, + // it should resign the election membership now. + kgm.exitElectionMembership(group) + return + } + + // If this host is already assigned a replica of this keyspace group, that is to is already initialized, just update the meta. + if oldAM, oldGroup := kgm.getKeyspaceGroupMeta(group.ID); oldAM != nil { + log.Info("keyspace group already initialized, so update meta only", + zap.Uint32("keyspace-group-id", group.ID)) + kgm.updateKeyspaceGroupMembership(oldGroup, group, true) + return + } + + // If the keyspace group is not initialized, initialize it. + uniqueName := fmt.Sprintf("%s-%05d", kgm.electionNamePrefix, group.ID) + uniqueID := memberutil.GenerateUniqueID(uniqueName) + log.Info("joining primary election", + zap.Uint32("keyspace-group-id", group.ID), + zap.String("participant-name", uniqueName), + zap.Uint64("participant-id", uniqueID)) + // Initialize the participant info to join the primary election. + participant := member.NewParticipant(kgm.etcdClient) + participant.InitInfo( + uniqueName, uniqueID, path.Join(kgm.tsoSvcRootPath, fmt.Sprintf("%05d", group.ID)), + primaryElectionSuffix, "keyspace group primary election", kgm.cfg.GetAdvertiseListenAddr()) + // If the keyspace group is in split, we should ensure that the primary elected by the new keyspace group + // is always on the same TSO Server node as the primary of the old keyspace group, and this constraint cannot + // be broken until the entire split process is completed. + if group.IsSplitTarget() { + splitSource := group.SplitSource() + log.Info("keyspace group is in split", + zap.Uint32("target", group.ID), + zap.Uint32("source", splitSource)) + splitSourceAM, splitSourceGroup := kgm.getKeyspaceGroupMeta(splitSource) + if !validateSplit(splitSourceAM, group, splitSourceGroup) { + // Put the group into the retry list to retry later. + kgm.groupUpdateRetryList[group.ID] = group + return + } + participant.SetCampaignChecker(func(leadership *election.Leadership) bool { + return splitSourceAM.GetMember().IsLeader() + }) + } + // Only the default keyspace group uses the legacy service root path for LoadTimestamp/SyncTimestamp. + var ( + tsRootPath string + storage *endpoint.StorageEndpoint + ) + if group.ID == mcsutils.DefaultKeyspaceGroupID { + tsRootPath = kgm.legacySvcRootPath + storage = kgm.legacySvcStorage + } else { + tsRootPath = kgm.tsoSvcRootPath + storage = kgm.tsoSvcStorage + } + // Initialize all kinds of maps. + am := NewAllocatorManager(kgm.ctx, group.ID, participant, tsRootPath, storage, kgm.cfg, true) + log.Info("created allocator manager", + zap.Uint32("keyspace-group-id", group.ID), + zap.String("timestamp-path", am.GetTimestampPath(""))) + kgm.Lock() + group.KeyspaceLookupTable = make(map[uint32]struct{}) + for _, kid := range group.Keyspaces { + group.KeyspaceLookupTable[kid] = struct{}{} + kgm.keyspaceLookupTable[kid] = group.ID + } + kgm.kgs[group.ID] = group + kgm.ams[group.ID] = am + kgm.Unlock() +} + +// validateSplit checks whether the meta info of split keyspace group +// to ensure that the split process could be continued. +func validateSplit( + sourceAM *AllocatorManager, + targetGroup, sourceGroup *endpoint.KeyspaceGroup, +) bool { + splitSourceID := targetGroup.SplitSource() + // Make sure that the split source keyspace group has been initialized. + if sourceAM == nil || sourceGroup == nil { + log.Error("the split source keyspace group is not initialized", + zap.Uint32("target", targetGroup.ID), + zap.Uint32("source", splitSourceID)) + return false + } + // Since the target group is derived from the source group and both of them + // could not be modified during the split process, so we can only check the + // member count of the source group here. + memberCount := len(sourceGroup.Members) + if memberCount < mcsutils.KeyspaceGroupDefaultReplicaCount { + log.Error("the split source keyspace group does not have enough members", + zap.Uint32("target", targetGroup.ID), + zap.Uint32("source", splitSourceID), + zap.Int("member-count", memberCount), + zap.Int("replica-count", mcsutils.KeyspaceGroupDefaultReplicaCount)) + return false + } + return true +} + +// updateKeyspaceGroupMembership updates the keyspace lookup table for the given keyspace group. +func (kgm *KeyspaceGroupManager) updateKeyspaceGroupMembership( + oldGroup, newGroup *endpoint.KeyspaceGroup, updateWithLock bool, +) { + var ( + oldKeyspaces []uint32 + oldKeyspaceLookupTable map[uint32]struct{} + ) + + if oldGroup != nil { + oldKeyspaces = oldGroup.Keyspaces + oldKeyspaceLookupTable = oldGroup.KeyspaceLookupTable + } + + groupID := newGroup.ID + newKeyspaces := newGroup.Keyspaces + oldLen, newLen := len(oldKeyspaces), len(newKeyspaces) + + // Sort the keyspaces in ascending order + sort.Slice(newKeyspaces, func(i, j int) bool { + return newKeyspaces[i] < newKeyspaces[j] + }) + + // Mostly, the membership has no change, so optimize for this case. + sameMembership := true + if oldLen != newLen { + sameMembership = false + } else { + for i := 0; i < oldLen; i++ { + if oldKeyspaces[i] != newKeyspaces[i] { + sameMembership = false + break + } + } + } + + if updateWithLock { + kgm.Lock() + defer kgm.Unlock() + } + + if sameMembership { + // The keyspace group membership is not changed. Reuse the old one. + newGroup.KeyspaceLookupTable = oldKeyspaceLookupTable + } else { + // The keyspace group membership is changed. Update the keyspace lookup table. + newGroup.KeyspaceLookupTable = make(map[uint32]struct{}) + for i, j := 0, 0; i < oldLen || j < newLen; { + if i < oldLen && j < newLen && oldKeyspaces[i] == newKeyspaces[j] { + newGroup.KeyspaceLookupTable[newKeyspaces[j]] = struct{}{} + i++ + j++ + } else if i < oldLen && j < newLen && oldKeyspaces[i] < newKeyspaces[j] || j == newLen { + delete(kgm.keyspaceLookupTable, oldKeyspaces[i]) + i++ + } else { + newGroup.KeyspaceLookupTable[newKeyspaces[j]] = struct{}{} + kgm.keyspaceLookupTable[newKeyspaces[j]] = groupID + j++ + } + } + if groupID == mcsutils.DefaultKeyspaceGroupID { + if _, ok := newGroup.KeyspaceLookupTable[mcsutils.DefaultKeyspaceID]; !ok { + log.Warn("default keyspace is not in default keyspace group. add it back") + kgm.keyspaceLookupTable[mcsutils.DefaultKeyspaceID] = groupID + newGroup.KeyspaceLookupTable[mcsutils.DefaultKeyspaceID] = struct{}{} + newGroup.Keyspaces = make([]uint32, 1+len(newKeyspaces)) + newGroup.Keyspaces[0] = mcsutils.DefaultKeyspaceID + copy(newGroup.Keyspaces[1:], newKeyspaces) + } + } else { + if _, ok := newGroup.KeyspaceLookupTable[mcsutils.DefaultKeyspaceID]; ok { + log.Warn("default keyspace is in non-default keyspace group. remove it") + kgm.keyspaceLookupTable[mcsutils.DefaultKeyspaceID] = mcsutils.DefaultKeyspaceGroupID + delete(newGroup.KeyspaceLookupTable, mcsutils.DefaultKeyspaceID) + newGroup.Keyspaces = newKeyspaces[1:] + } + } + } + // Check if the split is completed. + if oldGroup != nil && oldGroup.IsSplitTarget() && !newGroup.IsSplitting() { + kgm.ams[groupID].GetMember().(*member.Participant).SetCampaignChecker(nil) + } + kgm.kgs[groupID] = newGroup +} + +// deleteKeyspaceGroup deletes the given keyspace group. +func (kgm *KeyspaceGroupManager) deleteKeyspaceGroup(groupID uint32) { + log.Info("delete keyspace group", zap.Uint32("keyspace-group-id", groupID)) + + if groupID == mcsutils.DefaultKeyspaceGroupID { + log.Info("removed default keyspace group meta config from the storage. " + + "now every tso node/pod will initialize it") + group := &endpoint.KeyspaceGroup{ + ID: mcsutils.DefaultKeyspaceGroupID, + Members: []endpoint.KeyspaceGroupMember{{Address: kgm.tsoServiceID.ServiceAddr}}, + Keyspaces: []uint32{mcsutils.DefaultKeyspaceID}, + } + kgm.updateKeyspaceGroup(group) + return + } + + kgm.Lock() + defer kgm.Unlock() + + kg := kgm.kgs[groupID] + if kg != nil { + for _, kid := range kg.Keyspaces { + // if kid == kg.ID, it means the keyspace still belongs to this keyspace group, + // so we decouple the relationship in the global keyspace lookup table. + // if kid != kg.ID, it means the keyspace has been moved to another keyspace group + // which has already declared the ownership of the keyspace. + if kid == kg.ID { + delete(kgm.keyspaceLookupTable, kid) + } + } + kgm.kgs[groupID] = nil + } + + am := kgm.ams[groupID] + if am != nil { + am.close() + kgm.ams[groupID] = nil + } +} + +// exitElectionMembership exits the election membership of the given keyspace group by +// deinitializing the allocator manager, but still keeps the keyspace group info. +func (kgm *KeyspaceGroupManager) exitElectionMembership(group *endpoint.KeyspaceGroup) { + log.Info("resign election membership", zap.Uint32("keyspace-group-id", group.ID)) + + kgm.Lock() + defer kgm.Unlock() + + am := kgm.ams[group.ID] + if am != nil { + am.close() + kgm.ams[group.ID] = nil + } + + oldGroup := kgm.kgs[group.ID] + kgm.updateKeyspaceGroupMembership(oldGroup, group, false) +} + +// GetAllocatorManager returns the AllocatorManager of the given keyspace group +func (kgm *KeyspaceGroupManager) GetAllocatorManager(keyspaceGroupID uint32) (*AllocatorManager, error) { + if err := kgm.checkKeySpaceGroupID(keyspaceGroupID); err != nil { + return nil, err + } + if am, _ := kgm.getKeyspaceGroupMeta(keyspaceGroupID); am != nil { + return am, nil + } + return nil, genNotServedErr(errs.ErrGetAllocatorManager, keyspaceGroupID) +} + +// FindGroupByKeyspaceID returns the keyspace group that contains the keyspace with the given ID. +func (kgm *KeyspaceGroupManager) FindGroupByKeyspaceID( + keyspaceID uint32, +) (*AllocatorManager, *endpoint.KeyspaceGroup, uint32, error) { + curAM, curKeyspaceGroup, curKeyspaceGroupID, err := + kgm.getKeyspaceGroupMetaWithCheck(keyspaceID, mcsutils.DefaultKeyspaceGroupID) + if err != nil { + return nil, nil, curKeyspaceGroupID, err + } + return curAM, curKeyspaceGroup, curKeyspaceGroupID, nil +} + +// GetElectionMember returns the election member of the keyspace group serving the given keyspace. +func (kgm *KeyspaceGroupManager) GetElectionMember( + keyspaceID, keyspaceGroupID uint32, +) (ElectionMember, error) { + if err := kgm.checkKeySpaceGroupID(keyspaceGroupID); err != nil { + return nil, err + } + am, _, _, err := kgm.getKeyspaceGroupMetaWithCheck(keyspaceID, keyspaceGroupID) + if err != nil { + return nil, err + } + return am.GetMember(), nil +} + +// GetKeyspaceGroups returns all keyspace groups managed by the current keyspace group manager. +func (kgm *KeyspaceGroupManager) GetKeyspaceGroups() map[uint32]*endpoint.KeyspaceGroup { + kgm.RLock() + defer kgm.RUnlock() + keyspaceGroups := make(map[uint32]*endpoint.KeyspaceGroup) + for _, keyspaceGroupID := range kgm.keyspaceLookupTable { + if _, ok := keyspaceGroups[keyspaceGroupID]; ok { + continue + } + keyspaceGroups[keyspaceGroupID] = kgm.kgs[keyspaceGroupID] + } + return keyspaceGroups +} + +// HandleTSORequest forwards TSO allocation requests to correct TSO Allocators of the given keyspace group. +func (kgm *KeyspaceGroupManager) HandleTSORequest( + keyspaceID, keyspaceGroupID uint32, + dcLocation string, count uint32, +) (ts pdpb.Timestamp, curKeyspaceGroupID uint32, err error) { + if err := kgm.checkKeySpaceGroupID(keyspaceGroupID); err != nil { + return pdpb.Timestamp{}, keyspaceGroupID, err + } + am, _, curKeyspaceGroupID, err := kgm.getKeyspaceGroupMetaWithCheck(keyspaceID, keyspaceGroupID) + if err != nil { + return pdpb.Timestamp{}, curKeyspaceGroupID, err + } + err = kgm.checkTSOSplit(curKeyspaceGroupID, dcLocation) + if err != nil { + return pdpb.Timestamp{}, curKeyspaceGroupID, err + } + ts, err = am.HandleRequest(dcLocation, count) + return ts, curKeyspaceGroupID, err +} + +func (kgm *KeyspaceGroupManager) checkKeySpaceGroupID(id uint32) error { + if id < mcsutils.MaxKeyspaceGroupCountInUse { + return nil + } + return errs.ErrKeyspaceGroupIDInvalid.FastGenByArgs( + fmt.Sprintf("%d shouldn't >= %d", id, mcsutils.MaxKeyspaceGroupCountInUse)) +} + +// GetMinTS returns the minimum timestamp across all keyspace groups served by this TSO server/pod. +func (kgm *KeyspaceGroupManager) GetMinTS( + dcLocation string, +) (_ pdpb.Timestamp, kgAskedCount, kgTotalCount uint32, err error) { + kgm.RLock() + defer kgm.RUnlock() + + var minTS *pdpb.Timestamp + for i, am := range kgm.ams { + if kgm.kgs[i] != nil { + kgTotalCount++ + } + // If any keyspace group hasn't elected primary, we can't know its current timestamp of + // the group, so as to the min ts across all keyspace groups. Return error in this case. + if am != nil && !am.member.IsLeaderElected() { + return pdpb.Timestamp{}, kgAskedCount, kgTotalCount, errs.ErrGetMinTS.FastGenByArgs("leader is not elected") + } + // Skip the keyspace groups that are not served by this TSO Server/Pod. + if am == nil || !am.IsLeader() { + continue + } + kgAskedCount++ + // Skip the keyspace groups that are split targets, because they always have newer + // time lines than the existing split sources thus won't contribute to the min ts. + if kgm.kgs[i] != nil && kgm.kgs[i].IsSplitTarget() { + continue + } + ts, err := am.HandleRequest(dcLocation, 1) + if err != nil { + return pdpb.Timestamp{}, kgAskedCount, kgTotalCount, err + } + if minTS == nil || tsoutil.CompareTimestamp(&ts, minTS) < 0 { + minTS = &ts + } + } + + if minTS == nil { + // This TSO server/pod is not serving any keyspace group, return an empty timestamp, + // and the client needs to skip the empty timestamps when collecting the min timestamp + // from all TSO servers/pods. + return pdpb.Timestamp{}, kgAskedCount, kgTotalCount, nil + } + + return *minTS, kgAskedCount, kgTotalCount, nil +} + +func genNotServedErr(perr *perrors.Error, keyspaceGroupID uint32) error { + return perr.FastGenByArgs( + fmt.Sprintf( + "requested keyspace group with id %d %s by this host/pod", + keyspaceGroupID, errs.NotServedErr)) +} + +// checkTSOSplit checks if the given keyspace group is in split state, and if so, it will make sure the +// newly split TSO keep consistent with the original one. +func (kgm *KeyspaceGroupManager) checkTSOSplit( + keyspaceGroupID uint32, + dcLocation string, +) error { + splitAM, splitGroup := kgm.getKeyspaceGroupMeta(keyspaceGroupID) + // Only the split target keyspace group needs to check the TSO split. + if !splitGroup.IsSplitTarget() { + return nil + } + splitSource := splitGroup.SplitSource() + splitSourceAM, splitSourceGroup := kgm.getKeyspaceGroupMeta(splitSource) + if splitSourceAM == nil || splitSourceGroup == nil { + log.Error("the split source keyspace group is not initialized", + zap.Uint32("source", splitSource)) + return errs.ErrKeyspaceGroupNotInitialized.FastGenByArgs(splitSource) + } + splitAllocator, err := splitAM.GetAllocator(dcLocation) + if err != nil { + return err + } + splitSourceAllocator, err := splitSourceAM.GetAllocator(dcLocation) + if err != nil { + return err + } + splitTSO, err := splitAllocator.GenerateTSO(1) + if err != nil { + return err + } + splitSourceTSO, err := splitSourceAllocator.GenerateTSO(1) + if err != nil { + return err + } + if tsoutil.CompareTimestamp(&splitSourceTSO, &splitTSO) <= 0 { + return nil + } + // If the split source TSO is greater than the newly split TSO, we need to update the split + // TSO to make sure the following TSO will be greater than the split keyspaces ever had + // in the past. + splitSourceTSO.Physical += 1 + err = splitAllocator.SetTSO(tsoutil.GenerateTS(&splitSourceTSO), true, true) + if err != nil { + return err + } + // Finish the split state. + return kgm.finishSplitKeyspaceGroup(keyspaceGroupID) +} + +const keyspaceGroupsAPIPrefix = "/pd/api/v2/tso/keyspace-groups" + +// Put the code below into the critical section to prevent from sending too many HTTP requests. +func (kgm *KeyspaceGroupManager) finishSplitKeyspaceGroup(id uint32) error { + kgm.Lock() + defer kgm.Unlock() + // Check if the keyspace group is in split state. + splitGroup := kgm.kgs[id] + if !splitGroup.IsSplitTarget() { + return nil + } + // Check if the HTTP client is initialized. + if kgm.httpClient == nil { + return nil + } + statusCode, err := apiutil.DoDelete( + kgm.httpClient, + kgm.cfg.GeBackendEndpoints()+keyspaceGroupsAPIPrefix+fmt.Sprintf("/%d/split", id)) + if err != nil { + return err + } + if statusCode != http.StatusOK { + log.Warn("failed to finish split keyspace group", + zap.Uint32("keyspace-group-id", id), + zap.Int("status-code", statusCode)) + return errs.ErrSendRequest.FastGenByArgs() + } + // Pre-update the split keyspace group split state in memory. + splitGroup.SplitState = nil + kgm.kgs[id] = splitGroup + return nil +} diff --git a/pkg/tso/keyspace_group_manager_test.go b/pkg/tso/keyspace_group_manager_test.go new file mode 100644 index 00000000000..671fc8b73aa --- /dev/null +++ b/pkg/tso/keyspace_group_manager_test.go @@ -0,0 +1,899 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tso + +import ( + "context" + "encoding/json" + "fmt" + "math/rand" + "path" + "reflect" + "sort" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/google/uuid" + "github.com/pingcap/failpoint" + "github.com/stretchr/testify/require" + "github.com/stretchr/testify/suite" + "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/mcs/discovery" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/utils/memberutil" + "github.com/tikv/pd/pkg/utils/testutil" + "go.etcd.io/etcd/clientv3" + "go.etcd.io/etcd/mvcc/mvccpb" + "go.uber.org/goleak" +) + +func TestMain(m *testing.M) { + goleak.VerifyTestMain(m, testutil.LeakOptions...) +} + +type keyspaceGroupManagerTestSuite struct { + suite.Suite + ctx context.Context + cancel context.CancelFunc + backendEndpoints string + etcdClient *clientv3.Client + clean func() + cfg *TestServiceConfig +} + +func TestKeyspaceGroupManagerTestSuite(t *testing.T) { + suite.Run(t, new(keyspaceGroupManagerTestSuite)) +} + +func (suite *keyspaceGroupManagerTestSuite) SetupSuite() { + t := suite.T() + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + suite.backendEndpoints, suite.etcdClient, suite.clean = startEmbeddedEtcd(t) + + suite.cfg = &TestServiceConfig{ + Name: "tso-test-name", + BackendEndpoints: suite.backendEndpoints, + ListenAddr: "http://127.0.0.1:3379", + AdvertiseListenAddr: "http://127.0.0.1:3379", + LeaderLease: mcsutils.DefaultLeaderLease, + LocalTSOEnabled: false, + TSOUpdatePhysicalInterval: 50 * time.Millisecond, + TSOSaveInterval: time.Duration(mcsutils.DefaultLeaderLease) * time.Second, + MaxResetTSGap: time.Hour * 24, + TLSConfig: nil, + } +} + +func (suite *keyspaceGroupManagerTestSuite) TearDownSuite() { + suite.clean() + suite.cancel() +} + +// TestNewKeyspaceGroupManager tests the initialization of KeyspaceGroupManager. +// It should initialize the allocator manager with the desired configurations and parameters. +func (suite *keyspaceGroupManagerTestSuite) TestNewKeyspaceGroupManager() { + re := suite.Require() + + tsoServiceID := &discovery.ServiceRegistryEntry{ServiceAddr: suite.cfg.AdvertiseListenAddr} + guid := uuid.New().String() + legacySvcRootPath := path.Join("/pd", guid) + tsoSvcRootPath := path.Join("/ms", guid, "tso") + electionNamePrefix := "tso-server-" + guid + + kgm := suite.newKeyspaceGroupManager(tsoServiceID, electionNamePrefix, legacySvcRootPath, tsoSvcRootPath) + err := kgm.Initialize() + re.NoError(err) + + re.Equal(tsoServiceID, kgm.tsoServiceID) + re.Equal(suite.etcdClient, kgm.etcdClient) + re.Equal(electionNamePrefix, kgm.electionNamePrefix) + re.Equal(legacySvcRootPath, kgm.legacySvcRootPath) + re.Equal(tsoSvcRootPath, kgm.tsoSvcRootPath) + re.Equal(suite.cfg, kgm.cfg) + + am, err := kgm.GetAllocatorManager(mcsutils.DefaultKeyspaceGroupID) + re.NoError(err) + re.False(am.enableLocalTSO) + re.Equal(mcsutils.DefaultKeyspaceGroupID, am.kgID) + re.Equal(mcsutils.DefaultLeaderLease, am.leaderLease) + re.Equal(time.Hour*24, am.maxResetTSGap()) + re.Equal(legacySvcRootPath, am.rootPath) + re.Equal(time.Duration(mcsutils.DefaultLeaderLease)*time.Second, am.saveInterval) + re.Equal(time.Duration(50)*time.Millisecond, am.updatePhysicalInterval) + + kgm.Close() +} + +// TestLoadKeyspaceGroupsAssignment tests the loading of the keyspace group assignment. +func (suite *keyspaceGroupManagerTestSuite) TestLoadKeyspaceGroupsAssignment() { + re := suite.Require() + maxCountInUse := 512 + // Test loading of empty keyspace group assignment. + suite.runTestLoadKeyspaceGroupsAssignment(re, 0, 0, 100) + // Test loading of single keyspace group assignment. + suite.runTestLoadKeyspaceGroupsAssignment(re, 1, 0, 100) + // Test loading of multiple keyspace group assignment. + suite.runTestLoadKeyspaceGroupsAssignment(re, 3, 0, 100) + suite.runTestLoadKeyspaceGroupsAssignment(re, maxCountInUse-1, 0, 10) + suite.runTestLoadKeyspaceGroupsAssignment(re, maxCountInUse, 0, 10) + // Test loading of the keyspace group assignment which exceeds the maximum keyspace group count. + // In this case, the manager should only load/serve the first MaxKeyspaceGroupCountInUse keyspace + // groups and ignore the rest. + suite.runTestLoadKeyspaceGroupsAssignment(re, maxCountInUse+1, 0, 10) +} + +// TestLoadWithDifferentBatchSize tests the loading of the keyspace group assignment with the different batch size. +func (suite *keyspaceGroupManagerTestSuite) TestLoadWithDifferentBatchSize() { + re := suite.Require() + + batchSize := int64(17) + maxCount := uint32(1024) + params := []struct { + batchSize int64 + count int + probabilityAssignToMe int // percentage of assigning keyspace groups to this host/pod + }{ + {batchSize: 1, count: 1, probabilityAssignToMe: 100}, + {batchSize: 2, count: int(maxCount / 10), probabilityAssignToMe: 100}, + {batchSize: 7, count: int(maxCount / 10), probabilityAssignToMe: 100}, + {batchSize: batchSize, count: int(batchSize), probabilityAssignToMe: 50}, + {batchSize: int64(maxCount / 13), count: int(maxCount / 13), probabilityAssignToMe: 50}, + {batchSize: int64(maxCount), count: int(maxCount / 13), probabilityAssignToMe: 10}, + } + + for _, param := range params { + suite.runTestLoadKeyspaceGroupsAssignment(re, param.count-1, param.batchSize, param.probabilityAssignToMe) + suite.runTestLoadKeyspaceGroupsAssignment(re, param.count, param.batchSize, param.probabilityAssignToMe) + suite.runTestLoadKeyspaceGroupsAssignment(re, param.count+1, param.batchSize, param.probabilityAssignToMe) + } +} + +// TestLoadKeyspaceGroupsTimeout tests there is timeout when loading the initial keyspace group assignment +// from etcd. The initialization of the keyspace group manager should fail. +func (suite *keyspaceGroupManagerTestSuite) TestLoadKeyspaceGroupsTimeout() { + re := suite.Require() + + mgr := suite.newUniqueKeyspaceGroupManager(1) + re.NotNil(mgr) + defer mgr.Close() + + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, true, + mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, uint32(0), []uint32{0}) + + // Set the timeout to 1 second and inject the delayLoad to return 3 seconds to let + // the loading sleep 3 seconds. + mgr.loadKeyspaceGroupsTimeout = time.Second + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/utils/etcdutil/delayLoad", "return(3)")) + err := mgr.Initialize() + // If loading keyspace groups timeout, the initialization should fail with ErrLoadKeyspaceGroupsTerminated. + re.Equal(errs.ErrLoadKeyspaceGroupsTerminated, err) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/delayLoad")) +} + +// TestLoadKeyspaceGroupsSucceedWithTempFailures tests the initialization should succeed when there are temporary +// failures during loading the initial keyspace group assignment from etcd. +func (suite *keyspaceGroupManagerTestSuite) TestLoadKeyspaceGroupsSucceedWithTempFailures() { + re := suite.Require() + + mgr := suite.newUniqueKeyspaceGroupManager(1) + re.NotNil(mgr) + defer mgr.Close() + + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, true, + mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, uint32(0), []uint32{0}) + + // Set the max retry times to 3 and inject the loadTemporaryFail to return 2 to let + // loading from etcd fail 2 times but the whole initialization still succeeds. + mgr.loadFromEtcdMaxRetryTimes = 3 + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/utils/etcdutil/loadTemporaryFail", "return(2)")) + err := mgr.Initialize() + re.NoError(err) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/loadTemporaryFail")) +} + +// TestLoadKeyspaceGroupsFailed tests the initialization should fail when there are too many failures +// during loading the initial keyspace group assignment from etcd. +func (suite *keyspaceGroupManagerTestSuite) TestLoadKeyspaceGroupsFailed() { + re := suite.Require() + + mgr := suite.newUniqueKeyspaceGroupManager(1) + re.NotNil(mgr) + defer mgr.Close() + + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, true, + mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, uint32(0), []uint32{0}) + + // Set the max retry times to 3 and inject the loadTemporaryFail to return 3 to let + // loading from etcd fail 3 times which should cause the whole initialization to fail. + mgr.loadFromEtcdMaxRetryTimes = 3 + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/utils/etcdutil/loadTemporaryFail", "return(3)")) + err := mgr.Initialize() + re.Error(err) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/loadTemporaryFail")) +} + +// TestWatchAndDynamicallyApplyChanges tests the keyspace group manager watch and dynamically apply +// keyspace groups' membership/distribution meta changes. +func (suite *keyspaceGroupManagerTestSuite) TestWatchAndDynamicallyApplyChanges() { + re := suite.Require() + + // Start with the empty keyspace group assignment. + mgr := suite.newUniqueKeyspaceGroupManager(0) + re.NotNil(mgr) + defer mgr.Close() + err := mgr.Initialize() + re.NoError(err) + + rootPath := mgr.legacySvcRootPath + svcAddr := mgr.tsoServiceID.ServiceAddr + + // Initialize PUT/DELETE events + events := []*etcdEvent{} + // Assign keyspace group 0 to this host/pod/keyspace-group-manager. + // final result: assigned [0], loaded [0] + events = append(events, generateKeyspaceGroupPutEvent(0, []uint32{0}, []string{svcAddr})) + // Assign keyspace group 1 to this host/pod/keyspace-group-manager. + // final result: assigned [0,1], loaded [0,1] + events = append(events, generateKeyspaceGroupPutEvent(1, []uint32{1}, []string{"unknown", svcAddr})) + // Assign keyspace group 2 to other host/pod/keyspace-group-manager. + // final result: assigned [0,1], loaded [0,1,2] + events = append(events, generateKeyspaceGroupPutEvent(2, []uint32{2}, []string{"unknown"})) + // Assign keyspace group 3 to this host/pod/keyspace-group-manager. + // final result: assigned [0,1,3], loaded [0,1,2,3] + events = append(events, generateKeyspaceGroupPutEvent(3, []uint32{3}, []string{svcAddr})) + // Delete keyspace group 0. Every tso node/pod now should initialize keyspace group 0. + // final result: assigned [0,1,3], loaded [0,1,2,3] + events = append(events, generateKeyspaceGroupDeleteEvent(0)) + // Put keyspace group 4 which doesn't belong to anyone. + // final result: assigned [0,1,3], loaded [0,1,2,3,4] + events = append(events, generateKeyspaceGroupPutEvent(4, []uint32{4}, []string{})) + // Put keyspace group 5 which doesn't belong to anyone. + // final result: assigned [0,1,3], loaded [0,1,2,3,4,5] + events = append(events, generateKeyspaceGroupPutEvent(5, []uint32{5}, []string{})) + // Assign keyspace group 2 to this host/pod/keyspace-group-manager. + // final result: assigned [0,1,2,3], loaded [0,1,2,3,4,5] + events = append(events, generateKeyspaceGroupPutEvent(2, []uint32{2}, []string{svcAddr})) + // Reassign keyspace group 3 to no one. + // final result: assigned [0,1,2], loaded [0,1,2,3,4,5] + events = append(events, generateKeyspaceGroupPutEvent(3, []uint32{3}, []string{})) + // Reassign keyspace group 4 to this host/pod/keyspace-group-manager. + // final result: assigned [0,1,2,4], loaded [0,1,2,3,4,5] + events = append(events, generateKeyspaceGroupPutEvent(4, []uint32{4}, []string{svcAddr})) + // Delete keyspace group 2. + // final result: assigned [0,1,4], loaded [0,1,3,4,5] + events = append(events, generateKeyspaceGroupDeleteEvent(2)) + + // Apply the keyspace group assignment change events to etcd. + suite.applyEtcdEvents(re, rootPath, events) + + // Verify the keyspace groups assigned. + // Eventually, this keyspace groups manager is expected to serve the following keyspace groups. + expectedAssignedGroups := []uint32{0, 1, 4} + testutil.Eventually(re, func() bool { + assignedGroups := collectAssignedKeyspaceGroupIDs(re, mgr) + return reflect.DeepEqual(expectedAssignedGroups, assignedGroups) + }) + + // Verify the keyspace groups loaded. + // Eventually, this keyspace groups manager is expected to load the following keyspace groups + // in which keyspace group 3, 5 aren't served by this tso node/pod. + expectedLoadedGroups := []uint32{0, 1, 3, 4, 5} + testutil.Eventually(re, func() bool { + loadedGroups := collectAllLoadedKeyspaceGroupIDs(mgr) + return reflect.DeepEqual(expectedLoadedGroups, loadedGroups) + }) +} + +// TestDefaultKeyspaceGroup tests the initialization logic of the default keyspace group. +// If the default keyspace group isn't configured in the etcd, every tso node/pod should initialize +// it and join the election for the primary of this group. +// If the default keyspace group is configured in the etcd, the tso nodes/pods which are assigned with +// this group will initialize it and join the election for the primary of this group. +func (suite *keyspaceGroupManagerTestSuite) TestInitDefaultKeyspaceGroup() { + re := suite.Require() + + var ( + expectedGroupIDs []uint32 + event *etcdEvent + ) + + // Start with the empty keyspace group assignment. + mgr := suite.newUniqueKeyspaceGroupManager(0) + defer mgr.Close() + err := mgr.Initialize() + re.NoError(err) + + rootPath := mgr.legacySvcRootPath + svcAddr := mgr.tsoServiceID.ServiceAddr + + expectedGroupIDs = []uint32{0} + assignedGroupIDs := collectAssignedKeyspaceGroupIDs(re, mgr) + re.Equal(expectedGroupIDs, assignedGroupIDs) + + // Config keyspace group 0 in the storage but assigned to no one. + // final result: [] + expectedGroupIDs = []uint32{} + event = generateKeyspaceGroupPutEvent(0, []uint32{0}, []string{"unknown"}) + err = putKeyspaceGroupToEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg) + re.NoError(err) + testutil.Eventually(re, func() bool { + assignedGroupIDs := collectAssignedKeyspaceGroupIDs(re, mgr) + return reflect.DeepEqual(expectedGroupIDs, assignedGroupIDs) + }) + // Config keyspace group 0 in the storage and assigned to this host/pod/keyspace-group-manager. + // final result: [0] + expectedGroupIDs = []uint32{0} + event = generateKeyspaceGroupPutEvent(0, []uint32{0}, []string{svcAddr}) + err = putKeyspaceGroupToEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg) + re.NoError(err) + testutil.Eventually(re, func() bool { + assignedGroupIDs := collectAssignedKeyspaceGroupIDs(re, mgr) + return reflect.DeepEqual(expectedGroupIDs, assignedGroupIDs) + }) + // Delete keyspace group 0. Every tso node/pod now should initialize keyspace group 0. + // final result: [0] + expectedGroupIDs = []uint32{0} + event = generateKeyspaceGroupDeleteEvent(0) + err = deleteKeyspaceGroupInEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg.ID) + re.NoError(err) + testutil.Eventually(re, func() bool { + assignedGroupIDs := collectAssignedKeyspaceGroupIDs(re, mgr) + return reflect.DeepEqual(expectedGroupIDs, assignedGroupIDs) + }) + // Config keyspace group 0 in the storage and assigned to this host/pod/keyspace-group-manager. + // final result: [0] + expectedGroupIDs = []uint32{0} + event = generateKeyspaceGroupPutEvent(0, []uint32{0}, []string{svcAddr}) + err = putKeyspaceGroupToEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg) + re.NoError(err) + testutil.Eventually(re, func() bool { + assignedGroupIDs := collectAssignedKeyspaceGroupIDs(re, mgr) + return reflect.DeepEqual(expectedGroupIDs, assignedGroupIDs) + }) +} + +// TestGetKeyspaceGroupMetaWithCheck tests GetKeyspaceGroupMetaWithCheck. +func (suite *keyspaceGroupManagerTestSuite) TestGetKeyspaceGroupMetaWithCheck() { + re := suite.Require() + + mgr := suite.newUniqueKeyspaceGroupManager(1) + re.NotNil(mgr) + defer mgr.Close() + + var ( + am *AllocatorManager + kg *endpoint.KeyspaceGroup + kgid uint32 + err error + ) + + // Create keyspace group 0 which contains keyspace 0, 1, 2. + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, true, + mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, + uint32(0), []uint32{0, 1, 2}) + + err = mgr.Initialize() + re.NoError(err) + + // Should be able to get AM for the default/null keyspace and keyspace 1, 2 in keyspace group 0. + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(mcsutils.DefaultKeyspaceID, 0) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(mcsutils.NullKeyspaceID, 0) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(1, 0) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(2, 0) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + // Should still succeed even keyspace 3 isn't explicitly assigned to any + // keyspace group. It will be assigned to the default keyspace group. + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(3, 0) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + // Should succeed and get the meta of keyspace group 0, because keyspace 0 + // belongs to group 0, though the specified group 1 doesn't exist. + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(mcsutils.DefaultKeyspaceID, 1) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + // Should fail because keyspace 3 isn't explicitly assigned to any keyspace + // group, and the specified group isn't the default keyspace group. + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(3, 100) + re.Error(err) + re.Equal(uint32(100), kgid) + re.Nil(am) + re.Nil(kg) +} + +// TestDefaultMembershipRestriction tests the restriction of default keyspace always +// belongs to default keyspace group. +func (suite *keyspaceGroupManagerTestSuite) TestDefaultMembershipRestriction() { + re := suite.Require() + + mgr := suite.newUniqueKeyspaceGroupManager(1) + re.NotNil(mgr) + defer mgr.Close() + + rootPath := mgr.legacySvcRootPath + svcAddr := mgr.tsoServiceID.ServiceAddr + + var ( + am *AllocatorManager + kg *endpoint.KeyspaceGroup + kgid uint32 + err error + event *etcdEvent + ) + + // Create keyspace group 0 which contains keyspace 0, 1, 2. + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, true, + mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, + mcsutils.DefaultKeyspaceGroupID, []uint32{mcsutils.DefaultKeyspaceID, 1, 2}) + // Create keyspace group 3 which contains keyspace 3, 4. + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, true, + mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, + uint32(3), []uint32{3, 4}) + + err = mgr.Initialize() + re.NoError(err) + + // Should be able to get AM for keyspace 0 in keyspace group 0. + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck( + mcsutils.DefaultKeyspaceID, mcsutils.DefaultKeyspaceGroupID) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + + event = generateKeyspaceGroupPutEvent( + mcsutils.DefaultKeyspaceGroupID, []uint32{1, 2}, []string{svcAddr}) + err = putKeyspaceGroupToEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg) + re.NoError(err) + event = generateKeyspaceGroupPutEvent( + 3, []uint32{mcsutils.DefaultKeyspaceID, 3, 4}, []string{svcAddr}) + err = putKeyspaceGroupToEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg) + re.NoError(err) + + // Sleep for a while to wait for the events to propagate. If the restriction is not working, + // it will cause random failure. + time.Sleep(1 * time.Second) + // Should still be able to get AM for keyspace 0 in keyspace group 0. + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck( + mcsutils.DefaultKeyspaceID, mcsutils.DefaultKeyspaceGroupID) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) + // Should succeed and return the keyspace group meta from the default keyspace group + am, kg, kgid, err = mgr.getKeyspaceGroupMetaWithCheck(mcsutils.DefaultKeyspaceID, 3) + re.NoError(err) + re.Equal(mcsutils.DefaultKeyspaceGroupID, kgid) + re.NotNil(am) + re.NotNil(kg) +} + +// TestHandleTSORequestWithWrongMembership tests the case that HandleTSORequest receives +// a tso request with mismatched keyspace and keyspace group. +func (suite *keyspaceGroupManagerTestSuite) TestHandleTSORequestWithWrongMembership() { + re := suite.Require() + + mgr := suite.newUniqueKeyspaceGroupManager(1) + re.NotNil(mgr) + defer mgr.Close() + + // Create keyspace group 0 which contains keyspace 0, 1, 2. + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, true, + mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, + uint32(0), []uint32{0, 1, 2}) + + err := mgr.Initialize() + re.NoError(err) + + // Should fail because keyspace 0 is not in keyspace group 1 and the API returns + // the keyspace group 0 to which the keyspace 0 belongs. + _, keyspaceGroupBelongTo, err := mgr.HandleTSORequest(0, 1, GlobalDCLocation, 1) + re.Error(err) + re.Equal(uint32(0), keyspaceGroupBelongTo) +} + +type etcdEvent struct { + eventType mvccpb.Event_EventType + ksg *endpoint.KeyspaceGroup +} + +func generateKeyspaceGroupPutEvent( + groupID uint32, keyspaces []uint32, addrs []string, splitState ...*endpoint.SplitState, +) *etcdEvent { + members := []endpoint.KeyspaceGroupMember{} + for _, addr := range addrs { + members = append(members, endpoint.KeyspaceGroupMember{Address: addr}) + } + var ss *endpoint.SplitState + if len(splitState) > 0 { + ss = splitState[0] + } + + return &etcdEvent{ + eventType: mvccpb.PUT, + ksg: &endpoint.KeyspaceGroup{ + ID: groupID, + Members: members, + Keyspaces: keyspaces, + SplitState: ss, + }, + } +} + +func generateKeyspaceGroupDeleteEvent(groupID uint32) *etcdEvent { + return &etcdEvent{ + eventType: mvccpb.DELETE, + ksg: &endpoint.KeyspaceGroup{ + ID: groupID, + }, + } +} + +func (suite *keyspaceGroupManagerTestSuite) applyEtcdEvents( + re *require.Assertions, + rootPath string, + events []*etcdEvent, +) { + var err error + for _, event := range events { + switch event.eventType { + case mvccpb.PUT: + err = putKeyspaceGroupToEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg) + case mvccpb.DELETE: + err = deleteKeyspaceGroupInEtcd(suite.ctx, suite.etcdClient, rootPath, event.ksg.ID) + } + re.NoError(err) + } +} + +func (suite *keyspaceGroupManagerTestSuite) newKeyspaceGroupManager( + tsoServiceID *discovery.ServiceRegistryEntry, + electionNamePrefix, legacySvcRootPath, tsoSvcRootPath string, +) *KeyspaceGroupManager { + return NewKeyspaceGroupManager( + suite.ctx, tsoServiceID, suite.etcdClient, nil, + electionNamePrefix, legacySvcRootPath, tsoSvcRootPath, + suite.cfg) +} + +// runTestLoadMultipleKeyspaceGroupsAssignment tests the loading of multiple keyspace group assignment. +func (suite *keyspaceGroupManagerTestSuite) runTestLoadKeyspaceGroupsAssignment( + re *require.Assertions, + numberOfKeyspaceGroupsToAdd int, + loadKeyspaceGroupsBatchSize int64, // set to 0 to use the default value + probabilityAssignToMe int, // percentage of assigning keyspace groups to this host/pod +) { + expectedGroupIDs := []uint32{} + mgr := suite.newUniqueKeyspaceGroupManager(loadKeyspaceGroupsBatchSize) + re.NotNil(mgr) + defer mgr.Close() + + step := 30 + mux := sync.Mutex{} + wg := sync.WaitGroup{} + for i := 0; i < numberOfKeyspaceGroupsToAdd; i += step { + wg.Add(1) + go func(startID int) { + defer wg.Done() + + endID := startID + step + if endID > numberOfKeyspaceGroupsToAdd { + endID = numberOfKeyspaceGroupsToAdd + } + + randomGen := rand.New(rand.NewSource(time.Now().UnixNano())) + for j := startID; j < endID; j++ { + assignToMe := false + // Assign the keyspace group to this host/pod with the given probability, + // and the keyspace group manager only loads the keyspace groups with id + // less than len(mgr.ams). + if j < len(mgr.ams) && randomGen.Intn(100) < probabilityAssignToMe { + assignToMe = true + mux.Lock() + expectedGroupIDs = append(expectedGroupIDs, uint32(j)) + mux.Unlock() + } + addKeyspaceGroupAssignment( + suite.ctx, suite.etcdClient, + assignToMe, mgr.legacySvcRootPath, mgr.tsoServiceID.ServiceAddr, + uint32(j), []uint32{uint32(j)}) + } + }(i) + } + wg.Wait() + + err := mgr.Initialize() + re.NoError(err) + + // If no keyspace group is assigned to this host/pod, the default keyspace group should be initialized. + if numberOfKeyspaceGroupsToAdd <= 0 { + expectedGroupIDs = append(expectedGroupIDs, mcsutils.DefaultKeyspaceGroupID) + } + + // Verify the keyspace group assignment. + // Sort the keyspaces in ascending order + sort.Slice(expectedGroupIDs, func(i, j int) bool { + return expectedGroupIDs[i] < expectedGroupIDs[j] + }) + assignedGroupIDs := collectAssignedKeyspaceGroupIDs(re, mgr) + re.Equal(expectedGroupIDs, assignedGroupIDs) +} + +func (suite *keyspaceGroupManagerTestSuite) newUniqueKeyspaceGroupManager( + loadKeyspaceGroupsBatchSize int64, // set to 0 to use the default value +) *KeyspaceGroupManager { + tsoServiceID := &discovery.ServiceRegistryEntry{ServiceAddr: suite.cfg.AdvertiseListenAddr} + uniqueID := memberutil.GenerateUniqueID(uuid.New().String()) + uniqueStr := strconv.FormatUint(uniqueID, 10) + legacySvcRootPath := path.Join("/pd", uniqueStr) + tsoSvcRootPath := path.Join("/ms", uniqueStr, "tso") + electionNamePrefix := "kgm-test-" + uniqueStr + + keyspaceGroupManager := suite.newKeyspaceGroupManager(tsoServiceID, electionNamePrefix, legacySvcRootPath, tsoSvcRootPath) + + if loadKeyspaceGroupsBatchSize != 0 { + keyspaceGroupManager.loadKeyspaceGroupsBatchSize = loadKeyspaceGroupsBatchSize + } + return keyspaceGroupManager +} + +// putKeyspaceGroupToEtcd puts a keyspace group to etcd. +func putKeyspaceGroupToEtcd( + ctx context.Context, etcdClient *clientv3.Client, + rootPath string, group *endpoint.KeyspaceGroup, +) error { + key := strings.Join([]string{rootPath, endpoint.KeyspaceGroupIDPath(group.ID)}, "/") + value, err := json.Marshal(group) + if err != nil { + return err + } + + if _, err := etcdClient.Put(ctx, key, string(value)); err != nil { + return err + } + + return nil +} + +// deleteKeyspaceGroupInEtcd deletes a keyspace group in etcd. +func deleteKeyspaceGroupInEtcd( + ctx context.Context, etcdClient *clientv3.Client, + rootPath string, id uint32, +) error { + key := strings.Join([]string{rootPath, endpoint.KeyspaceGroupIDPath(id)}, "/") + + if _, err := etcdClient.Delete(ctx, key); err != nil { + return err + } + + return nil +} + +// addKeyspaceGroupAssignment adds a keyspace group assignment to etcd. +func addKeyspaceGroupAssignment( + ctx context.Context, etcdClient *clientv3.Client, + assignToMe bool, rootPath, svcAddr string, + groupID uint32, keyspaces []uint32, +) error { + var location string + if assignToMe { + location = svcAddr + } else { + location = uuid.NewString() + } + group := &endpoint.KeyspaceGroup{ + ID: groupID, + Members: []endpoint.KeyspaceGroupMember{{Address: location}}, + Keyspaces: keyspaces, + } + + key := strings.Join([]string{rootPath, endpoint.KeyspaceGroupIDPath(groupID)}, "/") + value, err := json.Marshal(group) + if err != nil { + return err + } + + if _, err := etcdClient.Put(ctx, key, string(value)); err != nil { + return err + } + + return nil +} + +func collectAssignedKeyspaceGroupIDs(re *require.Assertions, kgm *KeyspaceGroupManager) []uint32 { + kgm.RLock() + defer kgm.RUnlock() + + ids := []uint32{} + for i := 0; i < len(kgm.kgs); i++ { + kg := kgm.kgs[i] + if kg == nil { + re.Nil(kgm.ams[i], fmt.Sprintf("ksg is nil but am is not nil for id %d", i)) + } else { + am := kgm.ams[i] + if am != nil { + re.Equal(i, int(am.kgID)) + re.Equal(i, int(kg.ID)) + for _, m := range kg.Members { + if m.Address == kgm.tsoServiceID.ServiceAddr { + ids = append(ids, uint32(i)) + break + } + } + } + } + } + + return ids +} + +func collectAllLoadedKeyspaceGroupIDs(kgm *KeyspaceGroupManager) []uint32 { + kgm.RLock() + defer kgm.RUnlock() + + ids := []uint32{} + for i := 0; i < len(kgm.kgs); i++ { + kg := kgm.kgs[i] + if kg != nil { + ids = append(ids, uint32(i)) + } + } + + return ids +} + +func (suite *keyspaceGroupManagerTestSuite) TestUpdateKeyspaceGroupMembership() { + re := suite.Require() + + // Start from an empty keyspace group. + // Use non-default keyspace group ID. + // The default keyspace group always contains the default keyspace. + // We have dedicated tests for the default keyspace group. + groupID := uint32(1) + oldGroup := &endpoint.KeyspaceGroup{ID: groupID, Keyspaces: []uint32{}} + newGroup := &endpoint.KeyspaceGroup{ID: groupID, Keyspaces: []uint32{}} + kgm := &KeyspaceGroupManager{ + state: state{ + keyspaceLookupTable: make(map[uint32]uint32), + }} + + kgm.updateKeyspaceGroupMembership(oldGroup, newGroup, true) + verifyLocalKeyspaceLookupTable(re, newGroup.KeyspaceLookupTable, newGroup.Keyspaces) + verifyGlobalKeyspaceLookupTable(re, kgm.keyspaceLookupTable, newGroup.KeyspaceLookupTable) + + targetKeyspacesList := [][]uint32{ + {1}, // Add keyspace 1 to the keyspace group. + {1, 2}, // Add keyspace 2 to the keyspace group. + {1, 2}, // No change. + {1, 2, 3, 4}, // Add keyspace 3 and 4 to the keyspace group. + {5, 6, 7}, // Remove keyspace 1, 2, 3, 4 from the keyspace group and add 5, 6, 7 + {7, 8, 9}, // Partially update the keyspace group. + {1, 2, 3, 4, 5, 6, 7, 8, 9}, // Add more keyspace to the keyspace group. + {9, 8, 4, 5, 6}, // Out of order. + {9, 8, 4, 5, 6}, // No change. Out of order. + {8, 9}, // Remove + {10}, // Remove + {}, // End with the empty keyspace group. + } + + for _, keyspaces := range targetKeyspacesList { + oldGroup = newGroup + keyspacesCopy := make([]uint32, len(keyspaces)) + copy(keyspacesCopy, keyspaces) + newGroup = &endpoint.KeyspaceGroup{ID: groupID, Keyspaces: keyspacesCopy} + kgm.updateKeyspaceGroupMembership(oldGroup, newGroup, true) + verifyLocalKeyspaceLookupTable(re, newGroup.KeyspaceLookupTable, newGroup.Keyspaces) + verifyGlobalKeyspaceLookupTable(re, kgm.keyspaceLookupTable, newGroup.KeyspaceLookupTable) + + // Verify the keyspaces loaded is sorted. + re.Equal(len(keyspaces), len(newGroup.Keyspaces)) + for i := 0; i < len(newGroup.Keyspaces); i++ { + if i > 0 { + re.True(newGroup.Keyspaces[i-1] < newGroup.Keyspaces[i]) + } + } + } +} + +func verifyLocalKeyspaceLookupTable( + re *require.Assertions, keyspaceLookupTable map[uint32]struct{}, newKeyspaces []uint32, +) { + re.Equal(len(newKeyspaces), len(keyspaceLookupTable), + fmt.Sprintf("%v %v", newKeyspaces, keyspaceLookupTable)) + for _, keyspace := range newKeyspaces { + _, ok := keyspaceLookupTable[keyspace] + re.True(ok) + } +} + +func verifyGlobalKeyspaceLookupTable( + re *require.Assertions, + gKeyspaceLookupTable map[uint32]uint32, + lKeyspaceLookupTable map[uint32]struct{}, +) { + for keyspace := range gKeyspaceLookupTable { + _, ok := lKeyspaceLookupTable[keyspace] + re.True(ok) + } + for keyspace := range lKeyspaceLookupTable { + _, ok := gKeyspaceLookupTable[keyspace] + re.True(ok) + } +} + +func (suite *keyspaceGroupManagerTestSuite) TestGroupSplitUpdateRetry() { + re := suite.Require() + + // Start with the empty keyspace group assignment. + mgr := suite.newUniqueKeyspaceGroupManager(0) + re.NotNil(mgr) + defer mgr.Close() + err := mgr.Initialize() + re.NoError(err) + + rootPath := mgr.legacySvcRootPath + svcAddr := mgr.tsoServiceID.ServiceAddr + + events := []*etcdEvent{} + // Split target keyspace group event arrives first. + events = append(events, generateKeyspaceGroupPutEvent(2, []uint32{2} /* Mock 2 replicas */, []string{svcAddr, svcAddr}, &endpoint.SplitState{ + SplitSource: 1, + })) + // Split source keyspace group event arrives later. + events = append(events, generateKeyspaceGroupPutEvent(1, []uint32{1}, []string{svcAddr, svcAddr}, &endpoint.SplitState{ + SplitSource: 1, + })) + + // Eventually, this keyspace groups manager is expected to serve the following keyspace groups. + expectedGroupIDs := []uint32{0, 1, 2} + + // Apply the keyspace group assignment change events to etcd. + suite.applyEtcdEvents(re, rootPath, events) + + // Verify the keyspace group assignment. + testutil.Eventually(re, func() bool { + assignedGroupIDs := collectAssignedKeyspaceGroupIDs(re, mgr) + return reflect.DeepEqual(expectedGroupIDs, assignedGroupIDs) + }) +} diff --git a/pkg/tso/local_allocator.go b/pkg/tso/local_allocator.go index 769f18a12d2..9c2867966bc 100644 --- a/pkg/tso/local_allocator.go +++ b/pkg/tso/local_allocator.go @@ -25,6 +25,7 @@ import ( "github.com/pingcap/log" "github.com/tikv/pd/pkg/election" "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/utils/logutil" "github.com/tikv/pd/pkg/utils/tsoutil" "github.com/tikv/pd/pkg/utils/typeutil" @@ -43,7 +44,7 @@ type LocalTSOAllocator struct { // for election use, notice that the leadership that member holds is // the leadership for PD leader. Local TSO Allocator's leadership is for the // election of Local TSO Allocator leader among several PD servers and - // Local TSO Allocator only use member's some etcd and pbpd.Member info. + // Local TSO Allocator only use member's some etcd and pdpb.Member info. // So it's not conflicted. rootPath string allocatorLeader atomic.Value // stored as *pdpb.Member @@ -55,13 +56,24 @@ func NewLocalTSOAllocator( leadership *election.Leadership, dcLocation string, ) Allocator { + // Construct the timestampOracle path prefix, which is: + // 1. for the default keyspace group: + // lta/{dc-location} in /pd/{cluster_id}/lta/{dc-location}/timestamp + // 2. for the non-default keyspace groups: + // {group}/lta/{dc-location} in /ms/{cluster_id}/tso/{group}/lta/{dc-location}/timestamp + var tsPath string + if am.kgID == utils.DefaultKeyspaceGroupID { + tsPath = path.Join(localTSOAllocatorEtcdPrefix, dcLocation) + } else { + tsPath = path.Join(fmt.Sprintf("%05d", am.kgID), localTSOAllocatorEtcdPrefix, dcLocation) + } return &LocalTSOAllocator{ allocatorManager: am, leadership: leadership, timestampOracle: ×tampOracle{ client: leadership.GetClient(), rootPath: am.rootPath, - ltsPath: path.Join(localTSOAllocatorEtcdPrefix, dcLocation), + tsPath: tsPath, storage: am.storage, saveInterval: am.saveInterval, updatePhysicalInterval: am.updatePhysicalInterval, @@ -73,6 +85,14 @@ func NewLocalTSOAllocator( } } +// GetTimestampPath returns the timestamp path in etcd. +func (lta *LocalTSOAllocator) GetTimestampPath() string { + if lta == nil || lta.timestampOracle == nil { + return "" + } + return lta.timestampOracle.GetTimestampPath() +} + // GetDCLocation returns the local allocator's dc-location. func (lta *LocalTSOAllocator) GetDCLocation() string { return lta.timestampOracle.dcLocation @@ -106,7 +126,8 @@ func (lta *LocalTSOAllocator) SetTSO(tso uint64, ignoreSmaller, skipUpperBoundCh func (lta *LocalTSOAllocator) GenerateTSO(count uint32) (pdpb.Timestamp, error) { if !lta.leadership.Check() { tsoCounter.WithLabelValues("not_leader", lta.timestampOracle.dcLocation).Inc() - return pdpb.Timestamp{}, errs.ErrGenerateTimestamp.FastGenByArgs(fmt.Sprintf("requested pd %s of %s allocator", errs.NotLeaderErr, lta.timestampOracle.dcLocation)) + return pdpb.Timestamp{}, errs.ErrGenerateTimestamp.FastGenByArgs( + fmt.Sprintf("requested pd %s of %s allocator", errs.NotLeaderErr, lta.timestampOracle.dcLocation)) } return lta.timestampOracle.getTS(lta.leadership, count, lta.allocatorManager.GetSuffixBits()) } @@ -137,7 +158,7 @@ func (lta *LocalTSOAllocator) GetAllocatorLeader() *pdpb.Member { } // GetMember returns the Local TSO Allocator's member value. -func (lta *LocalTSOAllocator) GetMember() Member { +func (lta *LocalTSOAllocator) GetMember() ElectionMember { return lta.allocatorManager.member } @@ -192,7 +213,7 @@ func (lta *LocalTSOAllocator) isSameAllocatorLeader(leader *pdpb.Member) bool { // CheckAllocatorLeader checks who is the current Local TSO Allocator leader, and returns true if it is needed to check later. func (lta *LocalTSOAllocator) CheckAllocatorLeader() (*pdpb.Member, int64, bool) { - if err := lta.allocatorManager.member.PrecheckLeader(); err != nil { + if err := lta.allocatorManager.member.PreCheckLeader(); err != nil { log.Error("no etcd leader, check local tso allocator leader later", zap.String("dc-location", lta.timestampOracle.dcLocation), errs.ZapError(err)) time.Sleep(200 * time.Millisecond) diff --git a/pkg/tso/testutil.go b/pkg/tso/testutil.go new file mode 100644 index 00000000000..9225b21dfac --- /dev/null +++ b/pkg/tso/testutil.go @@ -0,0 +1,112 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tso + +import ( + "testing" + "time" + + "github.com/stretchr/testify/require" + "github.com/tikv/pd/pkg/utils/etcdutil" + "github.com/tikv/pd/pkg/utils/grpcutil" + "go.etcd.io/etcd/clientv3" + "go.etcd.io/etcd/embed" +) + +var _ ServiceConfig = (*TestServiceConfig)(nil) + +// TestServiceConfig implements the ServiceConfig interface. +type TestServiceConfig struct { + Name string // Name of the service. + BackendEndpoints string // Backend endpoints of the service. + ListenAddr string // Address the service listens on. + AdvertiseListenAddr string // Address the service advertises to the clients. + LeaderLease int64 // Leader lease. + LocalTSOEnabled bool // Whether local TSO is enabled. + TSOUpdatePhysicalInterval time.Duration // Interval to update TSO in physical storage. + TSOSaveInterval time.Duration // Interval to save TSO to physical storage. + MaxResetTSGap time.Duration // Maximum gap to reset TSO. + TLSConfig *grpcutil.TLSConfig // TLS configuration. +} + +// GetName returns the Name field of TestServiceConfig. +func (c *TestServiceConfig) GetName() string { + return c.Name +} + +// GeBackendEndpoints returns the BackendEndpoints field of TestServiceConfig. +func (c *TestServiceConfig) GeBackendEndpoints() string { + return c.BackendEndpoints +} + +// GetListenAddr returns the ListenAddr field of TestServiceConfig. +func (c *TestServiceConfig) GetListenAddr() string { + return c.ListenAddr +} + +// GetAdvertiseListenAddr returns the AdvertiseListenAddr field of TestServiceConfig. +func (c *TestServiceConfig) GetAdvertiseListenAddr() string { + return c.AdvertiseListenAddr +} + +// GetLeaderLease returns the LeaderLease field of TestServiceConfig. +func (c *TestServiceConfig) GetLeaderLease() int64 { + return c.LeaderLease +} + +// IsLocalTSOEnabled returns the LocalTSOEnabled field of TestServiceConfig. +func (c *TestServiceConfig) IsLocalTSOEnabled() bool { + return c.LocalTSOEnabled +} + +// GetTSOUpdatePhysicalInterval returns the TSOUpdatePhysicalInterval field of TestServiceConfig. +func (c *TestServiceConfig) GetTSOUpdatePhysicalInterval() time.Duration { + return c.TSOUpdatePhysicalInterval +} + +// GetTSOSaveInterval returns the TSOSaveInterval field of TestServiceConfig. +func (c *TestServiceConfig) GetTSOSaveInterval() time.Duration { + return c.TSOSaveInterval +} + +// GetMaxResetTSGap returns the MaxResetTSGap field of TestServiceConfig. +func (c *TestServiceConfig) GetMaxResetTSGap() time.Duration { + return c.MaxResetTSGap +} + +// GetTLSConfig returns the TLSConfig field of TestServiceConfig. +func (c *TestServiceConfig) GetTLSConfig() *grpcutil.TLSConfig { + return c.TLSConfig +} + +func startEmbeddedEtcd(t *testing.T) (backendEndpoint string, etcdClient *clientv3.Client, clean func()) { + re := require.New(t) + cfg := etcdutil.NewTestSingleConfig(t) + etcd, err := embed.StartEtcd(cfg) + re.NoError(err) + clean = func() { + etcd.Close() + } + + backendEndpoint = cfg.LCUrls[0].String() + re.NoError(err) + + etcdClient, err = clientv3.NewFromURL(backendEndpoint) + re.NoError(err) + + <-etcd.Server.ReadyNotify() + + return +} diff --git a/pkg/tso/tso.go b/pkg/tso/tso.go index 0b4ad7da745..aa1a424d8cd 100644 --- a/pkg/tso/tso.go +++ b/pkg/tso/tso.go @@ -62,8 +62,8 @@ type tsoObject struct { type timestampOracle struct { client *clientv3.Client rootPath string - // When ltsPath is empty, it means that it is a global timestampOracle. - ltsPath string + // When tsPath is empty, it means that it is a global timestampOracle. + tsPath string storage endpoint.TSOStorage // TODO: remove saveInterval saveInterval time.Duration @@ -116,7 +116,7 @@ func (t *timestampOracle) generateTSO(count int64, suffixBits int) (physical int t.tsoMux.logical += count logical = t.tsoMux.logical if suffixBits > 0 && t.suffix >= 0 { - logical = t.differentiateLogical(logical, suffixBits) + logical = t.calibrateLogical(logical, suffixBits) } // Return the last update time lastUpdateTime = t.tsoMux.updateTime @@ -137,12 +137,13 @@ func (t *timestampOracle) generateTSO(count int64, suffixBits int) (physical int // dc-1: xxxxxxxxxx00000001 // dc-2: xxxxxxxxxx00000010 // dc-3: xxxxxxxxxx00000011 -func (t *timestampOracle) differentiateLogical(rawLogical int64, suffixBits int) int64 { +func (t *timestampOracle) calibrateLogical(rawLogical int64, suffixBits int) int64 { return rawLogical<= current + maxResetTSGap func (t *timestampOracle) resetUserTimestamp(leadership *election.Leadership, tso uint64, ignoreSmaller bool) error { return t.resetUserTimestampInner(leadership, tso, ignoreSmaller, false) @@ -240,7 +242,7 @@ func (t *timestampOracle) resetUserTimestampInner(leadership *election.Leadershi // save into etcd only if nextPhysical is close to lastSavedTime if typeutil.SubRealTimeByWallClock(t.lastSavedTime.Load().(time.Time), nextPhysical) <= UpdateTimestampGuard { save := nextPhysical.Add(t.saveInterval) - if err := t.storage.SaveTimestamp(t.getTimestampPath(), save); err != nil { + if err := t.storage.SaveTimestamp(t.GetTimestampPath(), save); err != nil { tsoCounter.WithLabelValues("err_save_reset_ts", t.dcLocation).Inc() return err } @@ -285,7 +287,11 @@ func (t *timestampOracle) UpdateTimestamp(leadership *election.Leadership) error jetLag := typeutil.SubRealTimeByWallClock(now, prevPhysical) if jetLag > 3*t.updatePhysicalInterval && jetLag > jetLagWarningThreshold { - log.Warn("clock offset", zap.Duration("jet-lag", jetLag), zap.Time("prev-physical", prevPhysical), zap.Time("now", now), zap.Duration("update-physical-interval", t.updatePhysicalInterval)) + log.Warn("clock offset", + zap.Duration("jet-lag", jetLag), + zap.Time("prev-physical", prevPhysical), + zap.Time("now", now), + zap.Duration("update-physical-interval", t.updatePhysicalInterval)) tsoCounter.WithLabelValues("slow_save", t.dcLocation).Inc() } @@ -312,7 +318,11 @@ func (t *timestampOracle) UpdateTimestamp(leadership *election.Leadership) error // The time window needs to be updated and saved to etcd. if typeutil.SubRealTimeByWallClock(t.lastSavedTime.Load().(time.Time), next) <= UpdateTimestampGuard { save := next.Add(t.saveInterval) - if err := t.storage.SaveTimestamp(t.getTimestampPath(), save); err != nil { + if err := t.storage.SaveTimestamp(t.GetTimestampPath(), save); err != nil { + log.Warn("save timestamp failed", + zap.String("dc-location", t.dcLocation), + zap.String("timestamp-path", t.GetTimestampPath()), + zap.Error(err)) tsoCounter.WithLabelValues("err_save_update_ts", t.dcLocation).Inc() return err } @@ -358,7 +368,7 @@ func (t *timestampOracle) getTS(leadership *election.Leadership, count uint32, s } // In case lease expired after the first check. if !leadership.Check() { - return pdpb.Timestamp{}, errs.ErrGenerateTimestamp.FastGenByArgs("not the pd or local tso allocator leader anymore") + return pdpb.Timestamp{}, errs.ErrGenerateTimestamp.FastGenByArgs(fmt.Sprintf("requested %s anymore", errs.NotLeaderErr)) } resp.SuffixBits = uint32(suffixBits) return resp, nil diff --git a/pkg/utils/apiutil/multiservicesapi/middleware.go b/pkg/utils/apiutil/multiservicesapi/middleware.go index 4f889b52573..ed34ecc6afb 100644 --- a/pkg/utils/apiutil/multiservicesapi/middleware.go +++ b/pkg/utils/apiutil/multiservicesapi/middleware.go @@ -26,16 +26,19 @@ import ( "go.uber.org/zap" ) -// HTTP headers. const ( + // ServiceAllowDirectHandle is the header key to allow direct handle. ServiceAllowDirectHandle = "service-allow-direct-handle" - ServiceRedirectorHeader = "service-redirector" + // ServiceRedirectorHeader is the header key to indicate the request is redirected. + ServiceRedirectorHeader = "service-redirector" + // ServiceContextKey is the key to get service server from gin.Context. + ServiceContextKey = "service" ) // ServiceRedirector is a middleware to redirect the request to the right place. func ServiceRedirector() gin.HandlerFunc { return func(c *gin.Context) { - svr := c.MustGet("service").(bs.Server) + svr := c.MustGet(ServiceContextKey).(bs.Server) allowDirectHandle := len(c.Request.Header.Get(ServiceAllowDirectHandle)) > 0 isServing := svr.IsServing() if allowDirectHandle || isServing { diff --git a/pkg/utils/etcdutil/etcdutil.go b/pkg/utils/etcdutil/etcdutil.go index 423b4f0c42d..dc5bc7d095a 100644 --- a/pkg/utils/etcdutil/etcdutil.go +++ b/pkg/utils/etcdutil/etcdutil.go @@ -20,6 +20,7 @@ import ( "math/rand" "net/http" "net/url" + "sync" "time" "github.com/gogo/protobuf/proto" @@ -27,9 +28,11 @@ import ( "github.com/pingcap/failpoint" "github.com/pingcap/log" "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/utils/logutil" "github.com/tikv/pd/pkg/utils/typeutil" "go.etcd.io/etcd/clientv3" "go.etcd.io/etcd/etcdserver" + "go.etcd.io/etcd/mvcc/mvccpb" "go.etcd.io/etcd/pkg/types" "go.uber.org/zap" ) @@ -349,3 +352,285 @@ func InitOrGetClusterID(c *clientv3.Client, key string) (uint64, error) { return typeutil.BytesToUint64(response.Kvs[0].Value) } + +const ( + defaultLoadDataFromEtcdTimeout = 30 * time.Second + defaultLoadFromEtcdRetryInterval = 200 * time.Millisecond + defaultLoadFromEtcdRetryTimes = int(defaultLoadDataFromEtcdTimeout / defaultLoadFromEtcdRetryInterval) + defaultLoadBatchSize = 400 + defaultWatchChangeRetryInterval = 1 * time.Second +) + +// LoopWatcher loads data from etcd and sets a watcher for it. +type LoopWatcher struct { + ctx context.Context + wg *sync.WaitGroup + name string + client *clientv3.Client + + // key is the etcd key to watch. + key string + // opts is used to set etcd options. + opts []clientv3.OpOption + + // forceLoadCh is used to force loading data from etcd. + forceLoadCh chan struct{} + // isLoadedCh is used to notify that the data has been loaded from etcd first time. + isLoadedCh chan error + + // putFn is used to handle the put event. + putFn func(*mvccpb.KeyValue) error + // deleteFn is used to handle the delete event. + deleteFn func(*mvccpb.KeyValue) error + // postEventFn is used to call after handling all events. + postEventFn func() error + + // loadTimeout is used to set the timeout for loading data from etcd. + loadTimeout time.Duration + // loadRetryTimes is used to set the retry times for loading data from etcd. + loadRetryTimes int + // loadBatchSize is used to set the batch size for loading data from etcd. + loadBatchSize int64 + // watchChangeRetryInterval is used to set the retry interval for watching etcd change. + watchChangeRetryInterval time.Duration + // updateClientCh is used to update the etcd client. + // It's only used for testing. + updateClientCh chan *clientv3.Client +} + +// NewLoopWatcher creates a new LoopWatcher. +func NewLoopWatcher(ctx context.Context, wg *sync.WaitGroup, client *clientv3.Client, name, key string, + putFn, deleteFn func(*mvccpb.KeyValue) error, postEventFn func() error, opts ...clientv3.OpOption) *LoopWatcher { + return &LoopWatcher{ + ctx: ctx, + client: client, + name: name, + key: key, + wg: wg, + forceLoadCh: make(chan struct{}, 1), + isLoadedCh: make(chan error, 1), + updateClientCh: make(chan *clientv3.Client, 1), + putFn: putFn, + deleteFn: deleteFn, + postEventFn: postEventFn, + opts: opts, + loadTimeout: defaultLoadDataFromEtcdTimeout, + loadRetryTimes: defaultLoadFromEtcdRetryTimes, + loadBatchSize: defaultLoadBatchSize, + watchChangeRetryInterval: defaultWatchChangeRetryInterval, + } +} + +// StartWatchLoop starts a loop to watch the key. +func (lw *LoopWatcher) StartWatchLoop() { + defer logutil.LogPanic() + defer lw.wg.Done() + + ctx, cancel := context.WithCancel(lw.ctx) + defer cancel() + watchStartRevision := lw.initFromEtcd(ctx) + + log.Info("start to watch loop", zap.String("name", lw.name), zap.String("key", lw.key)) + for { + select { + case <-ctx.Done(): + log.Info("server is closed, exit watch loop", zap.String("name", lw.name), zap.String("key", lw.key)) + return + default: + } + nextRevision, err := lw.watch(ctx, watchStartRevision) + if err != nil { + log.Error("watcher canceled unexpectedly and a new watcher will start after a while for watch loop", + zap.String("name", lw.name), + zap.String("key", lw.key), + zap.Int64("next-revision", nextRevision), + zap.Time("retry-at", time.Now().Add(lw.watchChangeRetryInterval)), + zap.Error(err)) + watchStartRevision = nextRevision + time.Sleep(lw.watchChangeRetryInterval) + failpoint.Inject("updateClient", func() { + lw.client = <-lw.updateClientCh + }) + } + } +} + +func (lw *LoopWatcher) initFromEtcd(ctx context.Context) int64 { + var ( + watchStartRevision int64 + err error + ) + ticker := time.NewTicker(defaultLoadFromEtcdRetryInterval) + defer ticker.Stop() + ctx, cancel := context.WithTimeout(ctx, lw.loadTimeout) + defer cancel() + + for i := 0; i < lw.loadRetryTimes; i++ { + failpoint.Inject("loadTemporaryFail", func(val failpoint.Value) { + if maxFailTimes, ok := val.(int); ok && i < maxFailTimes { + err = errors.New("fail to read from etcd") + failpoint.Continue() + } + }) + failpoint.Inject("delayLoad", func(val failpoint.Value) { + if sleepIntervalSeconds, ok := val.(int); ok && sleepIntervalSeconds > 0 { + time.Sleep(time.Duration(sleepIntervalSeconds) * time.Second) + } + }) + watchStartRevision, err = lw.load(ctx) + if err == nil { + break + } + select { + case <-ctx.Done(): + lw.isLoadedCh <- errors.Errorf("ctx is done before load data from etcd") + return watchStartRevision + case <-ticker.C: + } + } + if err != nil { + log.Warn("meet error when loading in watch loop", zap.String("name", lw.name), zap.String("key", lw.key), zap.Error(err)) + } else { + log.Info("load finished in watch loop", zap.String("name", lw.name), zap.String("key", lw.key)) + } + lw.isLoadedCh <- err + return watchStartRevision +} + +func (lw *LoopWatcher) watch(ctx context.Context, revision int64) (nextRevision int64, err error) { + watcher := clientv3.NewWatcher(lw.client) + defer watcher.Close() + + for { + WatchChan: + // In order to prevent a watch stream being stuck in a partitioned node, + // make sure to wrap context with "WithRequireLeader". + watchChanCtx, watchChanCancel := context.WithCancel(clientv3.WithRequireLeader(ctx)) + defer watchChanCancel() + opts := append(lw.opts, clientv3.WithRev(revision)) + watchChan := watcher.Watch(watchChanCtx, lw.key, opts...) + select { + case <-ctx.Done(): + return revision, nil + case <-lw.forceLoadCh: + revision, err = lw.load(ctx) + if err != nil { + log.Warn("force load key failed in watch loop", zap.String("name", lw.name), + zap.String("key", lw.key), zap.Error(err)) + } + watchChanCancel() + goto WatchChan + case wresp := <-watchChan: + if wresp.CompactRevision != 0 { + log.Warn("required revision has been compacted, use the compact revision in watch loop", + zap.Int64("required-revision", revision), + zap.Int64("compact-revision", wresp.CompactRevision)) + revision = wresp.CompactRevision + watchChanCancel() + goto WatchChan + } else if wresp.Err() != nil { // wresp.Err() contains CompactRevision not equal to 0 + log.Error("watcher is canceled in watch loop", + zap.Int64("revision", revision), + errs.ZapError(errs.ErrEtcdWatcherCancel, wresp.Err())) + return revision, wresp.Err() + } + for _, event := range wresp.Events { + switch event.Type { + case clientv3.EventTypePut: + if err := lw.putFn(event.Kv); err != nil { + log.Error("put failed in watch loop", zap.String("name", lw.name), + zap.String("key", lw.key), zap.Error(err)) + } + log.Debug("put in watch loop", zap.String("name", lw.name), + zap.ByteString("key", event.Kv.Key), + zap.ByteString("value", event.Kv.Value)) + case clientv3.EventTypeDelete: + if err := lw.deleteFn(event.Kv); err != nil { + log.Error("delete failed in watch loop", zap.String("name", lw.name), + zap.String("key", lw.key), zap.Error(err)) + } + log.Debug("delete in watch loop", zap.String("name", lw.name), + zap.ByteString("key", event.Kv.Key)) + } + } + if err := lw.postEventFn(); err != nil { + log.Error("run post event failed in watch loop", zap.String("name", lw.name), + zap.String("key", lw.key), zap.Error(err)) + } + revision = wresp.Header.Revision + 1 + } + watchChanCancel() + } +} + +func (lw *LoopWatcher) load(ctx context.Context) (nextRevision int64, err error) { + ctx, cancel := context.WithTimeout(ctx, DefaultRequestTimeout) + defer cancel() + startKey := lw.key + // If limit is 0, it means no limit. + // If limit is not 0, we need to add 1 to limit to get the next key. + limit := lw.loadBatchSize + if limit != 0 { + limit++ + } + for { + // Sort by key to get the next key and we don't need to worry about the performance, + // Because the default sort is just SortByKey and SortAscend + opts := append(lw.opts, clientv3.WithSort(clientv3.SortByKey, clientv3.SortAscend), clientv3.WithLimit(limit)) + resp, err := clientv3.NewKV(lw.client).Get(ctx, startKey, opts...) + if err != nil { + log.Error("load failed in watch loop", zap.String("name", lw.name), + zap.String("key", lw.key), zap.Error(err)) + return 0, err + } + for i, item := range resp.Kvs { + if resp.More && i == len(resp.Kvs)-1 { + // The last key is the start key of the next batch. + // To avoid to get the same key in the next load, we need to skip the last key. + startKey = string(item.Key) + continue + } + err = lw.putFn(item) + if err != nil { + log.Error("put failed in watch loop when loading", zap.String("name", lw.name), zap.String("key", lw.key), zap.Error(err)) + } + } + // Note: if there are no keys in etcd, the resp.More is false. It also means the load is finished. + if !resp.More { + if err := lw.postEventFn(); err != nil { + log.Error("run post event failed in watch loop", zap.String("name", lw.name), + zap.String("key", lw.key), zap.Error(err)) + } + log.Info("load finished in watch loop", zap.String("name", lw.name), zap.String("key", lw.key)) + return resp.Header.Revision + 1, err + } + } +} + +// ForceLoad forces to load the key. +func (lw *LoopWatcher) ForceLoad() { + select { + case lw.forceLoadCh <- struct{}{}: + default: + } +} + +// WaitLoad waits for the result to obtain whether data is loaded. +func (lw *LoopWatcher) WaitLoad() error { + return <-lw.isLoadedCh +} + +// SetLoadRetryTimes sets the retry times when loading data from etcd. +func (lw *LoopWatcher) SetLoadRetryTimes(times int) { + lw.loadRetryTimes = times +} + +// SetLoadTimeout sets the timeout when loading data from etcd. +func (lw *LoopWatcher) SetLoadTimeout(timeout time.Duration) { + lw.loadTimeout = timeout +} + +// SetLoadBatchSize sets the batch size when loading data from etcd. +func (lw *LoopWatcher) SetLoadBatchSize(size int64) { + lw.loadBatchSize = size +} diff --git a/pkg/utils/etcdutil/etcdutil_test.go b/pkg/utils/etcdutil/etcdutil_test.go index e63761ca9a9..6bf63db79c9 100644 --- a/pkg/utils/etcdutil/etcdutil_test.go +++ b/pkg/utils/etcdutil/etcdutil_test.go @@ -21,19 +21,28 @@ import ( "io" "net" "strings" + "sync" "sync/atomic" "testing" "time" "github.com/pingcap/failpoint" "github.com/stretchr/testify/require" + "github.com/stretchr/testify/suite" "github.com/tikv/pd/pkg/utils/tempurl" + "github.com/tikv/pd/pkg/utils/testutil" "go.etcd.io/etcd/clientv3" "go.etcd.io/etcd/embed" "go.etcd.io/etcd/etcdserver/etcdserverpb" + "go.etcd.io/etcd/mvcc/mvccpb" "go.etcd.io/etcd/pkg/types" + "go.uber.org/goleak" ) +func TestMain(m *testing.M) { + goleak.VerifyTestMain(m, testutil.LeakOptions...) +} + func TestMemberHelpers(t *testing.T) { re := require.New(t) cfg1 := NewTestSingleConfig(t) @@ -47,6 +56,9 @@ func TestMemberHelpers(t *testing.T) { client1, err := clientv3.New(clientv3.Config{ Endpoints: []string{ep1}, }) + defer func() { + client1.Close() + }() re.NoError(err) <-etcd1.Server.ReadyNotify() @@ -66,6 +78,9 @@ func TestMemberHelpers(t *testing.T) { client2, err := clientv3.New(clientv3.Config{ Endpoints: []string{ep2}, }) + defer func() { + client2.Close() + }() re.NoError(err) checkMembers(re, client2, []*embed.Etcd{etcd1, etcd2}) @@ -98,6 +113,9 @@ func TestEtcdKVGet(t *testing.T) { client, err := clientv3.New(clientv3.Config{ Endpoints: []string{ep}, }) + defer func() { + client.Close() + }() re.NoError(err) <-etcd.Server.ReadyNotify() @@ -148,6 +166,9 @@ func TestEtcdKVPutWithTTL(t *testing.T) { client, err := clientv3.New(clientv3.Config{ Endpoints: []string{ep}, }) + defer func() { + client.Close() + }() re.NoError(err) <-etcd.Server.ReadyNotify() @@ -188,6 +209,9 @@ func TestInitClusterID(t *testing.T) { client, err := clientv3.New(clientv3.Config{ Endpoints: []string{ep}, }) + defer func() { + client.Close() + }() re.NoError(err) <-etcd.Server.ReadyNotify() @@ -214,6 +238,9 @@ func TestEtcdClientSync(t *testing.T) { // Start a etcd server. cfg1 := NewTestSingleConfig(t) etcd1, err := embed.StartEtcd(cfg1) + defer func() { + etcd1.Close() + }() re.NoError(err) // Create a etcd client with etcd1 as endpoint. @@ -221,6 +248,9 @@ func TestEtcdClientSync(t *testing.T) { urls, err := types.NewURLs([]string{ep1}) re.NoError(err) client1, err := createEtcdClientWithMultiEndpoint(nil, urls) + defer func() { + client1.Close() + }() re.NoError(err) <-etcd1.Server.ReadyNotify() @@ -241,7 +271,7 @@ func TestEtcdClientSync(t *testing.T) { re.Len(listResp3.Members, 1) re.Equal(uint64(etcd2.Server.ID()), listResp3.Members[0].ID) - require.NoError(t, failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/autoSyncInterval")) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/autoSyncInterval")) } func TestEtcdWithHangLeaderEnableCheck(t *testing.T) { @@ -251,13 +281,13 @@ func TestEtcdWithHangLeaderEnableCheck(t *testing.T) { re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/utils/etcdutil/autoSyncInterval", "return(true)")) err = checkEtcdWithHangLeader(t) re.NoError(err) - require.NoError(t, failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/autoSyncInterval")) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/autoSyncInterval")) // Test with disable check. re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/utils/etcdutil/closeKeepAliveCheck", "return(true)")) err = checkEtcdWithHangLeader(t) re.Error(err) - require.NoError(t, failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/closeKeepAliveCheck")) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/closeKeepAliveCheck")) } func TestEtcdScaleInAndOutWithoutMultiPoint(t *testing.T) { @@ -265,6 +295,9 @@ func TestEtcdScaleInAndOutWithoutMultiPoint(t *testing.T) { // Start a etcd server. cfg1 := NewTestSingleConfig(t) etcd1, err := embed.StartEtcd(cfg1) + defer func() { + etcd1.Close() + }() re.NoError(err) ep1 := cfg1.LCUrls[0].String() <-etcd1.Server.ReadyNotify() @@ -273,12 +306,21 @@ func TestEtcdScaleInAndOutWithoutMultiPoint(t *testing.T) { urls, err := types.NewURLs([]string{ep1}) re.NoError(err) client1, err := createEtcdClient(nil, urls[0]) // execute member change operation with this client + defer func() { + client1.Close() + }() re.NoError(err) client2, err := createEtcdClient(nil, urls[0]) // check member change with this client + defer func() { + client2.Close() + }() re.NoError(err) // Add a new member and check members etcd2 := checkAddEtcdMember(t, cfg1, client1) + defer func() { + etcd2.Close() + }() checkMembers(re, client2, []*embed.Etcd{etcd1, etcd2}) // scale in etcd1 @@ -292,6 +334,9 @@ func checkEtcdWithHangLeader(t *testing.T) error { // Start a etcd server. cfg1 := NewTestSingleConfig(t) etcd1, err := embed.StartEtcd(cfg1) + defer func() { + etcd1.Close() + }() re.NoError(err) ep1 := cfg1.LCUrls[0].String() <-etcd1.Server.ReadyNotify() @@ -305,6 +350,9 @@ func checkEtcdWithHangLeader(t *testing.T) error { urls, err := types.NewURLs([]string{proxyAddr}) re.NoError(err) client1, err := createEtcdClientWithMultiEndpoint(nil, urls) + defer func() { + client1.Close() + }() re.NoError(err) // Add a new member and set the client endpoints to etcd1 and etcd2. @@ -408,3 +456,288 @@ func ioCopy(dst io.Writer, src io.Reader, enableDiscard *atomic.Bool) (err error } return err } + +type loopWatcherTestSuite struct { + suite.Suite + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup + cleans []func() + etcd *embed.Etcd + client *clientv3.Client + config *embed.Config +} + +func TestLoopWatcherTestSuite(t *testing.T) { + suite.Run(t, new(loopWatcherTestSuite)) +} + +func (suite *loopWatcherTestSuite) SetupSuite() { + t := suite.T() + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + suite.cleans = make([]func(), 0) + // Start a etcd server and create a client with etcd1 as endpoint. + suite.config = NewTestSingleConfig(t) + suite.startEtcd() + ep1 := suite.config.LCUrls[0].String() + urls, err := types.NewURLs([]string{ep1}) + suite.NoError(err) + suite.client, err = createEtcdClient(nil, urls[0]) + suite.NoError(err) + suite.cleans = append(suite.cleans, func() { + suite.client.Close() + }) +} + +func (suite *loopWatcherTestSuite) TearDownSuite() { + suite.cancel() + suite.wg.Wait() + for _, clean := range suite.cleans { + clean() + } +} + +func (suite *loopWatcherTestSuite) TestLoadWithoutKey() { + cache := struct { + sync.RWMutex + data map[string]struct{} + }{ + data: make(map[string]struct{}), + } + watcher := NewLoopWatcher( + suite.ctx, + &suite.wg, + suite.client, + "test", + "TestLoadWithoutKey", + func(kv *mvccpb.KeyValue) error { + cache.Lock() + defer cache.Unlock() + cache.data[string(kv.Key)] = struct{}{} + return nil + }, + func(kv *mvccpb.KeyValue) error { return nil }, + func() error { return nil }, + ) + + suite.wg.Add(1) + go watcher.StartWatchLoop() + err := watcher.WaitLoad() + suite.NoError(err) // although no key, watcher returns no error + cache.RLock() + defer cache.RUnlock() + suite.Len(cache.data, 0) +} + +func (suite *loopWatcherTestSuite) TestCallBack() { + cache := struct { + sync.RWMutex + data map[string]struct{} + }{ + data: make(map[string]struct{}), + } + result := make([]string, 0) + watcher := NewLoopWatcher( + suite.ctx, + &suite.wg, + suite.client, + "test", + "TestCallBack", + func(kv *mvccpb.KeyValue) error { + result = append(result, string(kv.Key)) + return nil + }, + func(kv *mvccpb.KeyValue) error { + cache.Lock() + defer cache.Unlock() + delete(cache.data, string(kv.Key)) + return nil + }, + func() error { + cache.Lock() + defer cache.Unlock() + for _, r := range result { + cache.data[r] = struct{}{} + } + result = result[:0] + return nil + }, + clientv3.WithPrefix(), + ) + + suite.wg.Add(1) + go watcher.StartWatchLoop() + err := watcher.WaitLoad() + suite.NoError(err) + + // put 10 keys + for i := 0; i < 10; i++ { + suite.put(fmt.Sprintf("TestCallBack%d", i), "") + } + time.Sleep(time.Second) + cache.RLock() + suite.Len(cache.data, 10) + cache.RUnlock() + + // delete 10 keys + for i := 0; i < 10; i++ { + key := fmt.Sprintf("TestCallBack%d", i) + _, err = suite.client.Delete(suite.ctx, key) + suite.NoError(err) + } + time.Sleep(time.Second) + cache.RLock() + suite.Empty(cache.data) + cache.RUnlock() +} + +func (suite *loopWatcherTestSuite) TestWatcherLoadLimit() { + for count := 1; count < 10; count++ { + for limit := 0; limit < 10; limit++ { + ctx, cancel := context.WithCancel(suite.ctx) + for i := 0; i < count; i++ { + suite.put(fmt.Sprintf("TestWatcherLoadLimit%d", i), "") + } + cache := struct { + sync.RWMutex + data []string + }{ + data: make([]string, 0), + } + watcher := NewLoopWatcher( + ctx, + &suite.wg, + suite.client, + "test", + "TestWatcherLoadLimit", + func(kv *mvccpb.KeyValue) error { + cache.Lock() + defer cache.Unlock() + cache.data = append(cache.data, string(kv.Key)) + return nil + }, + func(kv *mvccpb.KeyValue) error { + return nil + }, + func() error { + return nil + }, + clientv3.WithPrefix(), + ) + suite.wg.Add(1) + go watcher.StartWatchLoop() + err := watcher.WaitLoad() + suite.NoError(err) + cache.RLock() + suite.Len(cache.data, count) + cache.RUnlock() + cancel() + } + } +} + +func (suite *loopWatcherTestSuite) TestWatcherBreak() { + cache := struct { + sync.RWMutex + data string + }{} + checkCache := func(expect string) { + testutil.Eventually(suite.Require(), func() bool { + cache.RLock() + defer cache.RUnlock() + return cache.data == expect + }, testutil.WithWaitFor(time.Second)) + } + + watcher := NewLoopWatcher( + suite.ctx, + &suite.wg, + suite.client, + "test", + "TestWatcherBreak", + func(kv *mvccpb.KeyValue) error { + if string(kv.Key) == "TestWatcherBreak" { + cache.Lock() + defer cache.Unlock() + cache.data = string(kv.Value) + } + return nil + }, + func(kv *mvccpb.KeyValue) error { return nil }, + func() error { return nil }, + ) + watcher.watchChangeRetryInterval = 100 * time.Millisecond + + suite.wg.Add(1) + go watcher.StartWatchLoop() + err := watcher.WaitLoad() + suite.NoError(err) + checkCache("") + + // we use close client and update client in failpoint to simulate the network error and recover + failpoint.Enable("github.com/tikv/pd/pkg/utils/etcdutil/updateClient", "return(true)") + + // Case1: restart the etcd server + suite.etcd.Close() + suite.startEtcd() + suite.put("TestWatcherBreak", "1") + checkCache("1") + + // Case2: close the etcd client and put a new value after watcher restarts + suite.client.Close() + suite.client, err = createEtcdClient(nil, suite.config.LCUrls[0]) + suite.NoError(err) + watcher.updateClientCh <- suite.client + suite.put("TestWatcherBreak", "2") + checkCache("2") + + // Case3: close the etcd client and put a new value before watcher restarts + suite.client.Close() + suite.client, err = createEtcdClient(nil, suite.config.LCUrls[0]) + suite.NoError(err) + suite.put("TestWatcherBreak", "3") + watcher.updateClientCh <- suite.client + checkCache("3") + + // Case4: close the etcd client and put a new value with compact + suite.client.Close() + suite.client, err = createEtcdClient(nil, suite.config.LCUrls[0]) + suite.NoError(err) + suite.put("TestWatcherBreak", "4") + resp, err := EtcdKVGet(suite.client, "TestWatcherBreak") + suite.NoError(err) + revision := resp.Header.Revision + resp2, err := suite.etcd.Server.Compact(suite.ctx, &etcdserverpb.CompactionRequest{Revision: revision}) + suite.NoError(err) + suite.Equal(revision, resp2.Header.Revision) + watcher.updateClientCh <- suite.client + checkCache("4") + + // Case5: there is an error data in cache + cache.Lock() + cache.data = "error" + cache.Unlock() + watcher.ForceLoad() + checkCache("4") + + failpoint.Disable("github.com/tikv/pd/pkg/utils/etcdutil/updateClient") +} + +func (suite *loopWatcherTestSuite) startEtcd() { + etcd1, err := embed.StartEtcd(suite.config) + suite.NoError(err) + suite.etcd = etcd1 + <-etcd1.Server.ReadyNotify() + suite.cleans = append(suite.cleans, func() { + suite.etcd.Close() + }) +} + +func (suite *loopWatcherTestSuite) put(key, value string) { + kv := clientv3.NewKV(suite.client) + _, err := kv.Put(suite.ctx, key, value) + suite.NoError(err) + resp, err := kv.Get(suite.ctx, key) + suite.NoError(err) + suite.Equal(value, string(resp.Kvs[0].Value)) +} diff --git a/pkg/utils/grpcutil/grpcutil_test.go b/pkg/utils/grpcutil/grpcutil_test.go index ac3ba3da3fc..ea1de4fb681 100644 --- a/pkg/utils/grpcutil/grpcutil_test.go +++ b/pkg/utils/grpcutil/grpcutil_test.go @@ -24,9 +24,9 @@ func TestToTLSConfig(t *testing.T) { t.Parallel() re := require.New(t) tlsConfig := TLSConfig{ - KeyPath: "../../../tests/client/cert/pd-server-key.pem", - CertPath: "../../../tests/client/cert/pd-server.pem", - CAPath: "../../../tests/client/cert/ca.pem", + KeyPath: "../../../tests/integrations/client/cert/pd-server-key.pem", + CertPath: "../../../tests/integrations/client/cert/pd-server.pem", + CAPath: "../../../tests/integrations/client/cert/ca.pem", } // test without bytes _, err := tlsConfig.ToTLSConfig() diff --git a/pkg/utils/testutil/operator_check.go b/pkg/utils/operatorutil/operator_check.go similarity index 99% rename from pkg/utils/testutil/operator_check.go rename to pkg/utils/operatorutil/operator_check.go index 178cb9b575d..7456b77146e 100644 --- a/pkg/utils/testutil/operator_check.go +++ b/pkg/utils/operatorutil/operator_check.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package testutil +package operatorutil import ( "github.com/stretchr/testify/require" diff --git a/pkg/utils/testutil/testutil.go b/pkg/utils/testutil/testutil.go index 7fc970afdab..7d31f2263c6 100644 --- a/pkg/utils/testutil/testutil.go +++ b/pkg/utils/testutil/testutil.go @@ -29,6 +29,9 @@ const ( defaultTickInterval = time.Millisecond * 100 ) +// CleanupFunc closes test pd server(s) and deletes any files left behind. +type CleanupFunc func() + // WaitOp represents available options when execute Eventually. type WaitOp struct { waitFor time.Duration diff --git a/pkg/utils/tsoutil/tso_dispatcher.go b/pkg/utils/tsoutil/tso_dispatcher.go new file mode 100644 index 00000000000..187fd7a527b --- /dev/null +++ b/pkg/utils/tsoutil/tso_dispatcher.go @@ -0,0 +1,243 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tsoutil + +import ( + "context" + "strings" + "sync" + "time" + + "github.com/pingcap/kvproto/pkg/pdpb" + "github.com/pingcap/log" + "github.com/prometheus/client_golang/prometheus" + "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/utils/etcdutil" + "github.com/tikv/pd/pkg/utils/logutil" + "go.uber.org/zap" + "google.golang.org/grpc" +) + +const ( + maxMergeRequests = 10000 + // DefaultTSOProxyTimeout defines the default timeout value of TSP Proxying + DefaultTSOProxyTimeout = 3 * time.Second +) + +type tsoResp interface { + GetTimestamp() *pdpb.Timestamp +} + +// TSODispatcher dispatches the TSO requests to the corresponding forwarding TSO channels. +type TSODispatcher struct { + tsoProxyHandleDuration prometheus.Histogram + tsoProxyBatchSize prometheus.Histogram + + // dispatchChs is used to dispatch different TSO requests to the corresponding forwarding TSO channels. + dispatchChs sync.Map // Store as map[string]chan Request +} + +// NewTSODispatcher creates and returns a TSODispatcher +func NewTSODispatcher(tsoProxyHandleDuration, tsoProxyBatchSize prometheus.Histogram) *TSODispatcher { + tsoDispatcher := &TSODispatcher{ + tsoProxyHandleDuration: tsoProxyHandleDuration, + tsoProxyBatchSize: tsoProxyBatchSize, + } + return tsoDispatcher +} + +// DispatchRequest is the entry point for dispatching/forwarding a tso request to the detination host +func (s *TSODispatcher) DispatchRequest( + ctx context.Context, + req Request, + tsoProtoFactory ProtoFactory, + doneCh <-chan struct{}, + errCh chan<- error, + tsoPrimaryWatchers ...*etcdutil.LoopWatcher) { + val, loaded := s.dispatchChs.LoadOrStore(req.getForwardedHost(), make(chan Request, maxMergeRequests)) + reqCh := val.(chan Request) + if !loaded { + tsDeadlineCh := make(chan deadline, 1) + go s.dispatch(ctx, tsoProtoFactory, req.getForwardedHost(), req.getClientConn(), reqCh, tsDeadlineCh, doneCh, errCh, tsoPrimaryWatchers...) + go watchTSDeadline(ctx, tsDeadlineCh) + } + reqCh <- req +} + +func (s *TSODispatcher) dispatch( + ctx context.Context, + tsoProtoFactory ProtoFactory, + forwardedHost string, + clientConn *grpc.ClientConn, + tsoRequestCh <-chan Request, + tsDeadlineCh chan<- deadline, + doneCh <-chan struct{}, + errCh chan<- error, + tsoPrimaryWatchers ...*etcdutil.LoopWatcher) { + defer logutil.LogPanic() + dispatcherCtx, ctxCancel := context.WithCancel(ctx) + defer ctxCancel() + defer s.dispatchChs.Delete(forwardedHost) + + log.Info("create tso forward stream", zap.String("forwarded-host", forwardedHost)) + forwardStream, cancel, err := tsoProtoFactory.createForwardStream(ctx, clientConn) + if err != nil || forwardStream == nil { + log.Error("create tso forwarding stream error", + zap.String("forwarded-host", forwardedHost), + errs.ZapError(errs.ErrGRPCCreateStream, err)) + select { + case <-dispatcherCtx.Done(): + return + case _, ok := <-doneCh: + if !ok { + return + } + case errCh <- err: + close(errCh) + return + } + } + defer cancel() + + requests := make([]Request, maxMergeRequests+1) + needUpdateServicePrimaryAddr := len(tsoPrimaryWatchers) > 0 && tsoPrimaryWatchers[0] != nil + for { + select { + case first := <-tsoRequestCh: + pendingTSOReqCount := len(tsoRequestCh) + 1 + requests[0] = first + for i := 1; i < pendingTSOReqCount; i++ { + requests[i] = <-tsoRequestCh + } + done := make(chan struct{}) + dl := deadline{ + timer: time.After(DefaultTSOProxyTimeout), + done: done, + cancel: cancel, + } + select { + case tsDeadlineCh <- dl: + case <-dispatcherCtx.Done(): + return + } + err = s.processRequests(forwardStream, requests[:pendingTSOReqCount], tsoProtoFactory) + close(done) + if err != nil { + log.Error("proxy forward tso error", + zap.String("forwarded-host", forwardedHost), + errs.ZapError(errs.ErrGRPCSend, err)) + if needUpdateServicePrimaryAddr && strings.Contains(err.Error(), errs.NotLeaderErr) { + tsoPrimaryWatchers[0].ForceLoad() + } + select { + case <-dispatcherCtx.Done(): + return + case _, ok := <-doneCh: + if !ok { + return + } + case errCh <- err: + close(errCh) + return + } + } + case <-dispatcherCtx.Done(): + return + } + } +} + +func (s *TSODispatcher) processRequests(forwardStream stream, requests []Request, tsoProtoFactory ProtoFactory) error { + // Merge the requests + count := uint32(0) + for _, request := range requests { + count += request.getCount() + } + + start := time.Now() + resp, err := requests[0].process(forwardStream, count, tsoProtoFactory) + if err != nil { + return err + } + s.tsoProxyHandleDuration.Observe(time.Since(start).Seconds()) + s.tsoProxyBatchSize.Observe(float64(count)) + // Split the response + ts := resp.GetTimestamp() + physical, logical, suffixBits := ts.GetPhysical(), ts.GetLogical(), ts.GetSuffixBits() + // `logical` is the largest ts's logical part here, we need to do the subtracting before we finish each TSO request. + // This is different from the logic of client batch, for example, if we have a largest ts whose logical part is 10, + // count is 5, then the splitting results should be 5 and 10. + firstLogical := addLogical(logical, -int64(count), suffixBits) + return s.finishRequest(requests, physical, firstLogical, suffixBits) +} + +// Because of the suffix, we need to shift the count before we add it to the logical part. +func addLogical(logical, count int64, suffixBits uint32) int64 { + return logical + count< +# ./ci-subtask.sh -packages=(`go list ./...`) -dirs=(`find . -iname "*_test.go" -exec dirname {} \; | sort -u | sed -e "s/^\./github.com\/tikv\/pd/"`) -tasks=($(comm -12 <(printf "%s\n" "${packages[@]}") <(printf "%s\n" "${dirs[@]}"))) +if [[ $3 ]]; then + # Get integration test list. + makefile_dirs=($(find . -iname "Makefile" -exec dirname {} \; | sort -u)) + submod_dirs=($(find . -iname "go.mod" -exec dirname {} \; | sort -u)) + integration_tasks=$(comm -12 <(printf "%s\n" "${makefile_dirs[@]}") <(printf "%s\n" "${submod_dirs[@]}") | grep "./tests/integrations/*") + # Currently, we only have 3 integration tests, so we can hardcode the task index. + for t in ${integration_tasks[@]}; do + if [[ "$t" = "./tests/integrations/client" && "$2" = 9 ]]; then + printf "%s " "$t" + break + elif [[ "$t" = "./tests/integrations/tso" && "$2" = 7 ]]; then + printf "%s " "$t" + break + elif [[ "$t" = "./tests/integrations/mcs" && "$2" = 2 ]]; then + printf "%s " "$t" + break + fi + done +else + # Get package test list. + packages=($(go list ./...)) + dirs=($(find . -iname "*_test.go" -exec dirname {} \; | sort -u | sed -e "s/^\./github.com\/tikv\/pd/")) + tasks=($(comm -12 <(printf "%s\n" "${packages[@]}") <(printf "%s\n" "${dirs[@]}"))) -weight () { - [[ $1 == "github.com/tikv/pd/server/api" ]] && return 30 - [[ $1 == "github.com/tikv/pd/server/schedule" ]] && return 30 - [[ $1 =~ "pd/tests" ]] && return 5 - return 1 -} + weight() { + [[ $1 == "github.com/tikv/pd/server/api" ]] && return 30 + [[ $1 == "github.com/tikv/pd/pkg/schedule" ]] && return 30 + [[ $1 =~ "pd/tests" ]] && return 5 + return 1 + } -scores=(`seq "$1" | xargs -I{} echo 0`) + scores=($(seq "$1" | xargs -I{} echo 0)) -res=() -for t in ${tasks[@]}; do - min_i=0 - for i in ${!scores[@]}; do - [[ ${scores[i]} -lt ${scores[$min_i]} ]] && min_i=$i + res=() + for t in ${tasks[@]}; do + min_i=0 + for i in ${!scores[@]}; do + [[ ${scores[i]} -lt ${scores[$min_i]} ]] && min_i=$i + done + weight $t + scores[$min_i]=$((${scores[$min_i]} + $?)) + [[ $(($min_i + 1)) -eq $2 ]] && res+=($t) done - weight $t - scores[$min_i]=$((${scores[$min_i]} + $?)) - [[ $(($min_i+1)) -eq $2 ]] && res+=($t) -done -printf "%s " "${res[@]}" + printf "%s " "${res[@]}" +fi diff --git a/server/api/region.go b/server/api/region.go index 95ee732b982..ba5e53b082c 100644 --- a/server/api/region.go +++ b/server/api/region.go @@ -30,10 +30,10 @@ import ( "github.com/pingcap/kvproto/pkg/replication_modepb" "github.com/pingcap/log" "github.com/tikv/pd/pkg/core" + "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/utils/apiutil" "github.com/tikv/pd/pkg/utils/typeutil" "github.com/tikv/pd/server" - "github.com/tikv/pd/server/keyspace" "github.com/tikv/pd/server/schedule/filter" "github.com/tikv/pd/server/statistics" "github.com/unrolled/render" diff --git a/server/api/region_label_test.go b/server/api/region_label_test.go index a24345f6aac..2476f4ba1e9 100644 --- a/server/api/region_label_test.go +++ b/server/api/region_label_test.go @@ -47,13 +47,13 @@ func (suite *regionLabelTestSuite) SetupSuite() { addr := suite.svr.GetAddr() suite.urlPrefix = fmt.Sprintf("%s%s/api/v1/config/region-label/", addr, apiPrefix) - suite.NoError(failpoint.Enable("github.com/tikv/pd/server/keyspace/skipSplitRegion", "return(true)")) + suite.NoError(failpoint.Enable("github.com/tikv/pd/pkg/keyspace/skipSplitRegion", "return(true)")) mustBootstrapCluster(re, suite.svr) } func (suite *regionLabelTestSuite) TearDownSuite() { suite.cleanup() - suite.NoError(failpoint.Disable("github.com/tikv/pd/server/keyspace/skipSplitRegion")) + suite.NoError(failpoint.Disable("github.com/tikv/pd/pkg/keyspace/skipSplitRegion")) } func (suite *regionLabelTestSuite) TestGetSet() { diff --git a/server/apiv2/handlers/keyspace.go b/server/apiv2/handlers/keyspace.go index db4b1433e3a..d9c4251ffed 100644 --- a/server/apiv2/handlers/keyspace.go +++ b/server/apiv2/handlers/keyspace.go @@ -25,11 +25,13 @@ import ( "github.com/pingcap/errors" "github.com/pingcap/kvproto/pkg/keyspacepb" "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/server" "github.com/tikv/pd/server/apiv2/middlewares" - "github.com/tikv/pd/server/keyspace" ) +const managerUninitializedErr = "keyspace manager is not initialized" + // RegisterKeyspace register keyspace related handlers to router paths. func RegisterKeyspace(r *gin.RouterGroup) { router := r.Group("keyspaces") @@ -65,8 +67,12 @@ type CreateKeyspaceParams struct { // @Failure 500 {string} string "PD server failed to proceed the request." // @Router /keyspaces [post] func CreateKeyspace(c *gin.Context) { - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) manager := svr.GetKeyspaceManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, managerUninitializedErr) + return + } createParams := &CreateKeyspaceParams{} err := c.BindJSON(createParams) if err != nil { @@ -76,8 +82,8 @@ func CreateKeyspace(c *gin.Context) { req := &keyspace.CreateKeyspaceRequest{ Name: createParams.Name, Config: createParams.Config, - Now: time.Now().Unix(), IsPreAlloc: false, + CreateTime: time.Now().Unix(), } meta, err := manager.CreateKeyspace(req) if err != nil { @@ -96,8 +102,12 @@ func CreateKeyspace(c *gin.Context) { // @Failure 500 {string} string "PD server failed to proceed the request." // @Router /keyspaces/{name} [get] func LoadKeyspace(c *gin.Context) { - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) manager := svr.GetKeyspaceManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, managerUninitializedErr) + return + } name := c.Param("name") meta, err := manager.LoadKeyspace(name) if err != nil { @@ -121,8 +131,12 @@ func LoadKeyspaceByID(c *gin.Context) { c.AbortWithStatusJSON(http.StatusInternalServerError, "invalid keyspace id") return } - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) manager := svr.GetKeyspaceManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, managerUninitializedErr) + return + } meta, err := manager.LoadKeyspaceByID(uint32(id)) if err != nil { c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) @@ -131,18 +145,18 @@ func LoadKeyspaceByID(c *gin.Context) { c.IndentedJSON(http.StatusOK, &KeyspaceMeta{meta}) } -// parseLoadAllQuery parses LoadAllKeyspaces' query parameters. +// parseLoadAllQuery parses LoadAllKeyspaces'/GetKeyspaceGroups' query parameters. // page_token: -// The keyspace id of the scan start. If not set, scan from keyspace with id 1. -// It's string of spaceID of the previous scan result's last element (next_page_token). +// The keyspace/keyspace group id of the scan start. If not set, scan from keyspace/keyspace group with id 1. +// It's string of ID of the previous scan result's last element (next_page_token). // limit: -// The maximum number of keyspace metas to return. If not set, no limit is posed. -// Every scan scans limit + 1 keyspaces (if limit != 0), the extra scanned keyspace +// The maximum number of keyspace metas/keyspace groups to return. If not set, no limit is posed. +// Every scan scans limit + 1 keyspaces/keyspace groups (if limit != 0), the extra scanned keyspace/keyspace group // is to check if there's more, and used to set next_page_token in response. func parseLoadAllQuery(c *gin.Context) (scanStart uint32, scanLimit int, err error) { pageToken, set := c.GetQuery("page_token") if !set || pageToken == "" { - // If pageToken is empty or unset, then scan from spaceID of 1. + // If pageToken is empty or unset, then scan from ID of 1. scanStart = 0 } else { scanStart64, err := strconv.ParseUint(pageToken, 10, 32) @@ -188,8 +202,12 @@ type LoadAllKeyspacesResponse struct { // @Failure 500 {string} string "PD server failed to proceed the request." // @Router /keyspaces [get] func LoadAllKeyspaces(c *gin.Context) { - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) manager := svr.GetKeyspaceManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, managerUninitializedErr) + return + } scanStart, scanLimit, err := parseLoadAllQuery(c) if err != nil { c.AbortWithStatusJSON(http.StatusBadRequest, err.Error()) @@ -247,8 +265,12 @@ type UpdateConfigParams struct { // @Failure 500 {string} string "PD server failed to proceed the request." // Router /keyspaces/{name}/config [patch] func UpdateKeyspaceConfig(c *gin.Context) { - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) manager := svr.GetKeyspaceManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, managerUninitializedErr) + return + } name := c.Param("name") configParams := &UpdateConfigParams{} err := c.BindJSON(configParams) @@ -302,8 +324,12 @@ type UpdateStateParam struct { // @Failure 500 {string} string "PD server failed to proceed the request." // Router /keyspaces/{name}/state [put] func UpdateKeyspaceState(c *gin.Context) { - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) manager := svr.GetKeyspaceManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, managerUninitializedErr) + return + } name := c.Param("name") param := &UpdateStateParam{} err := c.BindJSON(param) @@ -386,7 +412,7 @@ func (meta *KeyspaceMeta) UnmarshalJSON(data []byte) error { // // Deprecated: use PUT /keyspace/{name}/state instead. func EnableKeyspace(c *gin.Context) { - updateKeyspaceState(c, keyspacepb.KeyspaceState_ENABLED) + updateState(c, keyspacepb.KeyspaceState_ENABLED) } // DisableKeyspace disables target keyspace. @@ -400,7 +426,7 @@ func EnableKeyspace(c *gin.Context) { // // Deprecated: use PUT /keyspace/{name}/state instead. func DisableKeyspace(c *gin.Context) { - updateKeyspaceState(c, keyspacepb.KeyspaceState_DISABLED) + updateState(c, keyspacepb.KeyspaceState_DISABLED) } // ArchiveKeyspace archives target keyspace. @@ -414,10 +440,10 @@ func DisableKeyspace(c *gin.Context) { // // Deprecated: use PUT /keyspace/{name}/state instead. func ArchiveKeyspace(c *gin.Context) { - updateKeyspaceState(c, keyspacepb.KeyspaceState_ARCHIVED) + updateState(c, keyspacepb.KeyspaceState_ARCHIVED) } -func updateKeyspaceState(c *gin.Context, state keyspacepb.KeyspaceState) { +func updateState(c *gin.Context, state keyspacepb.KeyspaceState) { svr := c.MustGet("server").(*server.Server) manager := svr.GetKeyspaceManager() name := c.Param("name") diff --git a/server/apiv2/handlers/tso_keyspace_group.go b/server/apiv2/handlers/tso_keyspace_group.go new file mode 100644 index 00000000000..02a4c49fc86 --- /dev/null +++ b/server/apiv2/handlers/tso_keyspace_group.go @@ -0,0 +1,351 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package handlers + +import ( + "net/http" + "strconv" + "sync" + + "github.com/gin-gonic/gin" + "github.com/pkg/errors" + "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/server" + "github.com/tikv/pd/server/apiv2/middlewares" +) + +const groupManagerUninitializedErr = "keyspace group manager is not initialized" + +// RegisterTSOKeyspaceGroup registers keyspace group handlers to the server. +func RegisterTSOKeyspaceGroup(r *gin.RouterGroup) { + router := r.Group("tso/keyspace-groups") + router.Use(middlewares.BootstrapChecker()) + router.POST("", CreateKeyspaceGroups) + router.GET("", GetKeyspaceGroups) + router.GET("/:id", GetKeyspaceGroupByID) + router.DELETE("/:id", DeleteKeyspaceGroupByID) + router.POST("/:id/alloc", AllocNodesForKeyspaceGroup) + router.POST("/:id/nodes", SetNodesForKeyspaceGroup) + router.POST("/:id/split", SplitKeyspaceGroupByID) + router.DELETE("/:id/split", FinishSplitKeyspaceByID) +} + +// CreateKeyspaceGroupParams defines the params for creating keyspace groups. +type CreateKeyspaceGroupParams struct { + KeyspaceGroups []*endpoint.KeyspaceGroup `json:"keyspace-groups"` +} + +// CreateKeyspaceGroups creates keyspace groups. +func CreateKeyspaceGroups(c *gin.Context) { + createParams := &CreateKeyspaceGroupParams{} + err := c.BindJSON(createParams) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, errs.ErrBindJSON.Wrap(err).GenWithStackByCause()) + return + } + for _, keyspaceGroup := range createParams.KeyspaceGroups { + if !isValid(keyspaceGroup.ID) { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + if keyspaceGroup.UserKind == "" { + keyspaceGroup.UserKind = endpoint.Basic.String() + } else if !endpoint.IsUserKindValid(keyspaceGroup.UserKind) { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid user kind") + return + } + } + + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + manager := svr.GetKeyspaceGroupManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, groupManagerUninitializedErr) + return + } + err = manager.CreateKeyspaceGroups(createParams.KeyspaceGroups) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + c.JSON(http.StatusOK, nil) +} + +// GetKeyspaceGroups gets keyspace groups from the start ID with limit. +// If limit is 0, it will load all keyspace groups from the start ID. +func GetKeyspaceGroups(c *gin.Context) { + scanStart, scanLimit, err := parseLoadAllQuery(c) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, err.Error()) + return + } + + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + manager := svr.GetKeyspaceGroupManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, groupManagerUninitializedErr) + return + } + keyspaceGroups, err := manager.GetKeyspaceGroups(scanStart, scanLimit) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + + c.IndentedJSON(http.StatusOK, keyspaceGroups) +} + +// GetKeyspaceGroupByID gets keyspace group by ID. +func GetKeyspaceGroupByID(c *gin.Context) { + id, err := validateKeyspaceGroupID(c) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + manager := svr.GetKeyspaceGroupManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, groupManagerUninitializedErr) + return + } + kg, err := manager.GetKeyspaceGroupByID(id) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + + c.IndentedJSON(http.StatusOK, kg) +} + +// DeleteKeyspaceGroupByID deletes keyspace group by ID. +func DeleteKeyspaceGroupByID(c *gin.Context) { + id, err := validateKeyspaceGroupID(c) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + manager := svr.GetKeyspaceGroupManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, groupManagerUninitializedErr) + return + } + kg, err := manager.DeleteKeyspaceGroupByID(id) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + c.IndentedJSON(http.StatusOK, kg) +} + +// SplitKeyspaceGroupByIDParams defines the params for splitting a keyspace group. +type SplitKeyspaceGroupByIDParams struct { + NewID uint32 `json:"new-id"` + Keyspaces []uint32 `json:"keyspaces"` +} + +var patrolKeyspaceAssignmentState struct { + sync.RWMutex + patrolled bool +} + +// SplitKeyspaceGroupByID splits keyspace group by ID into a new keyspace group with the given new ID. +// And the keyspaces in the old keyspace group will be moved to the new keyspace group. +func SplitKeyspaceGroupByID(c *gin.Context) { + id, err := validateKeyspaceGroupID(c) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + splitParams := &SplitKeyspaceGroupByIDParams{} + err = c.BindJSON(splitParams) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, errs.ErrBindJSON.Wrap(err).GenWithStackByCause()) + return + } + if !isValid(splitParams.NewID) { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + if len(splitParams.Keyspaces) == 0 { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid empty keyspaces") + return + } + + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + patrolKeyspaceAssignmentState.Lock() + if !patrolKeyspaceAssignmentState.patrolled { + // Patrol keyspace assignment before splitting keyspace group. + manager := svr.GetKeyspaceManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, managerUninitializedErr) + return + } + err = manager.PatrolKeyspaceAssignment() + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + patrolKeyspaceAssignmentState.Unlock() + return + } + patrolKeyspaceAssignmentState.patrolled = true + } + patrolKeyspaceAssignmentState.Unlock() + // Split keyspace group. + groupManager := svr.GetKeyspaceGroupManager() + if groupManager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, groupManagerUninitializedErr) + return + } + err = groupManager.SplitKeyspaceGroupByID(id, splitParams.NewID, splitParams.Keyspaces) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + c.JSON(http.StatusOK, nil) +} + +// FinishSplitKeyspaceByID finishes split keyspace group by ID. +func FinishSplitKeyspaceByID(c *gin.Context) { + id, err := validateKeyspaceGroupID(c) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + manager := svr.GetKeyspaceGroupManager() + err = manager.FinishSplitKeyspaceByID(id) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + c.JSON(http.StatusOK, nil) +} + +// AllocNodesForKeyspaceGroupParams defines the params for allocating nodes for keyspace groups. +type AllocNodesForKeyspaceGroupParams struct { + Replica int `json:"replica"` +} + +// AllocNodesForKeyspaceGroup allocates nodes for keyspace group. +func AllocNodesForKeyspaceGroup(c *gin.Context) { + id, err := validateKeyspaceGroupID(c) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + manager := svr.GetKeyspaceGroupManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, groupManagerUninitializedErr) + return + } + allocParams := &AllocNodesForKeyspaceGroupParams{} + err = c.BindJSON(allocParams) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, errs.ErrBindJSON.Wrap(err).GenWithStackByCause()) + return + } + if manager.GetNodesCount() < allocParams.Replica || allocParams.Replica < utils.KeyspaceGroupDefaultReplicaCount { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid replica, should be in [2, nodes_num]") + return + } + keyspaceGroup, err := manager.GetKeyspaceGroupByID(id) + if err != nil || keyspaceGroup == nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "keyspace group does not exist") + return + } + if len(keyspaceGroup.Members) >= allocParams.Replica { + c.AbortWithStatusJSON(http.StatusBadRequest, "existed replica is larger than the new replica") + return + } + // get the nodes + nodes, err := manager.AllocNodesForKeyspaceGroup(id, allocParams.Replica) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + c.JSON(http.StatusOK, nodes) +} + +// SetNodesForKeyspaceGroupParams defines the params for setting nodes for keyspace groups. +// Notes: it should be used carefully. +type SetNodesForKeyspaceGroupParams struct { + Nodes []string `json:"nodes"` +} + +// SetNodesForKeyspaceGroup sets nodes for keyspace group. +func SetNodesForKeyspaceGroup(c *gin.Context) { + id, err := validateKeyspaceGroupID(c) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid keyspace group id") + return + } + svr := c.MustGet(middlewares.ServerContextKey).(*server.Server) + manager := svr.GetKeyspaceGroupManager() + if manager == nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, groupManagerUninitializedErr) + return + } + setParams := &SetNodesForKeyspaceGroupParams{} + err = c.BindJSON(setParams) + if err != nil { + c.AbortWithStatusJSON(http.StatusBadRequest, errs.ErrBindJSON.Wrap(err).GenWithStackByCause()) + return + } + // check if keyspace group exists + keyspaceGroup, err := manager.GetKeyspaceGroupByID(id) + if err != nil || keyspaceGroup == nil { + c.AbortWithStatusJSON(http.StatusBadRequest, "keyspace group does not exist") + return + } + // check if nodes is less than default replica count + if len(setParams.Nodes) < utils.KeyspaceGroupDefaultReplicaCount { + c.AbortWithStatusJSON(http.StatusBadRequest, "invalid num of nodes") + return + } + // check if node exists + for _, node := range setParams.Nodes { + if !manager.IsExistNode(node) { + c.AbortWithStatusJSON(http.StatusBadRequest, "node does not exist") + return + } + } + // set nodes + err = manager.SetNodesForKeyspaceGroup(id, setParams.Nodes) + if err != nil { + c.AbortWithStatusJSON(http.StatusInternalServerError, err.Error()) + return + } + c.JSON(http.StatusOK, nil) +} + +func validateKeyspaceGroupID(c *gin.Context) (uint32, error) { + id, err := strconv.ParseUint(c.Param("id"), 10, 64) + if err != nil { + return 0, err + } + if !isValid(uint32(id)) { + return 0, errors.Errorf("invalid keyspace group id: %d", id) + } + return uint32(id), nil +} + +func isValid(id uint32) bool { + return id >= utils.DefaultKeyspaceGroupID && id <= utils.MaxKeyspaceGroupCountInUse +} diff --git a/server/apiv2/middlewares/bootstrap_checker.go b/server/apiv2/middlewares/bootstrap_checker.go index 384847be931..794316d3d0f 100644 --- a/server/apiv2/middlewares/bootstrap_checker.go +++ b/server/apiv2/middlewares/bootstrap_checker.go @@ -22,10 +22,13 @@ import ( "github.com/tikv/pd/server" ) +// ServerContextKey is the key to get server from gin.Context. +const ServerContextKey = "server" + // BootstrapChecker is a middleware to check if raft cluster is started. func BootstrapChecker() gin.HandlerFunc { return func(c *gin.Context) { - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(ServerContextKey).(*server.Server) rc := svr.GetRaftCluster() if rc == nil { c.AbortWithStatusJSON(http.StatusInternalServerError, errs.ErrNotBootstrapped.FastGenByArgs().Error()) diff --git a/server/apiv2/middlewares/redirector.go b/server/apiv2/middlewares/redirector.go index 6f92c15754e..5539dd089dc 100644 --- a/server/apiv2/middlewares/redirector.go +++ b/server/apiv2/middlewares/redirector.go @@ -30,7 +30,7 @@ import ( // Redirector is a middleware to redirect the request to the right place. func Redirector() gin.HandlerFunc { return func(c *gin.Context) { - svr := c.MustGet("server").(*server.Server) + svr := c.MustGet(ServerContextKey).(*server.Server) allowFollowerHandle := len(c.Request.Header.Get(serverapi.PDAllowFollowerHandle)) > 0 isLeader := svr.GetMember().IsLeader() if !svr.IsClosed() && (allowFollowerHandle || isLeader) { diff --git a/server/apiv2/router.go b/server/apiv2/router.go index 5307b2b5c0f..383d336caae 100644 --- a/server/apiv2/router.go +++ b/server/apiv2/router.go @@ -57,11 +57,12 @@ func NewV2Handler(_ context.Context, svr *server.Server) (http.Handler, apiutil. }) router := gin.New() router.Use(func(c *gin.Context) { - c.Set("server", svr) + c.Set(middlewares.ServerContextKey, svr) c.Next() }) router.Use(middlewares.Redirector()) root := router.Group(apiV2Prefix) handlers.RegisterKeyspace(root) + handlers.RegisterTSOKeyspaceGroup(root) return router, group, nil } diff --git a/server/cluster/cluster.go b/server/cluster/cluster.go index bd33d2b996e..880794b63e8 100644 --- a/server/cluster/cluster.go +++ b/server/cluster/cluster.go @@ -22,7 +22,6 @@ import ( "strconv" "strings" "sync" - "sync/atomic" "time" "github.com/coreos/go-semver/semver" @@ -36,6 +35,7 @@ import ( "github.com/tikv/pd/pkg/errs" "github.com/tikv/pd/pkg/gctuner" "github.com/tikv/pd/pkg/id" + "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/memory" "github.com/tikv/pd/pkg/progress" "github.com/tikv/pd/pkg/slice" @@ -106,6 +106,8 @@ type Server interface { GetBasicCluster() *core.BasicCluster GetMembers() ([]*pdpb.Member, error) ReplicateFileToMember(ctx context.Context, member *pdpb.Member, name string, data []byte) error + GetKeyspaceGroupManager() *keyspace.GroupManager + IsAPIServiceMode() bool } // RaftCluster is used for cluster config management. @@ -126,7 +128,7 @@ type RaftCluster struct { etcdClient *clientv3.Client httpClient *http.Client - running atomic.Bool + running bool meta *metapb.Cluster storeConfigManager *config.StoreConfigManager storage storage.Storage @@ -154,6 +156,7 @@ type RaftCluster struct { progressManager *progress.Manager regionSyncer *syncer.RegionSyncer changedRegions chan *core.RegionInfo + keyspaceGroupManager *keyspace.GroupManager } // Status saves some state information. @@ -231,7 +234,8 @@ func (c *RaftCluster) InitCluster( id id.Allocator, opt *config.PersistOptions, storage storage.Storage, - basicCluster *core.BasicCluster) { + basicCluster *core.BasicCluster, + keyspaceGroupManager *keyspace.GroupManager) { c.core, c.opt, c.storage, c.id = basicCluster, opt, storage, id c.ctx, c.cancel = context.WithCancel(c.serverCtx) c.labelLevelStats = statistics.NewLabelStatistics() @@ -241,19 +245,20 @@ func (c *RaftCluster) InitCluster( c.changedRegions = make(chan *core.RegionInfo, defaultChangedRegionsLimit) c.prevStoreLimit = make(map[uint64]map[storelimit.Type]float64) c.unsafeRecoveryController = newUnsafeRecoveryController(c) + c.keyspaceGroupManager = keyspaceGroupManager } // Start starts a cluster. func (c *RaftCluster) Start(s Server) error { - if c.IsRunning() { + c.Lock() + defer c.Unlock() + + if c.running { log.Warn("raft cluster has already been started") return nil } - c.Lock() - defer c.Unlock() - - c.InitCluster(s.GetAllocator(), s.GetPersistOptions(), s.GetStorage(), s.GetBasicCluster()) + c.InitCluster(s.GetAllocator(), s.GetPersistOptions(), s.GetStorage(), s.GetBasicCluster(), s.GetKeyspaceGroupManager()) cluster, err := c.LoadClusterInfo() if err != nil { return err @@ -262,6 +267,13 @@ func (c *RaftCluster) Start(s Server) error { return nil } + if s.IsAPIServiceMode() { + err = c.keyspaceGroupManager.Bootstrap() + if err != nil { + return err + } + } + c.ruleManager = placement.NewRuleManager(c.storage, c, c.GetOpts()) if c.opt.IsPlacementRulesEnabled() { err = c.ruleManager.Initialize(c.opt.GetMaxReplicas(), c.opt.GetLocationLabels()) @@ -269,7 +281,6 @@ func (c *RaftCluster) Start(s Server) error { return err } } - c.regionLabeler, err = labeler.NewRegionLabeler(c.ctx, c.storage, regionLabelGCInterval) if err != nil { return err @@ -300,7 +311,7 @@ func (c *RaftCluster) Start(s Server) error { go c.runUpdateStoreStats() go c.startGCTuner() - c.running.Store(true) + c.running = true return nil } @@ -571,26 +582,31 @@ func (c *RaftCluster) runReplicationMode() { // Stop stops the cluster. func (c *RaftCluster) Stop() { c.Lock() - if !c.running.CompareAndSwap(true, false) { + if !c.running { c.Unlock() return } - + c.running = false c.coordinator.stop() c.cancel() c.Unlock() + c.wg.Wait() log.Info("raftcluster is stopped") } // IsRunning return if the cluster is running. func (c *RaftCluster) IsRunning() bool { - return c.running.Load() + c.RLock() + defer c.RUnlock() + return c.running } // Context returns the context of RaftCluster. func (c *RaftCluster) Context() context.Context { - if c.running.Load() { + c.RLock() + defer c.RUnlock() + if c.running { return c.ctx } return nil diff --git a/server/cluster/cluster_test.go b/server/cluster/cluster_test.go index c82ddf99b40..8caf9c5c42d 100644 --- a/server/cluster/cluster_test.go +++ b/server/cluster/cluster_test.go @@ -1839,7 +1839,7 @@ func newTestRaftCluster( basicCluster *core.BasicCluster, ) *RaftCluster { rc := &RaftCluster{serverCtx: ctx} - rc.InitCluster(id, opt, s, basicCluster) + rc.InitCluster(id, opt, s, basicCluster, nil) rc.ruleManager = placement.NewRuleManager(storage.NewStorageWithMemoryBackend(), rc, opt) if opt.IsPlacementRulesEnabled() { err := rc.ruleManager.Initialize(opt.GetMaxReplicas(), opt.GetLocationLabels()) diff --git a/server/cluster/coordinator_test.go b/server/cluster/coordinator_test.go index 5985f0dc0ef..3d66f1f199b 100644 --- a/server/cluster/coordinator_test.go +++ b/server/cluster/coordinator_test.go @@ -32,6 +32,7 @@ import ( "github.com/tikv/pd/pkg/core/storelimit" "github.com/tikv/pd/pkg/mock/mockhbstream" "github.com/tikv/pd/pkg/storage" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/pkg/utils/typeutil" "github.com/tikv/pd/server/config" @@ -201,10 +202,10 @@ func TestDispatch(t *testing.T) { // Wait for schedule and turn off balance. waitOperator(re, co, 1) - testutil.CheckTransferPeer(re, co.opController.GetOperator(1), operator.OpKind(0), 4, 1) + operatorutil.CheckTransferPeer(re, co.opController.GetOperator(1), operator.OpKind(0), 4, 1) re.NoError(co.removeScheduler(schedulers.BalanceRegionName)) waitOperator(re, co, 2) - testutil.CheckTransferLeader(re, co.opController.GetOperator(2), operator.OpKind(0), 4, 2) + operatorutil.CheckTransferLeader(re, co.opController.GetOperator(2), operator.OpKind(0), 4, 2) re.NoError(co.removeScheduler(schedulers.BalanceLeaderName)) stream := mockhbstream.NewHeartbeatStream() @@ -358,7 +359,7 @@ func TestCheckRegion(t *testing.T) { re.NoError(tc.addRegionStore(1, 1)) re.NoError(tc.addLeaderRegion(1, 2, 3)) checkRegionAndOperator(re, tc, co, 1, 1) - testutil.CheckAddPeer(re, co.opController.GetOperator(1), operator.OpReplica, 1) + operatorutil.CheckAddPeer(re, co.opController.GetOperator(1), operator.OpReplica, 1) checkRegionAndOperator(re, tc, co, 1, 0) r := tc.GetRegion(1) @@ -582,7 +583,7 @@ func TestPeerState(t *testing.T) { // Wait for schedule. waitOperator(re, co, 1) - testutil.CheckTransferPeer(re, co.opController.GetOperator(1), operator.OpKind(0), 4, 1) + operatorutil.CheckTransferPeer(re, co.opController.GetOperator(1), operator.OpKind(0), 4, 1) region := tc.GetRegion(1).Clone() diff --git a/server/config/config.go b/server/config/config.go index 2fc9fec0a2e..b33b84c3d43 100644 --- a/server/config/config.go +++ b/server/config/config.go @@ -1235,6 +1235,11 @@ func (c LabelPropertyConfig) Clone() LabelPropertyConfig { return m } +// GetLeaderLease returns the leader lease. +func (c *Config) GetLeaderLease() int64 { + return c.LeaderLease +} + // IsLocalTSOEnabled returns if the local TSO is enabled. func (c *Config) IsLocalTSOEnabled() bool { return c.EnableLocalTSO @@ -1421,7 +1426,7 @@ func (c *KeyspaceConfig) Validate() error { minCheckRegionSplitInterval, maxCheckRegionSplitInterval)) } if c.CheckRegionSplitInterval.Duration >= c.WaitRegionSplitTimeout.Duration { - return errors.New(fmt.Sprintf("[keyspace] check-region-split-interval should be less than wait-region-split-timeout")) + return errors.New("[keyspace] check-region-split-interval should be less than wait-region-split-timeout") } return nil } @@ -1445,3 +1450,23 @@ func (c *KeyspaceConfig) Clone() *KeyspaceConfig { cfg.PreAlloc = preAlloc return &cfg } + +// GetPreAlloc returns the keyspace to be allocated during keyspace manager initialization. +func (c *KeyspaceConfig) GetPreAlloc() []string { + return c.PreAlloc +} + +// ToWaitRegionSplit returns whether to wait for the region split to complete. +func (c *KeyspaceConfig) ToWaitRegionSplit() bool { + return c.WaitRegionSplit +} + +// GetWaitRegionSplitTimeout returns the max duration to wait region split. +func (c *KeyspaceConfig) GetWaitRegionSplitTimeout() time.Duration { + return c.WaitRegionSplitTimeout.Duration +} + +// GetCheckRegionSplitInterval returns the interval to check whether the region split is complete. +func (c *KeyspaceConfig) GetCheckRegionSplitInterval() time.Duration { + return c.CheckRegionSplitInterval.Duration +} diff --git a/server/grpc_service.go b/server/grpc_service.go index 59b60775f5b..6dfe06e69d5 100644 --- a/server/grpc_service.go +++ b/server/grpc_service.go @@ -51,10 +51,9 @@ import ( const ( heartbeatSendTimeout = 5 * time.Second - // tso - maxMergeTSORequests = 10000 - defaultTSOProxyTimeout = 3 * time.Second - maxRetryTimesGetGlobalTSOFromTSOServer = 3 + defaultTSOProxyTimeout = 3 * time.Second + maxRetryTimesRequestTSOServer = 3 + retryIntervalRequestTSOServer = 500 * time.Millisecond ) // gRPC errors @@ -93,13 +92,6 @@ func (s *GrpcServer) unaryMiddleware(ctx context.Context, header *pdpb.RequestHe return nil, nil } -func (s *GrpcServer) wrapErrorToHeader(errorType pdpb.ErrorType, message string) *pdpb.ResponseHeader { - return s.errorHeader(&pdpb.Error{ - Type: errorType, - Message: message, - }) -} - // GetClusterInfo implements gRPC PDServer. func (s *GrpcServer) GetClusterInfo(ctx context.Context, _ *pdpb.GetClusterInfoRequest) (*pdpb.GetClusterInfoResponse, error) { // Here we purposely do not check the cluster ID because the client does not know the correct cluster ID @@ -115,9 +107,11 @@ func (s *GrpcServer) GetClusterInfo(ctx context.Context, _ *pdpb.GetClusterInfoR }, nil } + var tsoServiceAddrs []string svcModes := make([]pdpb.ServiceMode, 0) if s.IsAPIServiceMode() { svcModes = append(svcModes, pdpb.ServiceMode_API_SVC_MODE) + tsoServiceAddrs = s.keyspaceGroupManager.GetTSOServiceAddrs() } else { svcModes = append(svcModes, pdpb.ServiceMode_PD_SVC_MODE) } @@ -125,6 +119,7 @@ func (s *GrpcServer) GetClusterInfo(ctx context.Context, _ *pdpb.GetClusterInfoR return &pdpb.GetClusterInfoResponse{ Header: s.header(), ServiceModes: svcModes, + TsoUrls: tsoServiceAddrs, }, nil } @@ -201,39 +196,29 @@ func (s *GrpcServer) Tso(stream pdpb.PD_TsoServer) error { return errors.WithStack(err) } - streamCtx := stream.Context() - if s.IsAPIServiceMode() { - forwardedHost, ok := s.GetServicePrimaryAddr(ctx, utils.TSOServiceName) - if !ok || forwardedHost == "" { - return ErrNotFoundTSOAddr - } + if forwardedHost, err := s.getForwardedHost(ctx, stream.Context()); err != nil { + return err + } else if len(forwardedHost) > 0 { + clientConn, err := s.getDelegateClient(s.ctx, forwardedHost) if err != nil { return errors.WithStack(err) } + if errCh == nil { doneCh = make(chan struct{}) defer close(doneCh) errCh = make(chan error) } - s.dispatchTSORequest(ctx, &tsoRequest{ - forwardedHost, - request, - stream, - }, forwardedHost, doneCh, errCh, true) - continue - } - forwardedHost := grpcutil.GetForwardedHost(streamCtx) - if !s.isLocalRequest(forwardedHost) { - if errCh == nil { - doneCh = make(chan struct{}) - defer close(doneCh) - errCh = make(chan error) + + var tsoProtoFactory tsoutil.ProtoFactory + if s.IsAPIServiceMode() { + tsoProtoFactory = s.tsoProtoFactory + } else { + tsoProtoFactory = s.pdProtoFactory } - s.dispatchTSORequest(ctx, &tsoRequest{ - forwardedHost, - request, - stream, - }, forwardedHost, doneCh, errCh, false) + + tsoRequest := tsoutil.NewPDProtoRequest(forwardedHost, clientConn, request, stream) + s.tsoDispatcher.DispatchRequest(ctx, tsoRequest, tsoProtoFactory, doneCh, errCh, s.tsoPrimaryWatcher) continue } @@ -246,7 +231,7 @@ func (s *GrpcServer) Tso(stream pdpb.PD_TsoServer) error { return status.Errorf(codes.FailedPrecondition, "mismatch cluster id, need %d but got %d", s.clusterID, request.GetHeader().GetClusterId()) } count := request.GetCount() - ts, err := s.tsoAllocatorManager.HandleTSORequest(request.GetDcLocation(), count) + ts, err := s.tsoAllocatorManager.HandleRequest(request.GetDcLocation(), count) if err != nil { return status.Errorf(codes.Unknown, err.Error()) } @@ -262,222 +247,17 @@ func (s *GrpcServer) Tso(stream pdpb.PD_TsoServer) error { } } -type tsoRequest struct { - forwardedHost string - request *pdpb.TsoRequest - stream pdpb.PD_TsoServer -} - -func (s *GrpcServer) dispatchTSORequest(ctx context.Context, request *tsoRequest, forwardedHost string, doneCh <-chan struct{}, errCh chan<- error, withTSOProto bool) { - tsoRequestChInterface, loaded := s.tsoDispatcher.LoadOrStore(forwardedHost, make(chan *tsoRequest, maxMergeTSORequests)) - if !loaded { - tsDeadlineCh := make(chan deadline, 1) - go s.handleDispatcher(ctx, forwardedHost, tsoRequestChInterface.(chan *tsoRequest), tsDeadlineCh, doneCh, errCh, withTSOProto) - go watchTSDeadline(ctx, tsDeadlineCh) - } - tsoRequestChInterface.(chan *tsoRequest) <- request -} - -func (s *GrpcServer) handleDispatcher(ctx context.Context, forwardedHost string, tsoRequestCh <-chan *tsoRequest, tsDeadlineCh chan<- deadline, doneCh <-chan struct{}, errCh chan<- error, withTSOProto bool) { - dispatcherCtx, ctxCancel := context.WithCancel(ctx) - defer ctxCancel() - defer s.tsoDispatcher.Delete(forwardedHost) - - var ( - forwardStream pdpb.PD_TsoClient - forwardMCSStream tsopb.TSO_TsoClient - cancel context.CancelFunc - ) - client, err := s.getDelegateClient(ctx, forwardedHost) - if err != nil { - goto errHandling - } - log.Info("create tso forward stream", zap.String("forwarded-host", forwardedHost)) - if withTSOProto { - forwardMCSStream, cancel, err = s.createMCSTSOForwardStream(client) - } else { - forwardStream, cancel, err = s.createTsoForwardStream(client) - } -errHandling: - if err != nil || (forwardStream == nil && !withTSOProto) || (forwardMCSStream == nil && withTSOProto) { - log.Error("create tso forwarding stream error", zap.String("forwarded-host", forwardedHost), errs.ZapError(errs.ErrGRPCCreateStream, err)) - select { - case <-dispatcherCtx.Done(): - return - case _, ok := <-doneCh: - if !ok { - return - } - case errCh <- err: - close(errCh) - return - } - } - defer cancel() - - requests := make([]*tsoRequest, maxMergeTSORequests+1) - for { - select { - case first := <-tsoRequestCh: - pendingTSOReqCount := len(tsoRequestCh) + 1 - requests[0] = first - for i := 1; i < pendingTSOReqCount; i++ { - requests[i] = <-tsoRequestCh - } - done := make(chan struct{}) - dl := deadline{ - timer: time.After(defaultTSOProxyTimeout), - done: done, - cancel: cancel, - } - select { - case tsDeadlineCh <- dl: - case <-dispatcherCtx.Done(): - return - } - err = s.processTSORequests(forwardStream, forwardMCSStream, requests[:pendingTSOReqCount]) - close(done) - if err != nil { - log.Error("proxy forward tso error", zap.String("forwarded-host", forwardedHost), errs.ZapError(errs.ErrGRPCSend, err)) - if strings.Contains(err.Error(), errs.NotLeaderErr) || strings.Contains(err.Error(), errs.MismatchLeaderErr) { - select { - case s.updateServicePrimaryAddrCh <- struct{}{}: - log.Info("update service primary address") - default: - } - } - select { - case <-dispatcherCtx.Done(): - return - case _, ok := <-doneCh: - if !ok { - return - } - case errCh <- err: - close(errCh) - return - } - } - case <-dispatcherCtx.Done(): - return - } - } -} - -type tsoResp interface { - GetTimestamp() *pdpb.Timestamp -} - -func (s *GrpcServer) processTSORequests(forwardStream pdpb.PD_TsoClient, forwardMCSStream tsopb.TSO_TsoClient, requests []*tsoRequest) error { - start := time.Now() - // Merge the requests - count := uint32(0) - for _, request := range requests { - count += request.request.GetCount() - } - var ( - resp tsoResp - err error - ) - if forwardStream != nil { - req := &pdpb.TsoRequest{ - Header: requests[0].request.GetHeader(), - Count: count, - // TODO: support Local TSO proxy forwarding. - DcLocation: requests[0].request.GetDcLocation(), - } - // Send to the tso server stream - if err := forwardStream.Send(req); err != nil { - return err - } - resp, err = forwardStream.Recv() - if err != nil { - return err - } - } - if forwardMCSStream != nil { - req := &tsopb.TsoRequest{ - Header: &tsopb.RequestHeader{ - ClusterId: requests[0].request.GetHeader().GetClusterId(), - KeyspaceId: utils.DefaultKeyspaceID, - KeyspaceGroupId: utils.DefaultKeySpaceGroupID, - }, - Count: count, - // TODO: support Local TSO proxy forwarding. - DcLocation: requests[0].request.GetDcLocation(), - } - // Send to the tso server stream. - if err := forwardMCSStream.Send(req); err != nil { - return err - } - resp, err = forwardMCSStream.Recv() - if err != nil { - return err - } - } - tsoProxyHandleDuration.Observe(time.Since(start).Seconds()) - tsoProxyBatchSize.Observe(float64(count)) - // Split the response - physical, logical, suffixBits := resp.GetTimestamp().GetPhysical(), resp.GetTimestamp().GetLogical(), resp.GetTimestamp().GetSuffixBits() - // `logical` is the largest ts's logical part here, we need to do the subtracting before we finish each TSO request. - // This is different from the logic of client batch, for example, if we have a largest ts whose logical part is 10, - // count is 5, then the splitting results should be 5 and 10. - firstLogical := addLogical(logical, -int64(count), suffixBits) - return s.finishTSORequest(requests, physical, firstLogical, suffixBits) -} - -// Because of the suffix, we need to shift the count before we add it to the logical part. -func addLogical(logical, count int64, suffixBits uint32) int64 { - return logical + count< 0 { - if err = stream.Send(&keyspacepb.WatchKeyspacesResponse{Header: s.header(), Keyspaces: keyspaces}); err != nil { - return err - } - } + keyspaces := make([]*keyspacepb.KeyspaceMeta, 0) + putFn := func(kv *mvccpb.KeyValue) error { + meta := &keyspacepb.KeyspaceMeta{} + if err := proto.Unmarshal(kv.Value, meta); err != nil { + defer cancel() // cancel context to stop watcher + return err } + keyspaces = append(keyspaces, meta) + return nil } -} - -func (s *KeyspaceServer) sendAllKeyspaceMeta(ctx context.Context, stream keyspacepb.Keyspace_WatchKeyspacesServer) error { - getResp, err := s.client.Get(ctx, path.Join(s.rootPath, endpoint.KeyspaceMetaPrefix()), clientv3.WithPrefix()) - if err != nil { - return err + deleteFn := func(kv *mvccpb.KeyValue) error { + return nil } - metas := make([]*keyspacepb.KeyspaceMeta, getResp.Count) - for i, kv := range getResp.Kvs { - meta := &keyspacepb.KeyspaceMeta{} - if err = proto.Unmarshal(kv.Value, meta); err != nil { + postEventFn := func() error { + defer func() { + keyspaces = keyspaces[:0] + }() + err := stream.Send(&keyspacepb.WatchKeyspacesResponse{ + Header: s.header(), + Keyspaces: keyspaces}) + if err != nil { + defer cancel() // cancel context to stop watcher return err } - metas[i] = meta + return nil } - return stream.Send(&keyspacepb.WatchKeyspacesResponse{Header: s.header(), Keyspaces: metas}) + + watcher := etcdutil.NewLoopWatcher( + ctx, + &s.serverLoopWg, + s.client, + "keyspace-server-watcher", + startKey, + putFn, + deleteFn, + postEventFn, + clientv3.WithPrefix(), + ) + s.serverLoopWg.Add(1) + go watcher.StartWatchLoop() + if err := watcher.WaitLoad(); err != nil { + cancel() // cancel context to stop watcher + return err + } + + <-ctx.Done() // wait for context done + return nil } // UpdateKeyspaceState updates the state of keyspace specified in the request. @@ -133,10 +130,7 @@ func (s *KeyspaceServer) UpdateKeyspaceState(_ context.Context, request *keyspac if err := s.validateRequest(request.GetHeader()); err != nil { return nil, err } - rc := s.GetRaftCluster() - if rc == nil { - return &keyspacepb.UpdateKeyspaceStateResponse{Header: s.notBootstrappedHeader()}, nil - } + manager := s.GetKeyspaceManager() meta, err := manager.UpdateKeyspaceStateByID(request.GetId(), request.GetState(), time.Now().Unix()) if err != nil { diff --git a/server/schedule/checker/merge_checker_test.go b/server/schedule/checker/merge_checker_test.go index df3a61477d9..3d5030b36f9 100644 --- a/server/schedule/checker/merge_checker_test.go +++ b/server/schedule/checker/merge_checker_test.go @@ -25,6 +25,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/core/storelimit" "github.com/tikv/pd/pkg/mock/mockcluster" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/pkg/versioninfo" "github.com/tikv/pd/server/config" @@ -253,7 +254,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { // partial store overlap not including leader ops := suite.mc.Check(suite.regions[2]) suite.NotNil(ops) - testutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ operator.AddLearner{ToStore: 1}, operator.PromoteLearner{ToStore: 1}, operator.RemovePeer{FromStore: 2}, @@ -267,7 +268,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { IsPassive: false, }, }) - testutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ operator.MergeRegion{ FromRegion: suite.regions[2].GetMeta(), ToRegion: suite.regions[1].GetMeta(), @@ -287,7 +288,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { suite.regions[2] = newRegion suite.cluster.PutRegion(suite.regions[2]) ops = suite.mc.Check(suite.regions[2]) - testutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ operator.AddLearner{ToStore: 4}, operator.PromoteLearner{ToStore: 4}, operator.RemovePeer{FromStore: 6}, @@ -297,7 +298,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { IsPassive: false, }, }) - testutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ operator.MergeRegion{ FromRegion: suite.regions[2].GetMeta(), ToRegion: suite.regions[1].GetMeta(), @@ -313,14 +314,14 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { })) suite.cluster.PutRegion(suite.regions[2]) ops = suite.mc.Check(suite.regions[2]) - testutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ operator.MergeRegion{ FromRegion: suite.regions[2].GetMeta(), ToRegion: suite.regions[1].GetMeta(), IsPassive: false, }, }) - testutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ operator.MergeRegion{ FromRegion: suite.regions[2].GetMeta(), ToRegion: suite.regions[1].GetMeta(), @@ -336,7 +337,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { }), core.WithLeader(&metapb.Peer{Id: 109, StoreId: 2})) suite.cluster.PutRegion(suite.regions[2]) ops = suite.mc.Check(suite.regions[2]) - testutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ operator.AddLearner{ToStore: 1}, operator.PromoteLearner{ToStore: 1}, operator.RemovePeer{FromStore: 3}, @@ -353,7 +354,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { IsPassive: false, }, }) - testutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ operator.MergeRegion{ FromRegion: suite.regions[2].GetMeta(), ToRegion: suite.regions[1].GetMeta(), @@ -372,7 +373,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { ) suite.cluster.PutRegion(suite.regions[1]) ops = suite.mc.Check(suite.regions[2]) - testutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ operator.AddLearner{ToStore: 1}, operator.PromoteLearner{ToStore: 1}, operator.RemovePeer{FromStore: 3}, @@ -392,7 +393,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { IsPassive: false, }, }) - testutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ operator.MergeRegion{ FromRegion: suite.regions[2].GetMeta(), ToRegion: suite.regions[1].GetMeta(), @@ -419,7 +420,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { ) suite.cluster.PutRegion(suite.regions[1]) ops = suite.mc.Check(suite.regions[2]) - testutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[0], []operator.OpStep{ operator.AddLearner{ToStore: 1}, operator.PromoteLearner{ToStore: 1}, operator.RemovePeer{FromStore: 3}, @@ -433,7 +434,7 @@ func (suite *mergeCheckerTestSuite) TestMatchPeers() { IsPassive: false, }, }) - testutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ + operatorutil.CheckSteps(suite.Require(), ops[1], []operator.OpStep{ operator.MergeRegion{ FromRegion: suite.regions[2].GetMeta(), ToRegion: suite.regions[1].GetMeta(), diff --git a/server/schedule/checker/replica_checker_test.go b/server/schedule/checker/replica_checker_test.go index 5e8157ac0c7..a6083c71759 100644 --- a/server/schedule/checker/replica_checker_test.go +++ b/server/schedule/checker/replica_checker_test.go @@ -26,7 +26,7 @@ import ( "github.com/tikv/pd/pkg/cache" "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mock/mockcluster" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/versioninfo" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule/operator" @@ -220,7 +220,7 @@ func (suite *replicaCheckerTestSuite) TestBasic() { // Region has 2 peers, we need to add a new peer. region := tc.GetRegion(1) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) // Disable make up replica feature. tc.SetEnableMakeUpReplica(false) @@ -230,17 +230,17 @@ func (suite *replicaCheckerTestSuite) TestBasic() { // Test healthFilter. // If store 4 is down, we add to store 3. tc.SetStoreDown(4) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) tc.SetStoreUp(4) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) // Test snapshotCountFilter. // If snapshotCount > MaxSnapshotCount, we add to store 3. tc.UpdateSnapshotCount(4, 3) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) // If snapshotCount < MaxSnapshotCount, we can add peer again. tc.UpdateSnapshotCount(4, 1) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) // Add peer in store 4, and we have enough replicas. peer4, _ := tc.AllocPeer(4) @@ -250,7 +250,7 @@ func (suite *replicaCheckerTestSuite) TestBasic() { // Add peer in store 3, and we have redundant replicas. peer3, _ := tc.AllocPeer(3) region = region.Clone(core.WithAddPeer(peer3)) - testutil.CheckRemovePeer(suite.Require(), rc.Check(region), 1) + operatorutil.CheckRemovePeer(suite.Require(), rc.Check(region), 1) // Disable remove extra replica feature. tc.SetEnableRemoveExtraReplica(false) @@ -267,13 +267,13 @@ func (suite *replicaCheckerTestSuite) TestBasic() { } region = region.Clone(core.WithDownPeers(append(region.GetDownPeers(), downPeer))) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2, 1) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2, 1) region = region.Clone(core.WithDownPeers(nil)) suite.Nil(rc.Check(region)) // Peer in store 3 is offline, transfer peer to store 1. tc.SetStoreOffline(3) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 1) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 1) } func (suite *replicaCheckerTestSuite) TestLostStore() { @@ -313,36 +313,36 @@ func (suite *replicaCheckerTestSuite) TestOffline() { region := tc.GetRegion(1) // Store 2 has different zone and smallest region score. - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2) peer2, _ := tc.AllocPeer(2) region = region.Clone(core.WithAddPeer(peer2)) // Store 3 has different zone and smallest region score. - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) peer3, _ := tc.AllocPeer(3) region = region.Clone(core.WithAddPeer(peer3)) // Store 4 has the same zone with store 3 and larger region score. peer4, _ := tc.AllocPeer(4) region = region.Clone(core.WithAddPeer(peer4)) - testutil.CheckRemovePeer(suite.Require(), rc.Check(region), 4) + operatorutil.CheckRemovePeer(suite.Require(), rc.Check(region), 4) // Test offline // the number of region peers more than the maxReplicas // remove the peer tc.SetStoreOffline(3) - testutil.CheckRemovePeer(suite.Require(), rc.Check(region), 3) + operatorutil.CheckRemovePeer(suite.Require(), rc.Check(region), 3) region = region.Clone(core.WithRemoveStorePeer(4)) // the number of region peers equals the maxReplicas // Transfer peer to store 4. - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 4) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 4) // Store 5 has a same label score with store 4, but the region score smaller than store 4, we will choose store 5. tc.AddLabelsStore(5, 3, map[string]string{"zone": "z4", "rack": "r1", "host": "h1"}) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 5) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 5) // Store 5 has too many snapshots, choose store 4 tc.UpdateSnapshotCount(5, 100) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 4) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3, 4) tc.UpdatePendingPeerCount(4, 100) suite.Nil(rc.Check(region)) } @@ -362,49 +362,49 @@ func (suite *replicaCheckerTestSuite) TestDistinctScore() { // We need 3 replicas. tc.AddLeaderRegion(1, 1) region := tc.GetRegion(1) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2) peer2, _ := tc.AllocPeer(2) region = region.Clone(core.WithAddPeer(peer2)) // Store 1,2,3 have the same zone, rack, and host. tc.AddLabelsStore(3, 5, map[string]string{"zone": "z1", "rack": "r1", "host": "h1"}) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 3) // Store 4 has smaller region score. tc.AddLabelsStore(4, 4, map[string]string{"zone": "z1", "rack": "r1", "host": "h1"}) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) // Store 5 has a different host. tc.AddLabelsStore(5, 5, map[string]string{"zone": "z1", "rack": "r1", "host": "h2"}) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 5) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 5) // Store 6 has a different rack. tc.AddLabelsStore(6, 6, map[string]string{"zone": "z1", "rack": "r2", "host": "h1"}) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 6) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 6) // Store 7 has a different zone. tc.AddLabelsStore(7, 7, map[string]string{"zone": "z2", "rack": "r1", "host": "h1"}) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 7) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 7) // Test stateFilter. tc.SetStoreOffline(7) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 6) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 6) tc.SetStoreUp(7) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 7) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 7) // Add peer to store 7. peer7, _ := tc.AllocPeer(7) region = region.Clone(core.WithAddPeer(peer7)) // Replace peer in store 1 with store 6 because it has a different rack. - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 1, 6) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 1, 6) // Disable locationReplacement feature. tc.SetEnableLocationReplacement(false) suite.Nil(rc.Check(region)) tc.SetEnableLocationReplacement(true) peer6, _ := tc.AllocPeer(6) region = region.Clone(core.WithAddPeer(peer6)) - testutil.CheckRemovePeer(suite.Require(), rc.Check(region), 1) + operatorutil.CheckRemovePeer(suite.Require(), rc.Check(region), 1) region = region.Clone(core.WithRemoveStorePeer(1), core.WithLeader(region.GetStorePeer(2))) suite.Nil(rc.Check(region)) @@ -418,10 +418,10 @@ func (suite *replicaCheckerTestSuite) TestDistinctScore() { // Store 2 and 6 have the same distinct score, but store 2 has larger region score. // So replace peer in store 2 with store 10. tc.AddLabelsStore(10, 1, map[string]string{"zone": "z3", "rack": "r1", "host": "h1"}) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2, 10) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2, 10) peer10, _ := tc.AllocPeer(10) region = region.Clone(core.WithAddPeer(peer10)) - testutil.CheckRemovePeer(suite.Require(), rc.Check(region), 2) + operatorutil.CheckRemovePeer(suite.Require(), rc.Check(region), 2) region = region.Clone(core.WithRemoveStorePeer(2)) suite.Nil(rc.Check(region)) } @@ -445,11 +445,11 @@ func (suite *replicaCheckerTestSuite) TestDistinctScore2() { tc.AddLeaderRegion(1, 1, 2, 4) region := tc.GetRegion(1) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 6) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 6) peer6, _ := tc.AllocPeer(6) region = region.Clone(core.WithAddPeer(peer6)) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 5) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 5) peer5, _ := tc.AllocPeer(5) region = region.Clone(core.WithAddPeer(peer5)) @@ -477,7 +477,7 @@ func (suite *replicaCheckerTestSuite) TestStorageThreshold() { // Move peer to better location. tc.UpdateStorageRatio(4, 0, 1) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 1, 4) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 1, 4) // If store4 is almost full, do not add peer on it. tc.UpdateStorageRatio(4, 0.9, 0.1) suite.Nil(rc.Check(region)) @@ -486,10 +486,10 @@ func (suite *replicaCheckerTestSuite) TestStorageThreshold() { region = tc.GetRegion(2) // Add peer on store4. tc.UpdateStorageRatio(4, 0, 1) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 4) // If store4 is almost full, do not add peer on it. tc.UpdateStorageRatio(4, 0.8, 0) - testutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2) + operatorutil.CheckAddPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2) } func (suite *replicaCheckerTestSuite) TestOpts() { @@ -515,9 +515,9 @@ func (suite *replicaCheckerTestSuite) TestOpts() { })) tc.SetStoreOffline(2) // RemoveDownReplica has higher priority than replaceOfflineReplica. - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 1, 4) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 1, 4) tc.SetEnableRemoveDownReplica(false) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2, 4) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpReplica, 2, 4) tc.SetEnableReplaceOfflineReplica(false) suite.Nil(rc.Check(region)) } @@ -544,10 +544,10 @@ func (suite *replicaCheckerTestSuite) TestFixDownPeer() { region = region.Clone(core.WithDownPeers([]*pdpb.PeerStats{ {Peer: region.GetStorePeer(4), DownSeconds: 6000}, })) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 5) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 5) tc.SetStoreDown(5) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 2) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 2) tc.SetIsolationLevel("zone") suite.Nil(rc.Check(region)) @@ -572,10 +572,10 @@ func (suite *replicaCheckerTestSuite) TestFixOfflinePeer() { suite.Nil(rc.Check(region)) tc.SetStoreOffline(4) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 5) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 5) tc.SetStoreOffline(5) - testutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 2) + operatorutil.CheckTransferPeer(suite.Require(), rc.Check(region), operator.OpRegion, 4, 2) tc.SetIsolationLevel("zone") suite.Nil(rc.Check(region)) diff --git a/server/schedule/checker/rule_checker_test.go b/server/schedule/checker/rule_checker_test.go index 52cf9be7c27..b6ea2331f8f 100644 --- a/server/schedule/checker/rule_checker_test.go +++ b/server/schedule/checker/rule_checker_test.go @@ -28,7 +28,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/errs" "github.com/tikv/pd/pkg/mock/mockcluster" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/versioninfo" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule/operator" @@ -806,10 +806,10 @@ func (suite *ruleCheckerTestSuite) TestFixDownPeer() { region = region.Clone(core.WithDownPeers([]*pdpb.PeerStats{ {Peer: region.GetStorePeer(4), DownSeconds: 6000}, })) - testutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 5) + operatorutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 5) suite.cluster.SetStoreDown(5) - testutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 2) + operatorutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 2) rule.IsolationLevel = "zone" suite.ruleManager.SetRule(rule) @@ -1021,10 +1021,10 @@ func (suite *ruleCheckerTestSuite) TestFixOfflinePeer() { suite.Nil(suite.rc.Check(region)) suite.cluster.SetStoreOffline(4) - testutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 5) + operatorutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 5) suite.cluster.SetStoreOffline(5) - testutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 2) + operatorutil.CheckTransferPeer(suite.Require(), suite.rc.Check(region), operator.OpRegion, 4, 2) rule.IsolationLevel = "zone" suite.ruleManager.SetRule(rule) diff --git a/server/schedule/operator_controller.go b/server/schedule/operator_controller.go index 650172c49af..70ca94ea5d6 100644 --- a/server/schedule/operator_controller.go +++ b/server/schedule/operator_controller.go @@ -862,8 +862,9 @@ func (oc *OperatorController) getOrCreateStoreLimit(storeID uint64, limitType st log.Error("invalid store ID", zap.Uint64("store-id", storeID)) return nil } - - limit := s.GetStoreLimit() - limit.Reset(ratePerSec, limitType) - return limit + // The other limits do not need to update by config exclude StoreRateLimit. + if limit, ok := s.GetStoreLimit().(*storelimit.StoreRateLimit); ok && limit.Rate(limitType) != ratePerSec { + oc.cluster.GetBasicCluster().ResetStoreLimit(storeID, limitType, ratePerSec) + } + return s.GetStoreLimit() } diff --git a/server/schedule/operator_controller_test.go b/server/schedule/operator_controller_test.go index 2417e697fc0..c18337793d4 100644 --- a/server/schedule/operator_controller_test.go +++ b/server/schedule/operator_controller_test.go @@ -274,7 +274,8 @@ func (suite *operatorControllerTestSuite) TestConcurrentRemoveOperator() { suite.True(op1.Start()) oc.SetOperator(op1) - suite.NoError(failpoint.Enable("github.com/tikv/pd/server/schedule/concurrentRemoveOperator", "return(true)")) + suite.NoError(failpoint.Enable("github.com/tikv/pd/pkg/schedule/concurrentRemoveOperator", "return(true)")) + defer suite.NoError(failpoint.Disable("github.com/tikv/pd/pkg/schedule/concurrentRemoveOperator")) var wg sync.WaitGroup wg.Add(2) diff --git a/server/schedulers/balance_test.go b/server/schedulers/balance_test.go index c531dfc76e1..41efb2d7541 100644 --- a/server/schedulers/balance_test.go +++ b/server/schedulers/balance_test.go @@ -28,7 +28,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/versioninfo" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule" @@ -434,16 +434,16 @@ func (suite *balanceLeaderSchedulerTestSuite) TestBalanceFilter() { suite.tc.AddLeaderStore(4, 16) suite.tc.AddLeaderRegion(1, 4, 1, 2, 3) - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 1) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 1) // Test stateFilter. // if store 4 is offline, we should consider it // because it still provides services suite.tc.SetStoreOffline(4) - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 1) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 1) // If store 1 is down, it will be filtered, // store 2 becomes the store with least leaders. suite.tc.SetStoreDown(1) - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 2) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 2) plans := suite.dryRun() suite.NotEmpty(plans) suite.Equal(0, plans[0].GetStep()) @@ -454,7 +454,7 @@ func (suite *balanceLeaderSchedulerTestSuite) TestBalanceFilter() { // If store 2 is busy, it will be filtered, // store 3 becomes the store with least leaders. suite.tc.SetStoreBusy(2, true) - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 3) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 3) // Test disconnectFilter. // If store 3 is disconnected, no operator can be created. @@ -476,9 +476,9 @@ func (suite *balanceLeaderSchedulerTestSuite) TestLeaderWeight() { suite.tc.UpdateStoreLeaderWeight(3, 1) suite.tc.UpdateStoreLeaderWeight(4, 2) suite.tc.AddLeaderRegion(1, 1, 2, 3, 4) - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 1, 4) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 1, 4) suite.tc.UpdateLeaderCount(4, 30) - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 1, 3) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 1, 3) } func (suite *balanceLeaderSchedulerTestSuite) TestBalancePolicy() { @@ -492,9 +492,9 @@ func (suite *balanceLeaderSchedulerTestSuite) TestBalancePolicy() { suite.tc.AddLeaderRegion(1, 2, 1, 3, 4) suite.tc.AddLeaderRegion(2, 1, 2, 3, 4) suite.tc.SetLeaderSchedulePolicy("count") - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 2, 3) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 2, 3) suite.tc.SetLeaderSchedulePolicy("size") - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 1, 4) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 1, 4) } func (suite *balanceLeaderSchedulerTestSuite) TestBalanceSelector() { @@ -510,7 +510,7 @@ func (suite *balanceLeaderSchedulerTestSuite) TestBalanceSelector() { suite.tc.AddLeaderRegion(2, 3, 1, 2) // store4 has max leader score, store1 has min leader score. // The scheduler try to move a leader out of 16 first. - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 2) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 2) // Stores: 1 2 3 4 // Leaders: 1 14 15 16 @@ -519,7 +519,7 @@ func (suite *balanceLeaderSchedulerTestSuite) TestBalanceSelector() { suite.tc.UpdateLeaderCount(2, 14) suite.tc.UpdateLeaderCount(3, 15) // Cannot move leader out of store4, move a leader into store1. - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 3, 1) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 3, 1) // Stores: 1 2 3 4 // Leaders: 1 2 15 16 @@ -529,7 +529,7 @@ func (suite *balanceLeaderSchedulerTestSuite) TestBalanceSelector() { suite.tc.AddLeaderRegion(1, 3, 2, 4) suite.tc.AddLeaderRegion(2, 1, 2, 3) // No leader in store16, no follower in store1. Now source and target are store3 and store2. - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 3, 2) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 3, 2) // Stores: 1 2 3 4 // Leaders: 9 10 10 11 @@ -552,7 +552,7 @@ func (suite *balanceLeaderSchedulerTestSuite) TestBalanceSelector() { suite.tc.AddLeaderStore(2, 13) suite.tc.AddLeaderStore(3, 0) suite.tc.AddLeaderStore(4, 16) - testutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 3) + operatorutil.CheckTransferLeader(suite.Require(), suite.schedule()[0], operator.OpKind(0), 4, 3) } type balanceLeaderRangeSchedulerTestSuite struct { @@ -780,7 +780,7 @@ func TestBalanceRegionSchedule1(t *testing.T) { tc.AddLeaderRegion(1, 4) ops, _ := sb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpKind(0), 4, 1) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpKind(0), 4, 1) // Test stateFilter. tc.SetStoreOffline(1) @@ -790,7 +790,7 @@ func TestBalanceRegionSchedule1(t *testing.T) { // store 2 becomes the store with least regions. ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpKind(0), 4, 2) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpKind(0), 4, 2) tc.SetStoreUp(1) // test region replicate not match opt.SetMaxReplicas(3) @@ -844,37 +844,37 @@ func checkReplica3(re *require.Assertions, tc *mockcluster.Cluster, sb schedule. tc.AddLabelsStore(4, 2, map[string]string{"zone": "z1", "rack": "r2", "host": "h1"}) ops, _ = sb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 2, 4) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 2, 4) // Store 5 has smaller region score than store 1. tc.AddLabelsStore(5, 2, map[string]string{"zone": "z1", "rack": "r1", "host": "h1"}) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 5) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 5) // Store 6 has smaller region score than store 5. tc.AddLabelsStore(6, 1, map[string]string{"zone": "z1", "rack": "r1", "host": "h1"}) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 6) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 6) // Store 7 has smaller region score with store 6. tc.AddLabelsStore(7, 0, map[string]string{"zone": "z1", "rack": "r1", "host": "h2"}) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 7) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 7) // If store 7 is not available, will choose store 6. tc.SetStoreDown(7) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 6) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 6) // Store 8 has smaller region score than store 7, but the distinct score decrease. tc.AddLabelsStore(8, 1, map[string]string{"zone": "z1", "rack": "r2", "host": "h3"}) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 6) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 6) // Take down 4,5,6,7 tc.SetStoreDown(4) @@ -925,19 +925,19 @@ func checkReplica5(re *require.Assertions, tc *mockcluster.Cluster, sb schedule. tc.AddLabelsStore(6, 1, map[string]string{"zone": "z5", "rack": "r2", "host": "h1"}) ops, _ := sb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 5, 6) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 5, 6) // Store 7 has larger region score and same distinct score with store 6. tc.AddLabelsStore(7, 5, map[string]string{"zone": "z6", "rack": "r1", "host": "h1"}) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 5, 6) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 5, 6) // Store 1 has smaller region score and higher distinct score. tc.AddLeaderRegion(1, 2, 3, 4, 5, 6) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 5, 1) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 5, 1) // Store 6 has smaller region score and higher distinct score. tc.AddLabelsStore(11, 29, map[string]string{"zone": "z1", "rack": "r2", "host": "h1"}) @@ -946,7 +946,7 @@ func checkReplica5(re *require.Assertions, tc *mockcluster.Cluster, sb schedule. tc.AddLeaderRegion(1, 2, 3, 11, 12, 13) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 11, 6) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 11, 6) } // TestBalanceRegionSchedule2 for corner case 1: @@ -1033,7 +1033,7 @@ func TestBalanceRegionSchedule2(t *testing.T) { // if the space of store 5 is normal, we can balance region to store 5 ops1, _ = sb.Schedule(tc, false) op = ops1[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 5) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 5) // the used size of store 5 reach (highSpace, lowSpace) origin := tc.GetStore(5) @@ -1051,7 +1051,7 @@ func TestBalanceRegionSchedule2(t *testing.T) { // Then it will try store 4. ops1, _ = sb.Schedule(tc, false) op = ops1[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 4) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 4) } func TestBalanceRegionStoreWeight(t *testing.T) { @@ -1082,12 +1082,12 @@ func TestBalanceRegionStoreWeight(t *testing.T) { tc.AddLeaderRegion(1, 1) ops, _ := sb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 4) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 4) tc.UpdateRegionCount(4, 30) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 3) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 3) } func TestBalanceRegionReplacePendingRegion(t *testing.T) { @@ -1144,7 +1144,7 @@ func TestBalanceRegionOpInfluence(t *testing.T) { } ops, _ := sb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpKind(0), 2, 1) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpKind(0), 2, 1) } func checkReplacePendingRegion(re *require.Assertions, tc *mockcluster.Cluster, sb schedule.Scheduler) { @@ -1168,7 +1168,7 @@ func checkReplacePendingRegion(re *require.Assertions, tc *mockcluster.Cluster, re.Equal(uint64(3), op.RegionID()) ops, _ = sb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 4) + operatorutil.CheckTransferPeer(re, op, operator.OpKind(0), 1, 4) } func TestBalanceRegionShouldNotBalance(t *testing.T) { diff --git a/server/schedulers/evict_leader_test.go b/server/schedulers/evict_leader_test.go index c2f6af4f71d..2ea8681e5bf 100644 --- a/server/schedulers/evict_leader_test.go +++ b/server/schedulers/evict_leader_test.go @@ -25,7 +25,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule" "github.com/tikv/pd/server/schedule/operator" @@ -52,7 +52,7 @@ func TestEvictLeader(t *testing.T) { re.NoError(err) re.True(sl.IsScheduleAllowed(tc)) ops, _ := sl.Schedule(tc, false) - testutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2, 3}) + operatorutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2, 3}) re.False(ops[0].Step(0).(operator.TransferLeader).IsFinish(tc.MockRegionInfo(1, 1, []uint64{2, 3}, []uint64{}, &metapb.RegionEpoch{ConfVer: 0, Version: 0}))) re.True(ops[0].Step(0).(operator.TransferLeader).IsFinish(tc.MockRegionInfo(1, 2, []uint64{1, 3}, []uint64{}, &metapb.RegionEpoch{ConfVer: 0, Version: 0}))) } @@ -83,11 +83,11 @@ func TestEvictLeaderWithUnhealthyPeer(t *testing.T) { // only pending tc.PutRegion(region.Clone(withPendingPeer)) ops, _ := sl.Schedule(tc, false) - testutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{3}) + operatorutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{3}) // only down tc.PutRegion(region.Clone(withDownPeer)) ops, _ = sl.Schedule(tc, false) - testutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2}) + operatorutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2}) // pending + down tc.PutRegion(region.Clone(withPendingPeer, withDownPeer)) ops, _ = sl.Schedule(tc, false) diff --git a/server/schedulers/evict_slow_store_test.go b/server/schedulers/evict_slow_store_test.go index fe1c1331a68..0ad83f2f5f4 100644 --- a/server/schedulers/evict_slow_store_test.go +++ b/server/schedulers/evict_slow_store_test.go @@ -25,7 +25,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule" "github.com/tikv/pd/server/schedule/operator" @@ -82,7 +82,7 @@ func (suite *evictSlowStoreTestSuite) TestEvictSlowStore() { suite.True(suite.es.IsScheduleAllowed(suite.tc)) // Add evict leader scheduler to store 1 ops, _ := suite.es.Schedule(suite.tc, false) - testutil.CheckMultiTargetTransferLeader(suite.Require(), ops[0], operator.OpLeader, 1, []uint64{2}) + operatorutil.CheckMultiTargetTransferLeader(suite.Require(), ops[0], operator.OpLeader, 1, []uint64{2}) suite.Equal(EvictSlowStoreType, ops[0].Desc()) // Cannot balance leaders to store 1 ops, _ = suite.bs.Schedule(suite.tc, false) @@ -95,7 +95,7 @@ func (suite *evictSlowStoreTestSuite) TestEvictSlowStore() { ops, _ = suite.es.Schedule(suite.tc, false) suite.Empty(ops) ops, _ = suite.bs.Schedule(suite.tc, false) - testutil.CheckTransferLeader(suite.Require(), ops[0], operator.OpLeader, 2, 1) + operatorutil.CheckTransferLeader(suite.Require(), ops[0], operator.OpLeader, 2, 1) // no slow store need to evict. ops, _ = suite.es.Schedule(suite.tc, false) diff --git a/server/schedulers/evict_slow_trend_test.go b/server/schedulers/evict_slow_trend_test.go index d51891cb231..016c2afa4e6 100644 --- a/server/schedulers/evict_slow_trend_test.go +++ b/server/schedulers/evict_slow_trend_test.go @@ -25,7 +25,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule" "github.com/tikv/pd/server/schedule/operator" @@ -117,7 +117,7 @@ func (suite *evictSlowTrendTestSuite) TestEvictSlowTrend() { suite.tc.PutStore(newStoreInfo) } ops, _ = suite.es.Schedule(suite.tc, false) - testutil.CheckMultiTargetTransferLeader(suite.Require(), ops[0], operator.OpLeader, 1, []uint64{2, 3}) + operatorutil.CheckMultiTargetTransferLeader(suite.Require(), ops[0], operator.OpLeader, 1, []uint64{2, 3}) suite.Equal(EvictSlowTrendType, ops[0].Desc()) suite.Equal(es2.conf.candidate(), uint64(0)) suite.Equal(es2.conf.evictedStore(), uint64(1)) @@ -140,7 +140,7 @@ func (suite *evictSlowTrendTestSuite) TestEvictSlowTrend() { suite.Empty(ops) suite.Zero(es2.conf.evictedStore()) ops, _ = suite.bs.Schedule(suite.tc, false) - testutil.CheckTransferLeader(suite.Require(), ops[0], operator.OpLeader, 3, 1) + operatorutil.CheckTransferLeader(suite.Require(), ops[0], operator.OpLeader, 3, 1) // no slow store need to evict. ops, _ = suite.es.Schedule(suite.tc, false) diff --git a/server/schedulers/hot_region_test.go b/server/schedulers/hot_region_test.go index 7e514efb923..b51e7bb8a98 100644 --- a/server/schedulers/hot_region_test.go +++ b/server/schedulers/hot_region_test.go @@ -28,7 +28,7 @@ import ( "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" "github.com/tikv/pd/pkg/storage/endpoint" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/utils/typeutil" "github.com/tikv/pd/pkg/versioninfo" "github.com/tikv/pd/server/config" @@ -264,15 +264,15 @@ func checkByteRateOnly(re *require.Assertions, tc *mockcluster.Cluster, hb sched switch op.Len() { case 1: // balance by leader selected - testutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) case 4: // balance by peer selected if op.RegionID() == 2 { // peer in store 1 of the region 2 can transfer to store 5 or store 6 because of the label - testutil.CheckTransferPeerWithLeaderTransferFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferPeerWithLeaderTransferFrom(re, op, operator.OpHotRegion, 1) } else { // peer in store 1 of the region 1,3 can only transfer to store 6 - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) } default: re.FailNow("wrong op: " + op.String()) @@ -292,10 +292,10 @@ func checkByteRateOnly(re *require.Assertions, tc *mockcluster.Cluster, hb sched re.Equal(4, op.Len()) if op.RegionID() == 2 { // peer in store 1 of the region 2 can transfer to store 5 or store 6 because of the label - testutil.CheckTransferPeerWithLeaderTransferFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferPeerWithLeaderTransferFrom(re, op, operator.OpHotRegion, 1) } else { // peer in store 1 of the region 1,3 can only transfer to store 6 - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) } } @@ -358,16 +358,16 @@ func checkByteRateOnly(re *require.Assertions, tc *mockcluster.Cluster, hb sched switch op.RegionID() { case 1, 2: if op.Len() == 3 { - testutil.CheckTransferPeer(re, op, operator.OpHotRegion, 3, 6) + operatorutil.CheckTransferPeer(re, op, operator.OpHotRegion, 3, 6) } else if op.Len() == 4 { - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) } else { re.FailNow("wrong operator: " + op.String()) } case 3: - testutil.CheckTransferPeer(re, op, operator.OpHotRegion, 1, 5) + operatorutil.CheckTransferPeer(re, op, operator.OpHotRegion, 1, 5) case 5: - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 3, 6) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 3, 6) default: re.FailNow("wrong operator: " + op.String()) } @@ -484,10 +484,10 @@ func TestHotWriteRegionScheduleByteRateOnlyWithTiFlash(t *testing.T) { switch op.Len() { case 1: // balance by leader selected - testutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) case 2: // balance by peer selected - testutil.CheckTransferLearner(re, op, operator.OpHotRegion, 8, 10) + operatorutil.CheckTransferLearner(re, op, operator.OpHotRegion, 8, 10) default: re.FailNow("wrong op: " + op.String()) } @@ -498,7 +498,7 @@ func TestHotWriteRegionScheduleByteRateOnlyWithTiFlash(t *testing.T) { clearPendingInfluence(hb) ops, _ := hb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) } // | store_id | write_bytes_rate | // |----------|------------------| @@ -575,15 +575,15 @@ func TestHotWriteRegionScheduleByteRateOnlyWithTiFlash(t *testing.T) { switch op.Len() { case 1: // balance by leader selected - testutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) case 4: // balance by peer selected if op.RegionID() == 2 { // peer in store 1 of the region 2 can transfer to store 5 or store 6 because of the label - testutil.CheckTransferPeerWithLeaderTransferFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferPeerWithLeaderTransferFrom(re, op, operator.OpHotRegion, 1) } else { // peer in store 1 of the region 1,3 can only transfer to store 6 - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 6) } default: re.FailNow("wrong op: " + op.String()) @@ -628,7 +628,7 @@ func TestHotWriteRegionScheduleWithQuery(t *testing.T) { clearPendingInfluence(hb.(*hotScheduler)) ops, _ := hb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferLeader(re, op, operator.OpHotRegion, 1, 3) + operatorutil.CheckTransferLeader(re, op, operator.OpHotRegion, 1, 3) } } @@ -672,21 +672,21 @@ func TestHotWriteRegionScheduleWithKeyRate(t *testing.T) { ops, _ := hb.Schedule(tc, false) op := ops[0] // byteDecRatio <= 0.95 && keyDecRatio <= 0.95 - testutil.CheckTransferPeer(re, op, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op, operator.OpHotRegion, 1, 4) // store byte rate (min, max): (10, 10.5) | 9.5 | 9.5 | (9, 9.5) | 8.9 // store key rate (min, max): (10, 10.5) | 9.5 | 9.8 | (9, 9.5) | 9.2 ops, _ = hb.Schedule(tc, false) op = ops[0] // byteDecRatio <= 0.99 && keyDecRatio <= 0.95 - testutil.CheckTransferPeer(re, op, operator.OpHotRegion, 3, 5) + operatorutil.CheckTransferPeer(re, op, operator.OpHotRegion, 3, 5) // store byte rate (min, max): (10, 10.5) | 9.5 | (9.45, 9.5) | (9, 9.5) | (8.9, 8.95) // store key rate (min, max): (10, 10.5) | 9.5 | (9.7, 9.8) | (9, 9.5) | (9.2, 9.3) // byteDecRatio <= 0.95 // op = hb.Schedule(tc, false)[0] // FIXME: cover this case - // testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 5) + // operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 5) // store byte rate (min, max): (9.5, 10.5) | 9.5 | (9.45, 9.5) | (9, 9.5) | (8.9, 9.45) // store key rate (min, max): (9.2, 10.2) | 9.5 | (9.7, 9.8) | (9, 9.5) | (9.2, 9.8) } @@ -834,7 +834,7 @@ func TestHotWriteRegionScheduleWithLeader(t *testing.T) { clearPendingInfluence(hb.(*hotScheduler)) ops, _ := hb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 2) + operatorutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 2) ops, _ = hb.Schedule(tc, false) re.Empty(ops) } @@ -915,10 +915,10 @@ func TestHotWriteRegionScheduleWithPendingInfluence(t *testing.T) { switch op.Len() { case 1: // balance by leader selected - testutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) + operatorutil.CheckTransferLeaderFrom(re, op, operator.OpHotRegion, 1) case 4: // balance by peer selected - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 4) cnt++ if cnt == 3 { re.True(op.Cancel()) @@ -1012,7 +1012,7 @@ func TestHotWriteRegionScheduleWithRuleEnabled(t *testing.T) { ops, _ := hb.Schedule(tc, false) op := ops[0] // The targetID should always be 1 as leader is only allowed to be placed in store1 or store2 by placement rule - testutil.CheckTransferLeader(re, op, operator.OpHotRegion, 2, 1) + operatorutil.CheckTransferLeader(re, op, operator.OpHotRegion, 2, 1) ops, _ = hb.Schedule(tc, false) re.Empty(ops) } @@ -1084,7 +1084,7 @@ func TestHotReadRegionScheduleByteRateOnly(t *testing.T) { // move leader from store 1 to store 5 // it is better than transfer leader from store 1 to store 3 - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 5) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 5) re.Contains(hb.regionPendings, uint64(1)) re.True(typeutil.Float64Equal(512.0*units.KiB, hb.regionPendings[1].origin.Loads[statistics.RegionReadBytes])) clearPendingInfluence(hb) @@ -1123,7 +1123,7 @@ func TestHotReadRegionScheduleByteRateOnly(t *testing.T) { // We will move leader peer of region 1 from 1 to 5 ops, _ = hb.Schedule(tc, false) op = ops[0] - testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion|operator.OpLeader, 1, 5) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion|operator.OpLeader, 1, 5) re.Contains(hb.regionPendings, uint64(1)) re.True(typeutil.Float64Equal(512.0*units.KiB, hb.regionPendings[1].origin.Loads[statistics.RegionReadBytes])) clearPendingInfluence(hb) @@ -1172,7 +1172,7 @@ func TestHotReadRegionScheduleWithQuery(t *testing.T) { clearPendingInfluence(hb.(*hotScheduler)) ops, _ := hb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferLeader(re, op, operator.OpHotRegion, 1, 3) + operatorutil.CheckTransferLeader(re, op, operator.OpHotRegion, 1, 3) } } @@ -1215,21 +1215,21 @@ func TestHotReadRegionScheduleWithKeyRate(t *testing.T) { ops, _ := hb.Schedule(tc, false) op := ops[0] // byteDecRatio <= 0.95 && keyDecRatio <= 0.95 - testutil.CheckTransferLeader(re, op, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferLeader(re, op, operator.OpHotRegion, 1, 4) // store byte rate (min, max): (10, 10.5) | 9.5 | 9.5 | (9, 9.5) | 8.9 // store key rate (min, max): (10, 10.5) | 9.5 | 9.8 | (9, 9.5) | 9.2 ops, _ = hb.Schedule(tc, false) op = ops[0] // byteDecRatio <= 0.99 && keyDecRatio <= 0.95 - testutil.CheckTransferLeader(re, op, operator.OpHotRegion, 3, 5) + operatorutil.CheckTransferLeader(re, op, operator.OpHotRegion, 3, 5) // store byte rate (min, max): (10, 10.5) | 9.5 | (9.45, 9.5) | (9, 9.5) | (8.9, 8.95) // store key rate (min, max): (10, 10.5) | 9.5 | (9.7, 9.8) | (9, 9.5) | (9.2, 9.3) // byteDecRatio <= 0.95 // FIXME: cover this case // op = hb.Schedule(tc, false)[0] - // testutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 5) + // operatorutil.CheckTransferPeerWithLeaderTransfer(re, op, operator.OpHotRegion, 1, 5) // store byte rate (min, max): (9.5, 10.5) | 9.5 | (9.45, 9.5) | (9, 9.5) | (8.9, 9.45) // store key rate (min, max): (9.2, 10.2) | 9.5 | (9.7, 9.8) | (9, 9.5) | (9.2, 9.8) } @@ -1304,7 +1304,7 @@ func TestHotReadRegionScheduleWithPendingInfluence(t *testing.T) { ops, _ := hb.Schedule(tc, false) op1 := ops[0] - testutil.CheckTransferPeer(re, op1, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op1, operator.OpHotRegion, 1, 4) // After move-peer, store byte/key rate (min, max): (6.6, 7.1) | 6.1 | 6 | (5, 5.5) pendingAmpFactor = old @@ -1314,7 +1314,7 @@ func TestHotReadRegionScheduleWithPendingInfluence(t *testing.T) { ops, _ = hb.Schedule(tc, false) op2 := ops[0] - testutil.CheckTransferPeer(re, op2, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op2, operator.OpHotRegion, 1, 4) // After move-peer, store byte/key rate (min, max): (6.1, 7.1) | 6.1 | 6 | (5, 6) ops, _ = hb.Schedule(tc, false) @@ -1327,18 +1327,18 @@ func TestHotReadRegionScheduleWithPendingInfluence(t *testing.T) { ops, _ := hb.Schedule(tc, false) op1 := ops[0] - testutil.CheckTransferPeer(re, op1, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op1, operator.OpHotRegion, 1, 4) // After move-peer, store byte/key rate (min, max): (6.6, 7.1) | 6.1 | 6 | (5, 5.5) ops, _ = hb.Schedule(tc, false) op2 := ops[0] - testutil.CheckTransferPeer(re, op2, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op2, operator.OpHotRegion, 1, 4) // After move-peer, store byte/key rate (min, max): (6.1, 7.1) | 6.1 | 6 | (5, 6) re.True(op2.Cancel()) ops, _ = hb.Schedule(tc, false) op2 = ops[0] - testutil.CheckTransferPeer(re, op2, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op2, operator.OpHotRegion, 1, 4) // After move-peer, store byte/key rate (min, max): (6.1, 7.1) | 6.1 | (6, 6.5) | (5, 5.5) re.True(op1.Cancel()) @@ -1346,7 +1346,7 @@ func TestHotReadRegionScheduleWithPendingInfluence(t *testing.T) { ops, _ = hb.Schedule(tc, false) op3 := ops[0] - testutil.CheckTransferPeer(re, op3, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op3, operator.OpHotRegion, 1, 4) // store byte/key rate (min, max): (6.1, 7.1) | 6.1 | 6 | (5, 6) ops, _ = hb.Schedule(tc, false) @@ -1389,7 +1389,7 @@ func TestHotReadWithEvictLeaderScheduler(t *testing.T) { ops, _ := hb.Schedule(tc, false) re.Len(ops, 1) clearPendingInfluence(hb.(*hotScheduler)) - testutil.CheckTransferPeerWithLeaderTransfer(re, ops[0], operator.OpHotRegion|operator.OpLeader, 1, 4) + operatorutil.CheckTransferPeerWithLeaderTransfer(re, ops[0], operator.OpHotRegion|operator.OpLeader, 1, 4) // two dim are both enough uniform among three stores tc.SetStoreEvictLeader(4, true) ops, _ = hb.Schedule(tc, false) @@ -1944,7 +1944,7 @@ func TestHotReadPeerSchedule(t *testing.T) { tc.AddRegionWithPeerReadInfo(1, 3, 1, uint64(0.9*units.KiB*float64(10)), uint64(0.9*units.KiB*float64(10)), 10, []uint64{1, 2}, 3) ops, _ := hb.Schedule(tc, false) op := ops[0] - testutil.CheckTransferPeer(re, op, operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, op, operator.OpHotRegion, 1, 4) } func TestHotScheduleWithPriority(t *testing.T) { @@ -1988,12 +1988,12 @@ func TestHotScheduleWithPriority(t *testing.T) { hb.(*hotScheduler).conf.WritePeerPriorities = []string{statistics.BytePriority, statistics.KeyPriority} ops, _ := hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 1, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 1, 5) clearPendingInfluence(hb.(*hotScheduler)) hb.(*hotScheduler).conf.WritePeerPriorities = []string{statistics.KeyPriority, statistics.BytePriority} ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 4, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 4, 5) clearPendingInfluence(hb.(*hotScheduler)) // assert read priority schedule @@ -2010,12 +2010,12 @@ func TestHotScheduleWithPriority(t *testing.T) { hb.(*hotScheduler).conf.ReadPriorities = []string{statistics.BytePriority, statistics.KeyPriority} ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) clearPendingInfluence(hb.(*hotScheduler)) hb.(*hotScheduler).conf.ReadPriorities = []string{statistics.KeyPriority, statistics.BytePriority} ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 3) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 3) hb, err = schedule.CreateScheduler(statistics.Write.String(), schedule.NewOperatorController(ctx, nil, nil), storage.NewStorageWithMemoryBackend(), nil) hb.(*hotScheduler).conf.WriteLeaderPriorities = []string{statistics.KeyPriority, statistics.BytePriority} @@ -2035,7 +2035,7 @@ func TestHotScheduleWithPriority(t *testing.T) { hb.(*hotScheduler).conf.StrictPickingStore = false ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 1, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 1, 5) clearPendingInfluence(hb.(*hotScheduler)) tc.UpdateStorageWrittenStats(1, 6*units.MiB*statistics.StoreHeartBeatReportInterval, 6*units.MiB*statistics.StoreHeartBeatReportInterval) @@ -2050,7 +2050,7 @@ func TestHotScheduleWithPriority(t *testing.T) { hb.(*hotScheduler).conf.StrictPickingStore = false ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 4, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 4, 5) clearPendingInfluence(hb.(*hotScheduler)) } @@ -2092,7 +2092,7 @@ func TestHotScheduleWithStddev(t *testing.T) { stddevThreshold = -1.0 ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) clearPendingInfluence(hb.(*hotScheduler)) // skip -1 case (uniform cluster) @@ -2111,7 +2111,7 @@ func TestHotScheduleWithStddev(t *testing.T) { stddevThreshold = -1.0 ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) clearPendingInfluence(hb.(*hotScheduler)) } @@ -2152,11 +2152,11 @@ func TestHotWriteLeaderScheduleWithPriority(t *testing.T) { hb.(*hotScheduler).conf.WriteLeaderPriorities = []string{statistics.KeyPriority, statistics.BytePriority} ops, _ := hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) hb.(*hotScheduler).conf.WriteLeaderPriorities = []string{statistics.BytePriority, statistics.KeyPriority} ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 3) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 3) } func TestCompatibility(t *testing.T) { diff --git a/server/schedulers/hot_region_v2_test.go b/server/schedulers/hot_region_v2_test.go index 9db102da228..d2debc1c5bc 100644 --- a/server/schedulers/hot_region_v2_test.go +++ b/server/schedulers/hot_region_v2_test.go @@ -22,7 +22,7 @@ import ( "github.com/stretchr/testify/require" "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/versioninfo" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule" @@ -76,8 +76,8 @@ func TestHotWriteRegionScheduleWithRevertRegionsDimSecond(t *testing.T) { ops, _ = hb.Schedule(tc, false) /* The revert region is currently disabled for the -1 case. re.Len(ops, 2) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) - testutil.CheckTransferPeer(re, ops[1], operator.OpHotRegion, 5, 2) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[1], operator.OpHotRegion, 5, 2) */ re.Empty(ops) re.True(hb.searchRevertRegions[writePeer]) @@ -88,7 +88,7 @@ func TestHotWriteRegionScheduleWithRevertRegionsDimSecond(t *testing.T) { }) ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) re.False(hb.searchRevertRegions[writePeer]) clearPendingInfluence(hb) } @@ -128,7 +128,7 @@ func TestHotWriteRegionScheduleWithRevertRegionsDimFirst(t *testing.T) { // One operator can be generated when RankFormulaVersion == "v1". ops, _ := hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) re.False(hb.searchRevertRegions[writePeer]) clearPendingInfluence(hb) @@ -141,8 +141,8 @@ func TestHotWriteRegionScheduleWithRevertRegionsDimFirst(t *testing.T) { // Two operators can be generated when RankFormulaVersion == "v2". ops, _ = hb.Schedule(tc, false) re.Len(ops, 2) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) - testutil.CheckTransferPeer(re, ops[1], operator.OpHotRegion, 5, 2) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[1], operator.OpHotRegion, 5, 2) re.True(hb.searchRevertRegions[writePeer]) clearPendingInfluence(hb) } @@ -182,7 +182,7 @@ func TestHotWriteRegionScheduleWithRevertRegionsDimFirstOnly(t *testing.T) { // One operator can be generated when RankFormulaVersion == "v1". ops, _ := hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) re.False(hb.searchRevertRegions[writePeer]) clearPendingInfluence(hb) @@ -195,7 +195,7 @@ func TestHotWriteRegionScheduleWithRevertRegionsDimFirstOnly(t *testing.T) { // There is still the solution with one operator after that. ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) re.True(hb.searchRevertRegions[writePeer]) clearPendingInfluence(hb) // Two operators can be generated when there is a better solution @@ -204,8 +204,8 @@ func TestHotWriteRegionScheduleWithRevertRegionsDimFirstOnly(t *testing.T) { }) ops, _ = hb.Schedule(tc, false) re.Len(ops, 2) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) - testutil.CheckTransferPeer(re, ops[1], operator.OpHotRegion, 5, 2) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferPeer(re, ops[1], operator.OpHotRegion, 5, 2) re.True(hb.searchRevertRegions[writePeer]) clearPendingInfluence(hb) } @@ -256,8 +256,8 @@ func TestHotReadRegionScheduleWithRevertRegionsDimSecond(t *testing.T) { ops, _ = hb.Schedule(tc, false) /* The revert region is currently disabled for the -1 case. re.Len(ops, 2) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 2, 5) - testutil.CheckTransferLeader(re, ops[1], operator.OpHotRegion, 5, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferLeader(re, ops[1], operator.OpHotRegion, 5, 2) */ re.Empty(ops) re.True(hb.searchRevertRegions[readLeader]) @@ -268,7 +268,7 @@ func TestHotReadRegionScheduleWithRevertRegionsDimSecond(t *testing.T) { }) ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 2, 5) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 2, 5) re.False(hb.searchRevertRegions[readLeader]) clearPendingInfluence(hb) } @@ -304,7 +304,7 @@ func TestSkipUniformStore(t *testing.T) { stddevThreshold = 0.0 ops, _ := hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) clearPendingInfluence(hb.(*hotScheduler)) // when there is uniform store filter, not schedule stddevThreshold = 0.1 @@ -324,13 +324,13 @@ func TestSkipUniformStore(t *testing.T) { stddevThreshold = 0.0 ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 1, 2) clearPendingInfluence(hb.(*hotScheduler)) // when there is uniform store filter, schedule the second dim, which is no uniform stddevThreshold = 0.1 ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 3, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 3, 2) clearPendingInfluence(hb.(*hotScheduler)) // Case3: the second dim is enough uniform, we should schedule the first dim, although its rank is higher than the second dim @@ -345,12 +345,12 @@ func TestSkipUniformStore(t *testing.T) { stddevThreshold = 0.0 ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 3, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 3, 2) clearPendingInfluence(hb.(*hotScheduler)) // when there is uniform store filter, schedule the first dim, which is no uniform stddevThreshold = 0.1 ops, _ = hb.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 3, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpHotRegion, 3, 2) clearPendingInfluence(hb.(*hotScheduler)) } diff --git a/server/schedulers/scheduler_test.go b/server/schedulers/scheduler_test.go index 2464df420fa..6a83587db41 100644 --- a/server/schedulers/scheduler_test.go +++ b/server/schedulers/scheduler_test.go @@ -24,7 +24,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/pkg/versioninfo" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule" @@ -89,12 +89,12 @@ func TestRejectLeader(t *testing.T) { sl, err := schedule.CreateScheduler(LabelType, oc, storage.NewStorageWithMemoryBackend(), schedule.ConfigSliceDecoder(LabelType, []string{"", ""})) re.NoError(err) ops, _ := sl.Schedule(tc, false) - testutil.CheckTransferLeaderFrom(re, ops[0], operator.OpLeader, 1) + operatorutil.CheckTransferLeaderFrom(re, ops[0], operator.OpLeader, 1) // If store3 is disconnected, transfer leader to store 2. tc.SetStoreDisconnect(3) ops, _ = sl.Schedule(tc, false) - testutil.CheckTransferLeader(re, ops[0], operator.OpLeader, 1, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpLeader, 1, 2) // As store3 is disconnected, store1 rejects leader. Balancer will not create // any operators. @@ -121,7 +121,7 @@ func TestRejectLeader(t *testing.T) { origin, overlaps, rangeChanged := tc.SetRegion(region) tc.UpdateSubTree(region, origin, overlaps, rangeChanged) ops, _ = sl.Schedule(tc, false) - testutil.CheckTransferLeader(re, ops[0], operator.OpLeader, 1, 2) + operatorutil.CheckTransferLeader(re, ops[0], operator.OpLeader, 1, 2) } func TestRemoveRejectLeader(t *testing.T) { @@ -308,11 +308,11 @@ func TestShuffleRegionRole(t *testing.T) { conf.Roles = []string{"follower"} ops, _ := sl.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpKind(0), 2, 4) // transfer follower + operatorutil.CheckTransferPeer(re, ops[0], operator.OpKind(0), 2, 4) // transfer follower conf.Roles = []string{"learner"} ops, _ = sl.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferLearner(re, ops[0], operator.OpRegion, 3, 4) + operatorutil.CheckTransferLearner(re, ops[0], operator.OpRegion, 3, 4) } func TestSpecialUseHotRegion(t *testing.T) { @@ -346,7 +346,7 @@ func TestSpecialUseHotRegion(t *testing.T) { // balance region without label ops, _ := bs.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpKind(0), 1, 4) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpKind(0), 1, 4) // cannot balance to store 4 and 5 with label tc.AddLabelsStore(4, 0, map[string]string{"specialUse": "hotRegion"}) @@ -367,7 +367,7 @@ func TestSpecialUseHotRegion(t *testing.T) { tc.AddLeaderRegionWithWriteInfo(5, 3, 512*units.KiB*statistics.WriteReportInterval, 0, 0, statistics.WriteReportInterval, []uint64{1, 2}) ops, _ = hs.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 1, 4) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpHotRegion, 1, 4) } func TestSpecialUseReserved(t *testing.T) { @@ -398,7 +398,7 @@ func TestSpecialUseReserved(t *testing.T) { // balance region without label ops, _ := bs.Schedule(tc, false) re.Len(ops, 1) - testutil.CheckTransferPeer(re, ops[0], operator.OpKind(0), 1, 4) + operatorutil.CheckTransferPeer(re, ops[0], operator.OpKind(0), 1, 4) // cannot balance to store 4 with label tc.AddLabelsStore(4, 0, map[string]string{"specialUse": "reserved"}) diff --git a/server/schedulers/transfer_witness_leader_test.go b/server/schedulers/transfer_witness_leader_test.go index 8657feb3039..94b23b7adb9 100644 --- a/server/schedulers/transfer_witness_leader_test.go +++ b/server/schedulers/transfer_witness_leader_test.go @@ -24,7 +24,7 @@ import ( "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mock/mockcluster" "github.com/tikv/pd/pkg/storage" - "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/operatorutil" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/schedule" "github.com/tikv/pd/server/schedule/operator" @@ -50,7 +50,7 @@ func TestTransferWitnessLeader(t *testing.T) { RecvRegionInfo(sl) <- tc.GetRegion(1) re.True(sl.IsScheduleAllowed(tc)) ops, _ := sl.Schedule(tc, false) - testutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2, 3}) + operatorutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2, 3}) re.False(ops[0].Step(0).(operator.TransferLeader).IsFinish(tc.MockRegionInfo(1, 1, []uint64{2, 3}, []uint64{}, &metapb.RegionEpoch{ConfVer: 0, Version: 0}))) re.True(ops[0].Step(0).(operator.TransferLeader).IsFinish(tc.MockRegionInfo(1, 2, []uint64{1, 3}, []uint64{}, &metapb.RegionEpoch{ConfVer: 0, Version: 0}))) } @@ -82,14 +82,14 @@ func TestTransferWitnessLeaderWithUnhealthyPeer(t *testing.T) { tc.PutRegion(region.Clone(withPendingPeer)) RecvRegionInfo(sl) <- tc.GetRegion(1) ops, _ := sl.Schedule(tc, false) - testutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{3}) + operatorutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{3}) ops, _ = sl.Schedule(tc, false) re.Nil(ops) // only down tc.PutRegion(region.Clone(withDownPeer)) RecvRegionInfo(sl) <- tc.GetRegion(1) ops, _ = sl.Schedule(tc, false) - testutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2}) + operatorutil.CheckMultiTargetTransferLeader(re, ops[0], operator.OpLeader, 1, []uint64{2}) // pending + down tc.PutRegion(region.Clone(withPendingPeer, withDownPeer)) ops, _ = sl.Schedule(tc, false) diff --git a/server/server.go b/server/server.go index 65a73009420..f6d7ed44d3c 100644 --- a/server/server.go +++ b/server/server.go @@ -46,6 +46,7 @@ import ( "github.com/tikv/pd/pkg/encryption" "github.com/tikv/pd/pkg/errs" "github.com/tikv/pd/pkg/id" + "github.com/tikv/pd/pkg/keyspace" ms_server "github.com/tikv/pd/pkg/mcs/meta_storage/server" "github.com/tikv/pd/pkg/mcs/registry" rm_server "github.com/tikv/pd/pkg/mcs/resource_manager/server" @@ -70,13 +71,13 @@ import ( "github.com/tikv/pd/server/cluster" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/gc" - "github.com/tikv/pd/server/keyspace" syncer "github.com/tikv/pd/server/region_syncer" "github.com/tikv/pd/server/schedule" "github.com/tikv/pd/server/schedule/hbstream" "github.com/tikv/pd/server/schedule/placement" "go.etcd.io/etcd/clientv3" "go.etcd.io/etcd/embed" + "go.etcd.io/etcd/mvcc/mvccpb" "go.etcd.io/etcd/pkg/types" "go.uber.org/zap" "google.golang.org/grpc" @@ -84,7 +85,6 @@ import ( const ( serverMetricsInterval = time.Minute - leaderTickInterval = 50 * time.Millisecond // pdRootPath for all pd servers. pdRootPath = "/pd" pdAPIPrefix = "/pd/" @@ -105,8 +105,6 @@ const ( maxRetryTimesGetServicePrimary = 25 // retryIntervalGetServicePrimary is the retry interval for getting primary addr. retryIntervalGetServicePrimary = 100 * time.Millisecond - // TODO: move it to etcdutil - watchEtcdChangeRetryInterval = 1 * time.Second ) // EtcdStartTimeout the timeout of the startup etcd. @@ -124,8 +122,8 @@ var ( type Server struct { diagnosticspb.DiagnosticsServer - // Server state. - isServing int64 + // Server state. 0 is not running, 1 is running. + isRunning int64 // Server start timestamp startTimestamp int64 @@ -165,6 +163,8 @@ type Server struct { gcSafePointManager *gc.SafePointManager // keyspace manager keyspaceManager *keyspace.Manager + // keyspace group manager + keyspaceGroupManager *keyspace.GroupManager // for basicCluster operation. basicCluster *core.BasicCluster // for tso. @@ -197,7 +197,13 @@ type Server struct { // tsoDispatcher is used to dispatch different TSO requests to // the corresponding forwarding TSO channel. - tsoDispatcher sync.Map /* Store as map[string]chan *tsoRequest */ + tsoDispatcher *tsoutil.TSODispatcher + // tsoProtoFactory is the abstract factory for creating tso + // related data structures defined in the TSO grpc service + tsoProtoFactory *tsoutil.TSOProtoFactory + // pdProtoFactory is the abstract factory for creating tso + // related data structures defined in the PD grpc service + pdProtoFactory *tsoutil.PDProtoFactory serviceRateLimiter *ratelimit.Limiter serviceLabels map[string][]apiutil.AccessPath @@ -210,9 +216,7 @@ type Server struct { registry *registry.ServiceRegistry mode string servicePrimaryMap sync.Map /* Store as map[string]string */ - // updateServicePrimaryAddrCh is used to notify the server to update the service primary address. - // Note: it is only used in API service mode. - updateServicePrimaryAddrCh chan struct{} + tsoPrimaryWatcher *etcdutil.LoopWatcher } // HandlerBuilder builds a server HTTP handler. @@ -400,12 +404,11 @@ func (s *Server) startServer(ctx context.Context) error { } defaultStorage := storage.NewStorageWithEtcdBackend(s.client, s.rootPath) s.storage = storage.NewCoreStorage(defaultStorage, regionStorage) + s.tsoDispatcher = tsoutil.NewTSODispatcher(tsoProxyHandleDuration, tsoProxyBatchSize) + s.tsoProtoFactory = &tsoutil.TSOProtoFactory{} + s.pdProtoFactory = &tsoutil.PDProtoFactory{} if !s.IsAPIServiceMode() { - s.tsoAllocatorManager = tso.NewAllocatorManager( - s.member, s.rootPath, s.storage, s.cfg.IsLocalTSOEnabled(), s.cfg.GetTSOSaveInterval(), s.cfg.GetTSOUpdatePhysicalInterval(), s.cfg.GetTLSConfig(), - func() time.Duration { return s.persistOptions.GetMaxResetTSGap() }) - // Set up the Global TSO Allocator here, it will be initialized once the PD campaigns leader successfully. - s.tsoAllocatorManager.SetUpAllocator(ctx, tso.GlobalDCLocation, s.member.GetLeadership()) + s.tsoAllocatorManager = tso.NewAllocatorManager(s.ctx, mcs.DefaultKeyspaceGroupID, s.member, s.rootPath, s.storage, s, false) // When disabled the Local TSO, we should clean up the Local TSO Allocator's meta info written in etcd if it exists. if !s.cfg.EnableLocalTSO { if err = s.tsoAllocatorManager.CleanUpDCLocation(); err != nil { @@ -435,7 +438,10 @@ func (s *Server) startServer(ctx context.Context) error { Member: s.member.MemberValue(), Step: keyspace.AllocStep, }) - s.keyspaceManager = keyspace.NewKeyspaceManager(s.storage, s.cluster, keyspaceIDAllocator, s.cfg.Keyspace) + if s.IsAPIServiceMode() { + s.keyspaceGroupManager = keyspace.NewKeyspaceGroupManager(s.ctx, s.storage, s.client, s.clusterID) + } + s.keyspaceManager = keyspace.NewKeyspaceManager(s.ctx, s.storage, s.cluster, keyspaceIDAllocator, &s.cfg.Keyspace, s.keyspaceGroupManager) s.hbStreams = hbstream.NewHeartbeatStreams(ctx, s.clusterID, s.cluster) // initial hot_region_storage in here. s.hotRegionStorage, err = storage.NewHotRegionsStorage( @@ -450,7 +456,7 @@ func (s *Server) startServer(ctx context.Context) error { } // Server has started. - atomic.StoreInt64(&s.isServing, 1) + atomic.StoreInt64(&s.isRunning, 1) return nil } @@ -461,7 +467,7 @@ func (s *Server) AddCloseCallback(callbacks ...func()) { // Close closes the server. func (s *Server) Close() { - if !atomic.CompareAndSwapInt64(&s.isServing, 1, 0) { + if !atomic.CompareAndSwapInt64(&s.isRunning, 1, 0) { // server is already closed return } @@ -469,6 +475,9 @@ func (s *Server) Close() { log.Info("closing server") s.stopServerLoop() + if s.IsAPIServiceMode() { + s.keyspaceGroupManager.Close() + } if s.client != nil { if err := s.client.Close(); err != nil { @@ -506,7 +515,7 @@ func (s *Server) Close() { // IsClosed checks whether server is closed or not. func (s *Server) IsClosed() bool { - return atomic.LoadInt64(&s.isServing) == 0 + return atomic.LoadInt64(&s.isRunning) == 0 } // Run runs the pd server. @@ -554,12 +563,10 @@ func (s *Server) startServerLoop(ctx context.Context) { go s.etcdLeaderLoop() go s.serverMetricsLoop() go s.encryptionKeyManagerLoop() - if s.IsAPIServiceMode() { // disable tso service in api server - s.serverLoopWg.Add(1) - go s.startWatchServicePrimaryAddrLoop(mcs.TSOServiceName) - } else { // enable tso service + if s.IsAPIServiceMode() { + s.initTSOPrimaryWatcher() s.serverLoopWg.Add(1) - go s.tsoAllocatorLoop() + go s.tsoPrimaryWatcher.StartWatchLoop() } } @@ -585,17 +592,6 @@ func (s *Server) serverMetricsLoop() { } } -// tsoAllocatorLoop is used to run the TSO Allocator updating daemon. -func (s *Server) tsoAllocatorLoop() { - defer logutil.LogPanic() - defer s.serverLoopWg.Done() - - ctx, cancel := context.WithCancel(s.serverLoopCtx) - defer cancel() - s.tsoAllocatorManager.AllocatorDaemon(ctx) - log.Info("server is closed, exit allocator loop") -} - // encryptionKeyManagerLoop is used to start monitor encryption key changes. func (s *Server) encryptionKeyManagerLoop() { defer logutil.LogPanic() @@ -692,7 +688,7 @@ func (s *Server) bootstrapCluster(req *pdpb.BootstrapRequest) (*pdpb.BootstrapRe } if err = s.GetKeyspaceManager().Bootstrap(); err != nil { - log.Warn("bootstrap keyspace manager failed", errs.ZapError(err)) + log.Warn("bootstrapping keyspace manager failed", errs.ZapError(err)) } return &pdpb.BootstrapResponse{ @@ -822,6 +818,11 @@ func (s *Server) GetKeyspaceManager() *keyspace.Manager { return s.keyspaceManager } +// GetKeyspaceGroupManager returns the keyspace group manager of server. +func (s *Server) GetKeyspaceGroupManager() *keyspace.GroupManager { + return s.keyspaceGroupManager +} + // Name returns the unique etcd Name for this server in etcd cluster. func (s *Server) Name() string { return s.cfg.Name @@ -1329,7 +1330,7 @@ func (s *Server) GetServiceRateLimiter() *ratelimit.Limiter { return s.serviceRateLimiter } -// IsInRateLimitAllowList returns whethis given service label is in allow lost +// IsInRateLimitAllowList returns whether given service label is in allow lost func (s *Server) IsInRateLimitAllowList(serviceLabel string) bool { return s.serviceRateLimiter.IsInAllowList(serviceLabel) } @@ -1431,7 +1432,7 @@ func (s *Server) leaderLoop() { return } - leader, rev, checkAgain := s.member.CheckLeader() + leader, checkAgain := s.member.CheckLeader() if checkAgain { continue } @@ -1447,11 +1448,11 @@ func (s *Server) leaderLoop() { } syncer := s.cluster.GetRegionSyncer() if s.persistOptions.IsUseRegionStorage() { - syncer.StartSyncWithLeader(leader.GetClientUrls()[0]) + syncer.StartSyncWithLeader(leader.GetListenUrls()[0]) } log.Info("start to watch pd leader", zap.Stringer("pd-leader", leader)) // WatchLeader will keep looping and never return unless the PD leader has changed. - s.member.WatchLeader(s.serverLoopCtx, leader, rev) + leader.Watch(s.serverLoopCtx) syncer.StopSyncWithLeader() log.Info("pd leader has changed, try to re-campaign a pd leader") } @@ -1565,7 +1566,7 @@ func (s *Server) campaignLeader() { CheckPDVersion(s.persistOptions) log.Info(fmt.Sprintf("%s leader is ready to serve", s.mode), zap.String("leader-name", s.Name())) - leaderTicker := time.NewTicker(leaderTickInterval) + leaderTicker := time.NewTicker(mcs.LeaderTickInterval) defer leaderTicker.Stop() for { @@ -1752,126 +1753,46 @@ func (s *Server) GetServicePrimaryAddr(ctx context.Context, serviceName string) return "", false } -// startWatchServicePrimaryAddrLoop starts a loop to watch the primary address of a given service. -func (s *Server) startWatchServicePrimaryAddrLoop(serviceName string) { - defer logutil.LogPanic() - defer s.serverLoopWg.Done() - ctx, cancel := context.WithCancel(s.serverLoopCtx) - defer cancel() - s.updateServicePrimaryAddrCh = make(chan struct{}, 1) - serviceKey := s.servicePrimaryKey(serviceName) - var ( - revision int64 - err error - ) - for i := 0; i < maxRetryTimesGetServicePrimary; i++ { - revision, err = s.updateServicePrimaryAddr(serviceName) - if revision != 0 && err == nil { // update success - break - } - select { - case <-ctx.Done(): - return - case <-time.After(retryIntervalGetServicePrimary): - } - } - if err != nil { - log.Warn("service primary addr doesn't exist", zap.String("service-key", serviceKey), zap.Error(err)) - } - log.Info("start to watch service primary addr", zap.String("service-key", serviceKey)) - for { - select { - case <-ctx.Done(): - log.Info("server is closed, exist watch service primary addr loop", zap.String("service", serviceName)) - return - default: - } - nextRevision, err := s.watchServicePrimaryAddr(ctx, serviceName, revision) - if err != nil { - log.Error("watcher canceled unexpectedly and a new watcher will start after a while", - zap.Int64("next-revision", nextRevision), - zap.Time("retry-at", time.Now().Add(watchEtcdChangeRetryInterval)), - zap.Error(err)) - revision = nextRevision - time.Sleep(watchEtcdChangeRetryInterval) - } - } +// SetServicePrimaryAddr sets the primary address directly. +// Note: This function is only used for test. +func (s *Server) SetServicePrimaryAddr(serviceName, addr string) { + s.servicePrimaryMap.Store(serviceName, addr) } -// watchServicePrimaryAddr watches the primary address on etcd. -func (s *Server) watchServicePrimaryAddr(ctx context.Context, serviceName string, revision int64) (nextRevision int64, err error) { - serviceKey := s.servicePrimaryKey(serviceName) - watcher := clientv3.NewWatcher(s.client) - defer watcher.Close() +func (s *Server) servicePrimaryKey(serviceName string) string { + return fmt.Sprintf("/ms/%d/%s/%s/%s", s.clusterID, serviceName, fmt.Sprintf("%05d", 0), "primary") +} - for { - WatchChan: - watchChan := watcher.Watch(s.serverLoopCtx, serviceKey, clientv3.WithRev(revision)) - select { - case <-ctx.Done(): - return revision, nil - case <-s.updateServicePrimaryAddrCh: - revision, err = s.updateServicePrimaryAddr(serviceName) - if err != nil { - log.Warn("update service primary addr failed", zap.String("service-key", serviceKey), zap.Error(err)) - } - goto WatchChan - case wresp := <-watchChan: - if wresp.CompactRevision != 0 { - log.Warn("required revision has been compacted, use the compact revision", - zap.Int64("required-revision", revision), - zap.Int64("compact-revision", wresp.CompactRevision)) - revision = wresp.CompactRevision - goto WatchChan - } - if wresp.Err() != nil { - log.Error("watcher is canceled with", - zap.Int64("revision", revision), - errs.ZapError(errs.ErrEtcdWatcherCancel, wresp.Err())) - return revision, wresp.Err() - } - for _, event := range wresp.Events { - switch event.Type { - case clientv3.EventTypePut: - primary := &tsopb.Participant{} - if err := proto.Unmarshal(event.Kv.Value, primary); err != nil { - log.Error("watch service primary addr failed", zap.String("service-key", serviceKey), zap.Error(err)) - } else { - listenUrls := primary.GetListenUrls() - if len(listenUrls) > 0 { - // listenUrls[0] is the primary service endpoint of the keyspace group - s.servicePrimaryMap.Store(serviceName, listenUrls[0]) - } else { - log.Warn("service primary addr doesn't exist", zap.String("service-key", serviceKey)) - } - } - case clientv3.EventTypeDelete: - log.Warn("service primary is deleted", zap.String("service-key", serviceKey)) - s.servicePrimaryMap.Delete(serviceName) - } - } - revision = wresp.Header.Revision + 1 +func (s *Server) initTSOPrimaryWatcher() { + serviceName := mcs.TSOServiceName + tsoServicePrimaryKey := s.servicePrimaryKey(serviceName) + putFn := func(kv *mvccpb.KeyValue) error { + primary := &tsopb.Participant{} // TODO: use Generics + if err := proto.Unmarshal(kv.Value, primary); err != nil { + return err + } + listenUrls := primary.GetListenUrls() + if len(listenUrls) > 0 { + // listenUrls[0] is the primary service endpoint of the keyspace group + s.servicePrimaryMap.Store(serviceName, listenUrls[0]) + log.Info("update tso primary", zap.String("primary", listenUrls[0])) } + return nil } -} - -// updateServicePrimaryAddr updates the primary address from etcd with get operation. -func (s *Server) updateServicePrimaryAddr(serviceName string) (nextRevision int64, err error) { - serviceKey := s.servicePrimaryKey(serviceName) - primary := &tsopb.Participant{} - ok, revision, err := etcdutil.GetProtoMsgWithModRev(s.client, serviceKey, primary) - listenUrls := primary.GetListenUrls() - if !ok || err != nil || len(listenUrls) == 0 { - return 0, err + deleteFn := func(kv *mvccpb.KeyValue) error { + s.servicePrimaryMap.Delete(serviceName) + return nil } - // listenUrls[0] is the primary service endpoint of the keyspace group - s.servicePrimaryMap.Store(serviceName, listenUrls[0]) - log.Info("update service primary addr", zap.String("service-key", serviceKey), zap.String("primary-addr", listenUrls[0])) - return revision, nil -} - -func (s *Server) servicePrimaryKey(serviceName string) string { - return fmt.Sprintf("/ms/%d/%s/%s/%s", s.clusterID, serviceName, fmt.Sprintf("%05d", 0), "primary") + s.tsoPrimaryWatcher = etcdutil.NewLoopWatcher( + s.serverLoopCtx, + &s.serverLoopWg, + s.client, + "tso-primary-watcher", + tsoServicePrimaryKey, + putFn, + deleteFn, + func() error { return nil }, + ) } // RecoverAllocID recover alloc id. set current base id to input id @@ -1900,3 +1821,28 @@ func (s *Server) SetExternalTS(externalTS, globalTS uint64) error { s.GetRaftCluster().SetExternalTS(externalTS) return nil } + +// IsLocalTSOEnabled returns if the local TSO is enabled. +func (s *Server) IsLocalTSOEnabled() bool { + return s.cfg.IsLocalTSOEnabled() +} + +// GetLeaderLease returns the leader lease. +func (s *Server) GetLeaderLease() int64 { + return s.cfg.GetLeaderLease() +} + +// GetTSOSaveInterval returns TSO save interval. +func (s *Server) GetTSOSaveInterval() time.Duration { + return s.cfg.GetTSOSaveInterval() +} + +// GetTSOUpdatePhysicalInterval returns TSO update physical interval. +func (s *Server) GetTSOUpdatePhysicalInterval() time.Duration { + return s.cfg.GetTSOUpdatePhysicalInterval() +} + +// GetMaxResetTSGap gets the max gap to reset the tso. +func (s *Server) GetMaxResetTSGap() time.Duration { + return s.persistOptions.GetMaxResetTSGap() +} diff --git a/server/server_test.go b/server/server_test.go index 98039b2749f..09a4721728f 100644 --- a/server/server_test.go +++ b/server/server_test.go @@ -234,6 +234,7 @@ func (suite *leaderServerTestSuite) TestSourceIpForHeaderXReal() { cfg := NewTestSingleConfig(assertutil.CheckerWithNilAssert(suite.Require())) ctx, cancel := context.WithCancel(context.Background()) svr, err := CreateServer(ctx, cfg, nil, mockHandler) + suite.NoError(err) defer func() { cancel() svr.Close() diff --git a/tests/cluster.go b/tests/cluster.go index ba8040553bf..417468ce9bd 100644 --- a/tests/cluster.go +++ b/tests/cluster.go @@ -32,6 +32,7 @@ import ( "github.com/tikv/pd/pkg/dashboard" "github.com/tikv/pd/pkg/errs" "github.com/tikv/pd/pkg/id" + "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/swaggerserver" "github.com/tikv/pd/pkg/tso" @@ -43,7 +44,6 @@ import ( "github.com/tikv/pd/server/cluster" "github.com/tikv/pd/server/config" "github.com/tikv/pd/server/join" - "github.com/tikv/pd/server/keyspace" "go.etcd.io/etcd/clientv3" ) @@ -715,6 +715,20 @@ func (c *TestCluster) Join(ctx context.Context, opts ...ConfigOption) (*TestServ return s, nil } +// JoinAPIServer is used to add a new TestAPIServer into the cluster. +func (c *TestCluster) JoinAPIServer(ctx context.Context, opts ...ConfigOption) (*TestServer, error) { + conf, err := c.config.Join().Generate(opts...) + if err != nil { + return nil, err + } + s, err := NewTestAPIServer(ctx, conf) + if err != nil { + return nil, err + } + c.servers[conf.Name] = s + return s, nil +} + // Destroy is used to destroy a TestCluster. func (c *TestCluster) Destroy() { for _, s := range c.servers { diff --git a/tests/client/Makefile b/tests/integrations/client/Makefile similarity index 68% rename from tests/client/Makefile rename to tests/integrations/client/Makefile index 5663a2f3b45..e8c2f30e2fa 100644 --- a/tests/client/Makefile +++ b/tests/integrations/client/Makefile @@ -12,37 +12,35 @@ # See the License for the specific language governing permissions and # limitations under the License. -GO_TOOLS_BIN_PATH := $(shell pwd)/../../.tools/bin +ROOT_PATH := ../../.. +GO_TOOLS_BIN_PATH := $(ROOT_PATH)/.tools/bin PATH := $(GO_TOOLS_BIN_PATH):$(PATH) SHELL := env PATH='$(PATH)' GOBIN='$(GO_TOOLS_BIN_PATH)' $(shell which bash) static: install-tools @ gofmt -s -l -d . 2>&1 | awk '{ print } END { if (NR > 0) { exit 1 } }' @ golangci-lint run ./... - @ revive -formatter friendly -config ../../revive.toml . + @ revive -formatter friendly -config $(ROOT_PATH)/revive.toml . tidy: @ go mod tidy git diff go.mod go.sum | cat git diff --quiet go.mod go.sum -test: enable-codegen - CGO_ENABLED=1 go test -v -tags deadlock -race -cover || { $(MAKE) disable-codegen && exit 1; } - $(MAKE) disable-codegen +test: failpoint-enable + CGO_ENABLED=1 go test -v -tags deadlock -race -cover || { $(MAKE) failpoint-disable && exit 1; } + $(MAKE) failpoint-disable -basic-test: - # skip - -ci-test-job: enable-codegen - CGO_ENABLED=1 go test -tags deadlock -race -covermode=atomic -coverprofile=covprofile -coverpkg=../../... github.com/tikv/pd/tests/client +ci-test-job: + CGO_ENABLED=1 go test -tags deadlock -race -covermode=atomic -coverprofile=covprofile -coverpkg=$(ROOT_PATH)/... github.com/tikv/pd/tests/integrations/client install-tools: - cd ../../ && $(MAKE) install-tools + cd $(ROOT_PATH) && $(MAKE) install-tools -enable-codegen: - cd ../../ && $(MAKE) failpoint-enable +failpoint-enable: + cd $(ROOT_PATH) && $(MAKE) failpoint-enable go mod tidy -disable-codegen: - cd ../../ && $(MAKE) failpoint-disable +failpoint-disable: + cd $(ROOT_PATH) && $(MAKE) failpoint-disable go mod tidy diff --git a/tests/client/cert-expired/ca-config.json b/tests/integrations/client/cert-expired/ca-config.json similarity index 100% rename from tests/client/cert-expired/ca-config.json rename to tests/integrations/client/cert-expired/ca-config.json diff --git a/tests/client/cert-expired/ca-csr.json b/tests/integrations/client/cert-expired/ca-csr.json similarity index 100% rename from tests/client/cert-expired/ca-csr.json rename to tests/integrations/client/cert-expired/ca-csr.json diff --git a/tests/client/cert-expired/ca-key.pem b/tests/integrations/client/cert-expired/ca-key.pem similarity index 100% rename from tests/client/cert-expired/ca-key.pem rename to tests/integrations/client/cert-expired/ca-key.pem diff --git a/tests/client/cert-expired/ca.csr b/tests/integrations/client/cert-expired/ca.csr similarity index 100% rename from tests/client/cert-expired/ca.csr rename to tests/integrations/client/cert-expired/ca.csr diff --git a/tests/client/cert-expired/ca.pem b/tests/integrations/client/cert-expired/ca.pem similarity index 100% rename from tests/client/cert-expired/ca.pem rename to tests/integrations/client/cert-expired/ca.pem diff --git a/tests/client/cert-expired/client-key.pem b/tests/integrations/client/cert-expired/client-key.pem similarity index 100% rename from tests/client/cert-expired/client-key.pem rename to tests/integrations/client/cert-expired/client-key.pem diff --git a/tests/client/cert-expired/client.csr b/tests/integrations/client/cert-expired/client.csr similarity index 100% rename from tests/client/cert-expired/client.csr rename to tests/integrations/client/cert-expired/client.csr diff --git a/tests/client/cert-expired/client.pem b/tests/integrations/client/cert-expired/client.pem similarity index 100% rename from tests/client/cert-expired/client.pem rename to tests/integrations/client/cert-expired/client.pem diff --git a/tests/client/cert-expired/gencerts.sh b/tests/integrations/client/cert-expired/gencerts.sh similarity index 100% rename from tests/client/cert-expired/gencerts.sh rename to tests/integrations/client/cert-expired/gencerts.sh diff --git a/tests/client/cert-expired/pd-server-key.pem b/tests/integrations/client/cert-expired/pd-server-key.pem similarity index 100% rename from tests/client/cert-expired/pd-server-key.pem rename to tests/integrations/client/cert-expired/pd-server-key.pem diff --git a/tests/client/cert-expired/pd-server.csr b/tests/integrations/client/cert-expired/pd-server.csr similarity index 100% rename from tests/client/cert-expired/pd-server.csr rename to tests/integrations/client/cert-expired/pd-server.csr diff --git a/tests/client/cert-expired/pd-server.pem b/tests/integrations/client/cert-expired/pd-server.pem similarity index 100% rename from tests/client/cert-expired/pd-server.pem rename to tests/integrations/client/cert-expired/pd-server.pem diff --git a/tests/client/cert/ca-config.json b/tests/integrations/client/cert/ca-config.json similarity index 100% rename from tests/client/cert/ca-config.json rename to tests/integrations/client/cert/ca-config.json diff --git a/tests/client/cert/ca-csr.json b/tests/integrations/client/cert/ca-csr.json similarity index 100% rename from tests/client/cert/ca-csr.json rename to tests/integrations/client/cert/ca-csr.json diff --git a/tests/client/cert/ca-key.pem b/tests/integrations/client/cert/ca-key.pem similarity index 100% rename from tests/client/cert/ca-key.pem rename to tests/integrations/client/cert/ca-key.pem diff --git a/tests/client/cert/ca.csr b/tests/integrations/client/cert/ca.csr similarity index 100% rename from tests/client/cert/ca.csr rename to tests/integrations/client/cert/ca.csr diff --git a/tests/client/cert/ca.pem b/tests/integrations/client/cert/ca.pem similarity index 100% rename from tests/client/cert/ca.pem rename to tests/integrations/client/cert/ca.pem diff --git a/tests/client/cert/client-key.pem b/tests/integrations/client/cert/client-key.pem similarity index 100% rename from tests/client/cert/client-key.pem rename to tests/integrations/client/cert/client-key.pem diff --git a/tests/client/cert/client.csr b/tests/integrations/client/cert/client.csr similarity index 100% rename from tests/client/cert/client.csr rename to tests/integrations/client/cert/client.csr diff --git a/tests/client/cert/client.pem b/tests/integrations/client/cert/client.pem similarity index 100% rename from tests/client/cert/client.pem rename to tests/integrations/client/cert/client.pem diff --git a/tests/client/cert/gencerts.sh b/tests/integrations/client/cert/gencerts.sh similarity index 100% rename from tests/client/cert/gencerts.sh rename to tests/integrations/client/cert/gencerts.sh diff --git a/tests/client/cert/pd-server-key.pem b/tests/integrations/client/cert/pd-server-key.pem similarity index 100% rename from tests/client/cert/pd-server-key.pem rename to tests/integrations/client/cert/pd-server-key.pem diff --git a/tests/client/cert/pd-server.csr b/tests/integrations/client/cert/pd-server.csr similarity index 100% rename from tests/client/cert/pd-server.csr rename to tests/integrations/client/cert/pd-server.csr diff --git a/tests/client/cert/pd-server.pem b/tests/integrations/client/cert/pd-server.pem similarity index 100% rename from tests/client/cert/pd-server.pem rename to tests/integrations/client/cert/pd-server.pem diff --git a/tests/client/client_test.go b/tests/integrations/client/client_test.go similarity index 99% rename from tests/client/client_test.go rename to tests/integrations/client/client_test.go index 9562296ccdd..84ad919c382 100644 --- a/tests/client/client_test.go +++ b/tests/integrations/client/client_test.go @@ -143,7 +143,7 @@ func TestClientLeaderChange(t *testing.T) { // Check URL list. cli.Close() - urls := innerCli.GetServiceDiscovery().GetURLs() + urls := innerCli.GetServiceDiscovery().GetServiceURLs() sort.Strings(urls) sort.Strings(endpoints) re.Equal(endpoints, urls) @@ -255,7 +255,7 @@ func TestTSOAllocatorLeader(t *testing.T) { cli.Close() for dcLocation, url := range getTSOAllocatorServingEndpointURLs(cli.(TSOAllocatorsGetter)) { if dcLocation == tso.GlobalDCLocation { - urls := innerCli.GetServiceDiscovery().GetURLs() + urls := innerCli.GetServiceDiscovery().GetServiceURLs() sort.Strings(urls) sort.Strings(endpoints) re.Equal(endpoints, urls) @@ -1367,7 +1367,10 @@ func TestWatch(t *testing.T) { defer client.Close() key := "test" - ch, err := client.Watch(ctx, []byte(key)) + resp, err := client.Get(ctx, []byte(key)) + re.NoError(err) + rev := resp.GetHeader().GetRevision() + ch, err := client.Watch(ctx, []byte(key), pd.WithRev(rev)) re.NoError(err) exit := make(chan struct{}) go func() { diff --git a/tests/client/client_tls_test.go b/tests/integrations/client/client_tls_test.go similarity index 100% rename from tests/client/client_tls_test.go rename to tests/integrations/client/client_tls_test.go diff --git a/tests/client/global_config_test.go b/tests/integrations/client/global_config_test.go similarity index 100% rename from tests/client/global_config_test.go rename to tests/integrations/client/global_config_test.go diff --git a/tests/client/go.mod b/tests/integrations/client/go.mod similarity index 92% rename from tests/client/go.mod rename to tests/integrations/client/go.mod index c4f138744cc..c303c567056 100644 --- a/tests/client/go.mod +++ b/tests/integrations/client/go.mod @@ -1,12 +1,20 @@ -module github.com/tikv/pd/tests/client +module github.com/tikv/pd/tests/integrations/client go 1.19 +replace ( + github.com/tikv/pd => ../../../ + github.com/tikv/pd/client => ../../../client +) + +// reset grpc and protobuf deps in order to import client and server at the same time +replace google.golang.org/grpc v1.51.0 => google.golang.org/grpc v1.26.0 + require ( github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 - github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba + github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 - github.com/stretchr/testify v1.8.1 + github.com/stretchr/testify v1.8.2 github.com/tikv/pd v0.0.0-00010101000000-000000000000 github.com/tikv/pd/client v0.0.0-00010101000000-000000000000 go.etcd.io/etcd v0.5.0-alpha.5.0.20220915004622-85b640cee793 @@ -121,8 +129,8 @@ require ( github.com/spf13/cobra v1.0.0 // indirect github.com/spf13/pflag v1.0.5 // indirect github.com/stretchr/objx v0.5.0 // indirect - github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 // indirect - github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba // indirect + github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 // indirect + github.com/swaggo/http-swagger v1.2.6 // indirect github.com/swaggo/swag v1.8.3 // indirect github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 // indirect github.com/tklauser/go-sysconf v0.3.11 // indirect @@ -142,14 +150,14 @@ require ( go.uber.org/multierr v1.7.0 // indirect golang.org/x/crypto v0.1.0 // indirect golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a // indirect - golang.org/x/image v0.0.0-20200119044424-58c23975cae1 // indirect - golang.org/x/net v0.2.0 // indirect - golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 // indirect + golang.org/x/image v0.5.0 // indirect + golang.org/x/net v0.9.0 // indirect + golang.org/x/oauth2 v0.4.0 // indirect golang.org/x/sync v0.1.0 // indirect - golang.org/x/sys v0.3.0 // indirect - golang.org/x/text v0.4.0 // indirect + golang.org/x/sys v0.7.0 // indirect + golang.org/x/text v0.9.0 // indirect golang.org/x/time v0.1.0 // indirect - golang.org/x/tools v0.2.0 // indirect + golang.org/x/tools v0.6.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/genproto v0.0.0-20221202195650-67e5cbc046fd // indirect google.golang.org/protobuf v1.28.1 // indirect @@ -163,11 +171,3 @@ require ( moul.io/zapgorm2 v1.1.0 // indirect sigs.k8s.io/yaml v1.2.0 // indirect ) - -replace ( - github.com/tikv/pd => ../../ - github.com/tikv/pd/client => ../../client -) - -// reset grpc and protobuf deps in order to import client and server at the same time -replace google.golang.org/grpc v1.51.0 => google.golang.org/grpc v1.26.0 diff --git a/tests/client/go.sum b/tests/integrations/client/go.sum similarity index 94% rename from tests/client/go.sum rename to tests/integrations/client/go.sum index 1f9936d1023..c426614602b 100644 --- a/tests/client/go.sum +++ b/tests/integrations/client/go.sum @@ -7,7 +7,6 @@ github.com/KyleBanks/depth v1.2.1/go.mod h1:jzSb9d0L43HxTQfT+oSA1EEp2q+ne2uh6Xge github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= -github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= @@ -20,6 +19,7 @@ github.com/VividCortex/mysqlerr v1.0.0 h1:5pZ2TZA+YnzPgzBfiUWGqWmKDVNBdrkf9g+DNe github.com/VividCortex/mysqlerr v1.0.0/go.mod h1:xERx8E4tBhLvpjzdUyQiSfUxeMcATEQrflDAfXsqcAE= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502 h1:L8IbaI/W6h5Cwgh0n4zGeZpVK78r/jBf9ASurHo9+/o= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502/go.mod h1:pmnBM9bxWSiHvC/gSWunUIyDvGn33EkP2CUjxFKtTTM= +github.com/agiledragon/gomonkey/v2 v2.3.1/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= @@ -73,6 +73,7 @@ github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbp github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/corona10/goimagehash v1.0.2 h1:pUfB0LnsJASMPGEZLj7tGY251vF+qLGqOgEP4rUs6kA= github.com/corona10/goimagehash v1.0.2/go.mod h1:/l9umBhvcHQXVtQO1V6Gp1yD20STawkhRnnX0D1bvVI= +github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/creack/pty v1.1.11/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= @@ -103,16 +104,12 @@ github.com/gin-contrib/cors v1.4.0/go.mod h1:bs9pNM0x/UsmHPBWT2xZz9ROh8xYjYkiURU github.com/gin-contrib/gzip v0.0.1 h1:ezvKOL6jH+jlzdHNE4h9h8q8uMpDQjyl0NN0Jd7jozc= github.com/gin-contrib/gzip v0.0.1/go.mod h1:fGBJBCdt6qCZuCAOwWuFhBB4OOq9EFqlo5dEaFhhu5w= github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= -github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE= github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI= github.com/gin-gonic/gin v1.3.0/go.mod h1:7cKuhb5qV2ggCFctp2fJQ+ErvciLZrIeoOSOm6mUr7Y= -github.com/gin-gonic/gin v1.4.0/go.mod h1:OW2EZn3DO8Ln9oIKOvM++LBO+5UPHJJDH72/q/3rZdM= github.com/gin-gonic/gin v1.6.3/go.mod h1:75u5sXoLsGZoRN5Sgbi1eraJ4GU3++wFwWzhwvtwp4M= github.com/gin-gonic/gin v1.8.1 h1:4+fr/el88TOO3ewCmQr8cx/CtZ/umlIRIs5M4NTNjf8= github.com/gin-gonic/gin v1.8.1/go.mod h1:ji8BvRH1azfM+SYow9zQ6SZMvR8qOMZHmsCuWR9tTTk= -github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAUcHlgXVRs= -github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY= @@ -122,18 +119,13 @@ github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM= github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= -github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= -github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= -github.com/go-openapi/jsonreference v0.19.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= github.com/go-openapi/jsonreference v0.19.6 h1:UBIxjkht+AWIgYzCDSv2GN+E/togfwXUJFRTWhl2Jjs= github.com/go-openapi/jsonreference v0.19.6/go.mod h1:diGHMEHg2IqXZGKxqyvWdfWU/aim5Dprw5bqpKkTvns= -github.com/go-openapi/spec v0.19.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= github.com/go-openapi/spec v0.20.4 h1:O8hJrt0UMnhHcluhIdUgCLRWyM2x7QkBXRvOs7m+O1M= github.com/go-openapi/spec v0.20.4/go.mod h1:faYFR1CvsJZ0mNsmsphTMSoRrNV3TEDoAM7FOEWeq8I= -github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= github.com/go-openapi/swag v0.19.15 h1:D2NRCBzS9/pEY3gP9Nl8aDqGUcPFrwG2p+CNFrLyrCM= github.com/go-openapi/swag v0.19.15/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= @@ -215,7 +207,7 @@ github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= +github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/mux v1.7.4 h1:VuZ8uybHlWmqV03+zRzdwKL4tUnIp1MAQtp1mIFE1bc= github.com/gorilla/mux v1.7.4/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= @@ -278,6 +270,7 @@ github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/ github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= @@ -301,15 +294,12 @@ github.com/leodido/go-urn v1.2.1/go.mod h1:zt4jvISO2HfUBqxjfIshjdMTYS56ZS/qv49ic github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4= github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= -github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA= github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= -github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= -github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= @@ -347,6 +337,11 @@ github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1Cpa github.com/onsi/gomega v1.20.1 h1:PA/3qinGoukvymdIDV8pii6tiZgC8kbmJO6Z5+b002Q= github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs= github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc= +github.com/otiai10/copy v1.7.0/go.mod h1:rmRl6QPdJj6EiUqXQ/4Nn2lLXoNQjFCQbbNrxgc/t3U= +github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE= +github.com/otiai10/curr v1.0.0/go.mod h1:LskTG5wDwr8Rs+nNQ+1LlxRjAtTZZjtJW4rMXl6j4vs= +github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo= +github.com/otiai10/mint v1.3.3/go.mod h1:/yxELlJQ0ufhjUwhshSj+wFjZ78CnZ48/1wtmBH1OTc= github.com/pascaldekloe/name v0.0.0-20180628100202-0fd16699aae1/go.mod h1:eD5JxqMiuNYyFNmyY9rkJ/slN8y59oEu4Ei7F8OoKWQ= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml/v2 v2.0.1 h1:8e3L2cCQzLFi2CR4g7vGFuFxX7Jl1kKX8gW+iV0GUKU= @@ -369,8 +364,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ue github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 h1:C3N3itkduZXDZFh4N3vQ5HEtld3S+Y+StULhWVvumU0= github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00/go.mod h1:4qGtCB0QK0wBzKtFEGDhxXnSnbQApw1gc9siScUl8ew= github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba h1:7g2yM0llENlRqtjboBKFBJ8N9SE01hPDpKuTwxBLpLM= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba/go.mod h1:RjuuhxITxwATlt5adgTedg3ehKk01M03L1U4jNHdeeQ= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 h1:VXQ6Du/nKZ9IQnI9NWMzKbftWu8NV5pQkSLKIRzzGN4= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1/go.mod h1:guCyM5N+o+ru0TsoZ1hi9lDjUMs2sIBjW3ARTEpVbnk= github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8= github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= @@ -441,6 +436,8 @@ github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072 h1:Txo4SXVJq/OgEjwgkWoxkMoTjGlcrgsQE/XSghjmu0w= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072/go.mod h1:+4nWMF0+CqEcU74SnX2NxaGqZ8zX4pcQ8Jcs77DbX5A= +github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= +github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= @@ -468,15 +465,14 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 h1:PyYN9JH5jY9j6av01SpfRMb+1DWg/i3MbGOKPxJ2wjM= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14/go.mod h1:gxQT6pBGRuIGunNf/+tSOB5OHvguWi8Tbt82WOkf35E= -github.com/swaggo/gin-swagger v1.2.0/go.mod h1:qlH2+W7zXGZkczuL+r2nEBR2JTT+/lX05Nn6vPhc7OI= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba h1:lUPlXKqgbqT2SVg2Y+eT9mu5wbqMnG+i/+Q9nK7C0Rs= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba/go.mod h1:O1lAbCgAAX/KZ80LM/OXwtWFI/5TvZlwxSg8Cq08PV0= -github.com/swaggo/swag v1.5.1/go.mod h1:1Bl9F/ZBpVWh22nY0zmYyASPO1lI/zIwRDrpZU+tv8Y= -github.com/swaggo/swag v1.6.3/go.mod h1:wcc83tB4Mb2aNiL/HP4MFeQdpHUrca+Rp/DRNgWAUio= +github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= +github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 h1:+iNTcqQJy0OZ5jk6a5NLib47eqXK8uYcPX+O4+cBpEM= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2/go.mod h1:lKJPbtWzJ9JhsTN1k1gZgleJWY/cqq0psdoMmaThG3w= +github.com/swaggo/http-swagger v1.2.6 h1:ihTjChUoSRMpFMjWw+0AkL1Ti4r6v8pCgVYLmQVRlRw= +github.com/swaggo/http-swagger v1.2.6/go.mod h1:CcoICgY3yVDk2u1LQUCMHbAj0fjlxIX+873psXlIKNA= +github.com/swaggo/swag v1.7.9/go.mod h1:gZ+TJ2w/Ve1RwQsA2IRoSOTidHz6DX+PIG8GWvbnoLU= github.com/swaggo/swag v1.8.3 h1:3pZSSCQ//gAH88lfmxM3Cd1+JCsxV8Md6f36b9hrZ5s= github.com/swaggo/swag v1.8.3/go.mod h1:jMLeXOOmYyjk8PvHTsXBdrubsNd9gUJTTCzL5iBnseg= github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 h1:1oFLiOyVl+W7bnBzGhf7BbIv9loSFQcieWWYIjLqcAw= @@ -497,17 +493,16 @@ github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1 github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966 h1:j6JEOq5QWFker+d7mFQYOhjTZonQ7YkLTHm56dbn+yM= github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= -github.com/ugorji/go v1.1.5-pre/go.mod h1:FwP/aQVg39TXzItUBMwnWp9T9gPQnXw4Poh4/oBQZ/0= github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw= github.com/ugorji/go v1.2.7/go.mod h1:nF9osbDWLy6bDVv/Rtoh6QgnvNDpmCalQV5urGCCS6M= github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= -github.com/ugorji/go/codec v1.1.5-pre/go.mod h1:tULtS6Gy1AE1yCENaw4Vb//HLH5njI2tfCQDUqRd8fI= github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY= github.com/ugorji/go/codec v1.2.7 h1:YPXUKf7fYbp/y8xloBqZOw2qaVggbfwMlI8WM3wZUJ0= github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95kRgeqEY= github.com/unrolled/render v1.0.1 h1:VDDnQQVfBMsOsp3VaCJszSO0nkBIVEYoPWeRThk9spY= github.com/unrolled/render v1.0.1/go.mod h1:gN9T0NhL4Bfbwu8ann7Ry/TGHYfosul+J0obPf6NBdM= github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA= +github.com/urfave/cli/v2 v2.3.0/go.mod h1:LJmUH05zAU44vOAcrfzZQKsZbVcdbOG8rtL3/XcUArI= github.com/urfave/negroni v0.3.0 h1:PaXOb61mWeZJxc1Ji2xJjpVg9QfPo0rrB+lHyBxGNSU= github.com/urfave/negroni v0.3.0/go.mod h1:Meg73S6kFm/4PpbYdq35yYWoCZ9mS/YSx+lKnmiohz4= github.com/vmihailenco/msgpack/v5 v5.3.5 h1:5gO0H1iULLWGhs2H5tbAHIZTV8/cYafcFOr9znI5mJU= @@ -520,6 +515,8 @@ github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1: github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg= github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= @@ -567,14 +564,16 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20200204104054-c9f3fb736b72/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU= golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a h1:tlXy25amD5A7gOfbXdqCGN5k8ESEed/Ee1E5RcrYnqU= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= -golang.org/x/image v0.0.0-20200119044424-58c23975cae1 h1:5h3ngYt7+vXCDZCup/HkCQgW5XwmSvR/nA2JmJ0RErg= golang.org/x/image v0.0.0-20200119044424-58c23975cae1/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= +golang.org/x/image v0.5.0 h1:5JMiNunQeQw++mMOz48/ISeNu3Iweh/JaZU8ZLqHRrI= +golang.org/x/image v0.5.0/go.mod h1:FVC7BI/5Ym8R25iw5OLsgshdUBbT1h5jZTpA+mvAdZ4= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= @@ -586,7 +585,8 @@ golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.6.0 h1:b9gGHsz9/HhJ3HF5DHQytPpuwocVTChQJK3AvoLRD5I= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -597,10 +597,8 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= -golang.org/x/net v0.0.0-20190611141213-3f473d35a33a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= @@ -612,14 +610,16 @@ golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwY golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM= +golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.2.0 h1:sZfSu1wtKLGlWI4ZZayP0ck9Y73K1ynO6gqzTdBVdPU= -golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 h1:nt+Q6cXKz4MosCSpnbMtqiQ8Oz0pxTef2B4Vca2lvfk= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg= +golang.org/x/oauth2 v0.4.0 h1:NF0gk8LVPg1Ml7SSbGyySuoxdsXitj7TvgvuRxIMc/M= +golang.org/x/oauth2 v0.4.0/go.mod h1:RznEsdpjGAINPTOF0UH/t+xJ75L18YO3Ho6Pyn+uRec= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -628,6 +628,7 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -637,10 +638,8 @@ golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181228144115-9a3f9b0469bb/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190610200419-93c9922d18ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -661,19 +660,25 @@ golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ= golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg= -golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.1.0 h1:xYY+Bajn2a7VBmTM5GikTmnK8ZuX8YgnQCqZpbBNtmA= @@ -683,10 +688,9 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190524210228-3d17549cdc6b/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.0.0-20190606050223-4d9ae51c2468/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190611222205-d73e1c7e250b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= @@ -701,8 +705,10 @@ golang.org/x/tools v0.0.0-20201125231158-b5590deeca9b/go.mod h1:emZCQorbCU4vsT4f golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210112230658-8b4aab62c064/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.2.0 h1:G6AHpWxTMGY1KyEYoAQ5WTtIekUUvDNjan3ugu60JvE= -golang.org/x/tools v0.2.0/go.mod h1:y4OqIKeOV/fWJetJ8bXPU1sEVniLMIyDAZWeHdV+NTA= +golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= diff --git a/tests/client/keyspace_test.go b/tests/integrations/client/keyspace_test.go similarity index 91% rename from tests/client/keyspace_test.go rename to tests/integrations/client/keyspace_test.go index a9fb953cdf5..28a9ceed116 100644 --- a/tests/client/keyspace_test.go +++ b/tests/integrations/client/keyspace_test.go @@ -20,9 +20,10 @@ import ( "github.com/pingcap/kvproto/pkg/keyspacepb" "github.com/stretchr/testify/require" + "github.com/tikv/pd/pkg/keyspace" + "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/slice" "github.com/tikv/pd/server" - "github.com/tikv/pd/server/keyspace" ) const ( @@ -37,12 +38,13 @@ func mustMakeTestKeyspaces(re *require.Assertions, server *server.Server, start, manager := server.GetKeyspaceManager() for i := 0; i < count; i++ { keyspaces[i], err = manager.CreateKeyspace(&keyspace.CreateKeyspaceRequest{ - Name: fmt.Sprintf("test_keyspace%d", start+i), + Name: fmt.Sprintf("test_keyspace_%d", start+i), Config: map[string]string{ testConfig1: "100", testConfig2: "200", }, - Now: now, + CreateTime: now, + IsPreAlloc: true, // skip wait region split }) re.NoError(err) } @@ -61,10 +63,10 @@ func (suite *clientTestSuite) TestLoadKeyspace() { _, err := suite.client.LoadKeyspace(suite.ctx, "non-existing keyspace") re.Error(err) // Loading default keyspace should be successful. - keyspaceDefault, err := suite.client.LoadKeyspace(suite.ctx, keyspace.DefaultKeyspaceName) + keyspaceDefault, err := suite.client.LoadKeyspace(suite.ctx, utils.DefaultKeyspaceName) re.NoError(err) - re.Equal(keyspace.DefaultKeyspaceID, keyspaceDefault.GetId()) - re.Equal(keyspace.DefaultKeyspaceName, keyspaceDefault.GetName()) + re.Equal(utils.DefaultKeyspaceID, keyspaceDefault.GetId()) + re.Equal(utils.DefaultKeyspaceName, keyspaceDefault.GetName()) } func (suite *clientTestSuite) TestWatchKeyspaces() { @@ -105,7 +107,7 @@ func (suite *clientTestSuite) TestWatchKeyspaces() { loaded = <-watchChan re.Equal([]*keyspacepb.KeyspaceMeta{expected}, loaded) // Updates to default keyspace's config should also be captured. - expected, err = suite.srv.GetKeyspaceManager().UpdateKeyspaceConfig(keyspace.DefaultKeyspaceName, []*keyspace.Mutation{ + expected, err = suite.srv.GetKeyspaceManager().UpdateKeyspaceConfig(utils.DefaultKeyspaceName, []*keyspace.Mutation{ { Op: keyspace.OpPut, Key: "config", @@ -120,9 +122,10 @@ func (suite *clientTestSuite) TestWatchKeyspaces() { func mustCreateKeyspaceAtState(re *require.Assertions, server *server.Server, index int, state keyspacepb.KeyspaceState) *keyspacepb.KeyspaceMeta { manager := server.GetKeyspaceManager() meta, err := manager.CreateKeyspace(&keyspace.CreateKeyspaceRequest{ - Name: fmt.Sprintf("test_keyspace%d", index), - Config: nil, - Now: 0, // Use 0 to indicate unchanged keyspace. + Name: fmt.Sprintf("test_keyspace_%d", index), + Config: nil, + CreateTime: 0, // Use 0 to indicate unchanged keyspace. + IsPreAlloc: true, // skip wait region split }) re.NoError(err) switch state { diff --git a/tests/mcs/Makefile b/tests/integrations/mcs/Makefile similarity index 64% rename from tests/mcs/Makefile rename to tests/integrations/mcs/Makefile index 86dc5f58bd2..8be9ad22429 100644 --- a/tests/mcs/Makefile +++ b/tests/integrations/mcs/Makefile @@ -12,34 +12,35 @@ # See the License for the specific language governing permissions and # limitations under the License. -GO_TOOLS_BIN_PATH := $(shell pwd)/../../.tools/bin +ROOT_PATH := ../../.. +GO_TOOLS_BIN_PATH := $(ROOT_PATH)/.tools/bin PATH := $(GO_TOOLS_BIN_PATH):$(PATH) SHELL := env PATH='$(PATH)' GOBIN='$(GO_TOOLS_BIN_PATH)' $(shell which bash) static: install-tools @ gofmt -s -l -d . 2>&1 | awk '{ print } END { if (NR > 0) { exit 1 } }' @ golangci-lint run ./... - @ revive -formatter friendly -config ../../revive.toml . + @ revive -formatter friendly -config $(ROOT_PATH)/revive.toml . tidy: @ go mod tidy git diff go.mod go.sum | cat git diff --quiet go.mod go.sum -test: enable-codegen - CGO_ENABLED=1 go test ./... -tags deadlock -race -cover || { $(MAKE) disable-codegen && exit 1; } - $(MAKE) disable-codegen +test: failpoint-enable + CGO_ENABLED=1 go test ./... -v -tags deadlock -race -cover || { $(MAKE) failpoint-disable && exit 1; } + $(MAKE) failpoint-disable -ci-test-job: enable-codegen - CGO_ENABLED=1 go test ./... -tags deadlock -race -covermode=atomic -coverprofile=covprofile -coverpkg=../../... github.com/tikv/pd/tests/mcs +ci-test-job: + CGO_ENABLED=1 go test ./... -tags deadlock -race -covermode=atomic -coverprofile=covprofile -coverpkg=$(ROOT_PATH)/... github.com/tikv/pd/tests/integrations/mcs install-tools: - cd ../../ && $(MAKE) install-tools + cd $(ROOT_PATH) && $(MAKE) install-tools -enable-codegen: - cd ../../ && $(MAKE) failpoint-enable +failpoint-enable: + cd $(ROOT_PATH) && $(MAKE) failpoint-enable go mod tidy -disable-codegen: - cd ../../ && $(MAKE) failpoint-disable +failpoint-disable: + cd $(ROOT_PATH) && $(MAKE) failpoint-disable go mod tidy diff --git a/tests/integrations/mcs/cluster.go b/tests/integrations/mcs/cluster.go new file mode 100644 index 00000000000..dbc9964b62b --- /dev/null +++ b/tests/integrations/mcs/cluster.go @@ -0,0 +1,158 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package mcs + +import ( + "context" + "fmt" + "time" + + "github.com/stretchr/testify/require" + tso "github.com/tikv/pd/pkg/mcs/tso/server" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/utils/tempurl" + "github.com/tikv/pd/pkg/utils/testutil" +) + +// TestTSOCluster is a test cluster for TSO. +type TestTSOCluster struct { + ctx context.Context + + backendEndpoints string + servers map[string]*tso.Server + cleanupFuncs map[string]testutil.CleanupFunc +} + +// NewTestTSOCluster creates a new TSO test cluster. +func NewTestTSOCluster(ctx context.Context, initialServerCount int, backendEndpoints string) (tc *TestTSOCluster, err error) { + tc = &TestTSOCluster{ + ctx: ctx, + backendEndpoints: backendEndpoints, + servers: make(map[string]*tso.Server, initialServerCount), + cleanupFuncs: make(map[string]testutil.CleanupFunc, initialServerCount), + } + for i := 0; i < initialServerCount; i++ { + err = tc.AddServer(tempurl.Alloc()) + if err != nil { + return nil, err + } + } + return tc, nil +} + +// AddServer adds a new TSO server to the test cluster. +func (tc *TestTSOCluster) AddServer(addr string) error { + cfg := tso.NewConfig() + cfg.BackendEndpoints = tc.backendEndpoints + cfg.ListenAddr = addr + cfg.Name = cfg.ListenAddr + generatedCfg, err := tso.GenerateConfig(cfg) + if err != nil { + return err + } + err = InitLogger(generatedCfg) + if err != nil { + return err + } + server, cleanup, err := NewTSOTestServer(tc.ctx, generatedCfg) + if err != nil { + return err + } + tc.servers[generatedCfg.GetListenAddr()] = server + tc.cleanupFuncs[generatedCfg.GetListenAddr()] = cleanup + return nil +} + +// Destroy stops and destroy the test cluster. +func (tc *TestTSOCluster) Destroy() { + for _, cleanup := range tc.cleanupFuncs { + cleanup() + } + tc.cleanupFuncs = nil + tc.servers = nil +} + +// DestroyServer stops and destroy the test server by the given address. +func (tc *TestTSOCluster) DestroyServer(addr string) { + tc.cleanupFuncs[addr]() + delete(tc.cleanupFuncs, addr) + delete(tc.servers, addr) +} + +// ResignPrimary resigns the primary TSO server. +func (tc *TestTSOCluster) ResignPrimary(keyspaceID, keyspaceGroupID uint32) error { + primaryServer := tc.GetPrimaryServer(keyspaceID, keyspaceGroupID) + if primaryServer == nil { + return fmt.Errorf("no tso server serves this keyspace %d", keyspaceID) + } + return primaryServer.ResignPrimary(keyspaceID, keyspaceGroupID) +} + +// GetPrimaryServer returns the primary TSO server of the given keyspace +func (tc *TestTSOCluster) GetPrimaryServer(keyspaceID, keyspaceGroupID uint32) *tso.Server { + for _, server := range tc.servers { + if server.IsKeyspaceServing(keyspaceID, keyspaceGroupID) { + return server + } + } + return nil +} + +// WaitForPrimaryServing waits for one of servers being elected to be the primary/leader of the given keyspace. +func (tc *TestTSOCluster) WaitForPrimaryServing(re *require.Assertions, keyspaceID, keyspaceGroupID uint32) *tso.Server { + var primary *tso.Server + testutil.Eventually(re, func() bool { + for _, server := range tc.servers { + if server.IsKeyspaceServing(keyspaceID, keyspaceGroupID) { + primary = server + return true + } + } + return false + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) + + return primary +} + +// WaitForDefaultPrimaryServing waits for one of servers being elected to be the primary/leader of the default keyspace. +func (tc *TestTSOCluster) WaitForDefaultPrimaryServing(re *require.Assertions) *tso.Server { + return tc.WaitForPrimaryServing(re, mcsutils.DefaultKeyspaceID, mcsutils.DefaultKeyspaceGroupID) +} + +// GetServer returns the TSO server by the given address. +func (tc *TestTSOCluster) GetServer(addr string) *tso.Server { + for srvAddr, server := range tc.servers { + if srvAddr == addr { + return server + } + } + return nil +} + +// GetServers returns all TSO servers. +func (tc *TestTSOCluster) GetServers() map[string]*tso.Server { + return tc.servers +} + +// GetKeyspaceGroupMember converts the TSO servers to KeyspaceGroupMember and returns. +func (tc *TestTSOCluster) GetKeyspaceGroupMember() (members []endpoint.KeyspaceGroupMember) { + for _, server := range tc.servers { + members = append(members, endpoint.KeyspaceGroupMember{ + Address: server.GetAddr(), + }) + } + return +} diff --git a/tests/mcs/discovery/register_test.go b/tests/integrations/mcs/discovery/register_test.go similarity index 89% rename from tests/mcs/discovery/register_test.go rename to tests/integrations/mcs/discovery/register_test.go index 6978f283026..27a5a937e4c 100644 --- a/tests/mcs/discovery/register_test.go +++ b/tests/integrations/mcs/discovery/register_test.go @@ -16,7 +16,9 @@ package register_test import ( "context" + "strconv" "testing" + "time" "github.com/stretchr/testify/suite" bs "github.com/tikv/pd/pkg/basicserver" @@ -25,7 +27,7 @@ import ( "github.com/tikv/pd/pkg/utils/tempurl" "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/tests" - "github.com/tikv/pd/tests/mcs" + "github.com/tikv/pd/tests/integrations/mcs" "go.uber.org/goleak" ) @@ -39,6 +41,7 @@ type serverRegisterTestSuite struct { cancel context.CancelFunc cluster *tests.TestCluster pdLeader *tests.TestServer + clusterID string backendEndpoints string } @@ -59,6 +62,7 @@ func (suite *serverRegisterTestSuite) SetupSuite() { leaderName := suite.cluster.WaitLeader() suite.pdLeader = suite.cluster.GetServer(leaderName) + suite.clusterID = strconv.FormatUint(suite.pdLeader.GetClusterID(), 10) suite.backendEndpoints = suite.pdLeader.GetAddr() } @@ -84,9 +88,9 @@ func (suite *serverRegisterTestSuite) checkServerRegister(serviceName string) { addr := s.GetAddr() client := suite.pdLeader.GetEtcdClient() - // test API server discovery - endpoints, err := discovery.Discover(client, serviceName) + + endpoints, err := discovery.Discover(client, suite.clusterID, serviceName) re.NoError(err) returnedEntry := &discovery.ServiceRegistryEntry{} returnedEntry.Deserialize([]byte(endpoints[0])) @@ -100,9 +104,12 @@ func (suite *serverRegisterTestSuite) checkServerRegister(serviceName string) { // test API server discovery after unregister cleanup() - endpoints, err = discovery.Discover(client, serviceName) + endpoints, err = discovery.Discover(client, suite.clusterID, serviceName) re.NoError(err) re.Empty(endpoints) + testutil.Eventually(re, func() bool { + return !s.IsServing() + }, testutil.WithWaitFor(3*time.Second), testutil.WithTickInterval(50*time.Millisecond)) } func (suite *serverRegisterTestSuite) TestServerPrimaryChange() { @@ -135,7 +142,7 @@ func (suite *serverRegisterTestSuite) checkServerPrimaryChange(serviceName strin expectedPrimary = mcs.WaitForPrimaryServing(suite.Require(), serverMap) // test API server discovery client := suite.pdLeader.GetEtcdClient() - endpoints, err := discovery.Discover(client, serviceName) + endpoints, err := discovery.Discover(client, suite.clusterID, serviceName) re.NoError(err) re.Len(endpoints, serverNum-1) diff --git a/tests/mcs/go.mod b/tests/integrations/mcs/go.mod similarity index 89% rename from tests/mcs/go.mod rename to tests/integrations/mcs/go.mod index c70695fdb9a..5cce44d160c 100644 --- a/tests/mcs/go.mod +++ b/tests/integrations/mcs/go.mod @@ -1,12 +1,20 @@ -module github.com/tikv/pd/tests/mcs +module github.com/tikv/pd/tests/integrations/mcs go 1.19 +replace ( + github.com/tikv/pd => ../../../ + github.com/tikv/pd/client => ../../../client +) + +// reset grpc and protobuf deps in order to import client and server at the same time +replace google.golang.org/grpc v1.51.0 => google.golang.org/grpc v1.26.0 + require ( github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 - github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba - github.com/pkg/errors v0.9.1 - github.com/stretchr/testify v1.8.1 + github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 + github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 + github.com/stretchr/testify v1.8.2 github.com/tikv/pd v0.0.0-00010101000000-000000000000 github.com/tikv/pd/client v0.0.0-00010101000000-000000000000 go.etcd.io/etcd v0.5.0-alpha.5.0.20220915004622-85b640cee793 @@ -21,7 +29,6 @@ require ( github.com/PuerkitoBio/purell v1.1.1 // indirect github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect github.com/ReneKroon/ttlcache/v2 v2.3.0 // indirect - github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect github.com/VividCortex/mysqlerr v1.0.0 // indirect github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502 // indirect github.com/aws/aws-sdk-go v1.35.3 // indirect @@ -98,10 +105,10 @@ require ( github.com/phf/go-queue v0.0.0-20170504031614-9abe38d0371d // indirect github.com/pingcap/errcode v0.3.0 // indirect github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c // indirect - github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 // indirect - github.com/pingcap/sysutil v0.0.0-20211208032423-041a72e5860d // indirect + github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21 // indirect github.com/pingcap/tidb-dashboard v0.0.0-20230209052558-a58fc2a7e924 // indirect github.com/pingcap/tipb v0.0.0-20220718022156-3e2483c20a9e // indirect + github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect github.com/prometheus/client_golang v1.11.0 // indirect @@ -111,7 +118,6 @@ require ( github.com/rs/cors v1.7.0 // indirect github.com/samber/lo v1.37.0 // indirect github.com/sasha-s/go-deadlock v0.2.0 // indirect - github.com/shirou/gopsutil v3.21.3+incompatible // indirect github.com/shirou/gopsutil/v3 v3.22.12 // indirect github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 // indirect github.com/sirupsen/logrus v1.6.0 // indirect @@ -120,8 +126,8 @@ require ( github.com/spf13/cobra v1.0.0 // indirect github.com/spf13/pflag v1.0.5 // indirect github.com/stretchr/objx v0.5.0 // indirect - github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 // indirect - github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba // indirect + github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 // indirect + github.com/swaggo/http-swagger v1.2.6 // indirect github.com/swaggo/swag v1.8.3 // indirect github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 // indirect github.com/tklauser/go-sysconf v0.3.11 // indirect @@ -142,14 +148,14 @@ require ( go.uber.org/zap v1.20.0 // indirect golang.org/x/crypto v0.1.0 // indirect golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a // indirect - golang.org/x/image v0.0.0-20200119044424-58c23975cae1 // indirect - golang.org/x/net v0.2.0 // indirect - golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 // indirect + golang.org/x/image v0.5.0 // indirect + golang.org/x/net v0.9.0 // indirect + golang.org/x/oauth2 v0.4.0 // indirect golang.org/x/sync v0.1.0 // indirect - golang.org/x/sys v0.3.0 // indirect - golang.org/x/text v0.4.0 // indirect + golang.org/x/sys v0.7.0 // indirect + golang.org/x/text v0.9.0 // indirect golang.org/x/time v0.1.0 // indirect - golang.org/x/tools v0.2.0 // indirect + golang.org/x/tools v0.6.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/genproto v0.0.0-20221202195650-67e5cbc046fd // indirect google.golang.org/protobuf v1.28.1 // indirect @@ -163,11 +169,3 @@ require ( moul.io/zapgorm2 v1.1.0 // indirect sigs.k8s.io/yaml v1.2.0 // indirect ) - -replace ( - github.com/tikv/pd => ../../ - github.com/tikv/pd/client => ../../client -) - -// reset grpc and protobuf deps in order to import client and server at the same time -replace google.golang.org/grpc v1.51.0 => google.golang.org/grpc v1.26.0 diff --git a/tests/mcs/go.sum b/tests/integrations/mcs/go.sum similarity index 92% rename from tests/mcs/go.sum rename to tests/integrations/mcs/go.sum index cfdba531df6..463940a5b79 100644 --- a/tests/mcs/go.sum +++ b/tests/integrations/mcs/go.sum @@ -7,19 +7,17 @@ github.com/KyleBanks/depth v1.2.1/go.mod h1:jzSb9d0L43HxTQfT+oSA1EEp2q+ne2uh6Xge github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= -github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= github.com/ReneKroon/ttlcache/v2 v2.3.0 h1:qZnUjRKIrbKHH6vF5T7Y9Izn5ObfTZfyYpGhvz2BKPo= github.com/ReneKroon/ttlcache/v2 v2.3.0/go.mod h1:zbo6Pv/28e21Z8CzzqgYRArQYGYtjONRxaAKGxzQvG4= -github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk= -github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg= github.com/VividCortex/mysqlerr v1.0.0 h1:5pZ2TZA+YnzPgzBfiUWGqWmKDVNBdrkf9g+DNe1Tiq8= github.com/VividCortex/mysqlerr v1.0.0/go.mod h1:xERx8E4tBhLvpjzdUyQiSfUxeMcATEQrflDAfXsqcAE= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502 h1:L8IbaI/W6h5Cwgh0n4zGeZpVK78r/jBf9ASurHo9+/o= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502/go.mod h1:pmnBM9bxWSiHvC/gSWunUIyDvGn33EkP2CUjxFKtTTM= +github.com/agiledragon/gomonkey/v2 v2.3.1/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= @@ -73,6 +71,7 @@ github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbp github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/corona10/goimagehash v1.0.2 h1:pUfB0LnsJASMPGEZLj7tGY251vF+qLGqOgEP4rUs6kA= github.com/corona10/goimagehash v1.0.2/go.mod h1:/l9umBhvcHQXVtQO1V6Gp1yD20STawkhRnnX0D1bvVI= +github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/creack/pty v1.1.11/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= @@ -105,37 +104,27 @@ github.com/gin-contrib/cors v1.4.0/go.mod h1:bs9pNM0x/UsmHPBWT2xZz9ROh8xYjYkiURU github.com/gin-contrib/gzip v0.0.1 h1:ezvKOL6jH+jlzdHNE4h9h8q8uMpDQjyl0NN0Jd7jozc= github.com/gin-contrib/gzip v0.0.1/go.mod h1:fGBJBCdt6qCZuCAOwWuFhBB4OOq9EFqlo5dEaFhhu5w= github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= -github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE= github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI= github.com/gin-gonic/gin v1.3.0/go.mod h1:7cKuhb5qV2ggCFctp2fJQ+ErvciLZrIeoOSOm6mUr7Y= -github.com/gin-gonic/gin v1.4.0/go.mod h1:OW2EZn3DO8Ln9oIKOvM++LBO+5UPHJJDH72/q/3rZdM= github.com/gin-gonic/gin v1.6.3/go.mod h1:75u5sXoLsGZoRN5Sgbi1eraJ4GU3++wFwWzhwvtwp4M= github.com/gin-gonic/gin v1.8.1 h1:4+fr/el88TOO3ewCmQr8cx/CtZ/umlIRIs5M4NTNjf8= github.com/gin-gonic/gin v1.8.1/go.mod h1:ji8BvRH1azfM+SYow9zQ6SZMvR8qOMZHmsCuWR9tTTk= -github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAUcHlgXVRs= -github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY= github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= -github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM= github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= -github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= -github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= -github.com/go-openapi/jsonreference v0.19.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= github.com/go-openapi/jsonreference v0.19.6 h1:UBIxjkht+AWIgYzCDSv2GN+E/togfwXUJFRTWhl2Jjs= github.com/go-openapi/jsonreference v0.19.6/go.mod h1:diGHMEHg2IqXZGKxqyvWdfWU/aim5Dprw5bqpKkTvns= -github.com/go-openapi/spec v0.19.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= github.com/go-openapi/spec v0.20.4 h1:O8hJrt0UMnhHcluhIdUgCLRWyM2x7QkBXRvOs7m+O1M= github.com/go-openapi/spec v0.20.4/go.mod h1:faYFR1CvsJZ0mNsmsphTMSoRrNV3TEDoAM7FOEWeq8I= -github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= github.com/go-openapi/swag v0.19.15 h1:D2NRCBzS9/pEY3gP9Nl8aDqGUcPFrwG2p+CNFrLyrCM= github.com/go-openapi/swag v0.19.15/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= @@ -217,7 +206,7 @@ github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= +github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/mux v1.7.4 h1:VuZ8uybHlWmqV03+zRzdwKL4tUnIp1MAQtp1mIFE1bc= github.com/gorilla/mux v1.7.4/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= @@ -280,6 +269,7 @@ github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/ github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= @@ -303,15 +293,12 @@ github.com/leodido/go-urn v1.2.1/go.mod h1:zt4jvISO2HfUBqxjfIshjdMTYS56ZS/qv49ic github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4= github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= -github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA= github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= -github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= -github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= @@ -349,6 +336,11 @@ github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1Cpa github.com/onsi/gomega v1.20.1 h1:PA/3qinGoukvymdIDV8pii6tiZgC8kbmJO6Z5+b002Q= github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs= github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc= +github.com/otiai10/copy v1.7.0/go.mod h1:rmRl6QPdJj6EiUqXQ/4Nn2lLXoNQjFCQbbNrxgc/t3U= +github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE= +github.com/otiai10/curr v1.0.0/go.mod h1:LskTG5wDwr8Rs+nNQ+1LlxRjAtTZZjtJW4rMXl6j4vs= +github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo= +github.com/otiai10/mint v1.3.3/go.mod h1:/yxELlJQ0ufhjUwhshSj+wFjZ78CnZ48/1wtmBH1OTc= github.com/pascaldekloe/name v0.0.0-20180628100202-0fd16699aae1/go.mod h1:eD5JxqMiuNYyFNmyY9rkJ/slN8y59oEu4Ei7F8OoKWQ= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml/v2 v2.0.1 h1:8e3L2cCQzLFi2CR4g7vGFuFxX7Jl1kKX8gW+iV0GUKU= @@ -357,8 +349,6 @@ github.com/petermattis/goid v0.0.0-20211229010228-4d14c490ee36 h1:64bxqeTEN0/xoE github.com/petermattis/goid v0.0.0-20211229010228-4d14c490ee36/go.mod h1:pxMtw7cyUw6B2bRH0ZBANSPg+AoSud1I1iyJHI69jH4= github.com/phf/go-queue v0.0.0-20170504031614-9abe38d0371d h1:U+PMnTlV2tu7RuMK5etusZG3Cf+rpow5hqQByeCzJ2g= github.com/phf/go-queue v0.0.0-20170504031614-9abe38d0371d/go.mod h1:lXfE4PvvTW5xOjO6Mba8zDPyw8M93B6AQ7frTGnMlA8= -github.com/pingcap/check v0.0.0-20190102082844-67f458068fc8/go.mod h1:B1+S9LNcuMyLH/4HMTViQOJevkGiik3wW2AN9zb2fNQ= -github.com/pingcap/check v0.0.0-20191107115940-caf2b9e6ccf4/go.mod h1:PYMCGwN0JHjoqGr3HrZoD+b8Tgx8bKnArhSq8YVzUMc= github.com/pingcap/check v0.0.0-20191216031241-8a5a85928f12 h1:rfD9v3+ppLPzoQBgZev0qYCpegrwyFx/BUpkApEiKdY= github.com/pingcap/errcode v0.3.0 h1:IF6LC/4+b1KNwrMlr2rBTUrojFPMexXBcDWZSpNwxjg= github.com/pingcap/errcode v0.3.0/go.mod h1:4b2X8xSqxIroj/IZ9MX/VGZhAwc11wB9wRIzHvz6SeM= @@ -370,14 +360,13 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ue github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 h1:C3N3itkduZXDZFh4N3vQ5HEtld3S+Y+StULhWVvumU0= github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00/go.mod h1:4qGtCB0QK0wBzKtFEGDhxXnSnbQApw1gc9siScUl8ew= github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba h1:7g2yM0llENlRqtjboBKFBJ8N9SE01hPDpKuTwxBLpLM= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba/go.mod h1:RjuuhxITxwATlt5adgTedg3ehKk01M03L1U4jNHdeeQ= -github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 h1:VXQ6Du/nKZ9IQnI9NWMzKbftWu8NV5pQkSLKIRzzGN4= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1/go.mod h1:guCyM5N+o+ru0TsoZ1hi9lDjUMs2sIBjW3ARTEpVbnk= github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= -github.com/pingcap/sysutil v0.0.0-20211208032423-041a72e5860d h1:k3/APKZjXOyJrFy8VyYwRlZhMelpD3qBLJNsw3bPl/g= -github.com/pingcap/sysutil v0.0.0-20211208032423-041a72e5860d/go.mod h1:7j18ezaWTao2LHOyMlsc2Dg1vW+mDY9dEbPzVyOlaeM= +github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21 h1:QV6jqlfOkh8hqvEAgwBZa+4bSgO0EeKC7s5c6Luam2I= +github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21/go.mod h1:QYnjfA95ZaMefyl1NO8oPtKeb8pYUdnDVhQgf+qdpjM= github.com/pingcap/tidb-dashboard v0.0.0-20230209052558-a58fc2a7e924 h1:49x3JR5zEYqjVqONKV9r/nrv0Rh5QU8ivIhktoLvP4g= github.com/pingcap/tidb-dashboard v0.0.0-20230209052558-a58fc2a7e924/go.mod h1:OUzFMMVjR1GKlf4LWLqza9QNKjCrYJ7stVn/3PN0djM= github.com/pingcap/tipb v0.0.0-20220718022156-3e2483c20a9e h1:FBaTXU8C3xgt/drM58VHxojHo/QoG1oPsgWTGvaSpO4= @@ -429,8 +418,7 @@ github.com/samber/lo v1.37.0/go.mod h1:9vaz2O4o8oOnK23pd2TrXufcbdbJIa3b6cstBWKpo github.com/sasha-s/go-deadlock v0.2.0 h1:lMqc+fUb7RrFS3gQLtoQsJ7/6TV/pAIFvBsqX73DK8Y= github.com/sasha-s/go-deadlock v0.2.0/go.mod h1:StQn567HiB1fF2yJ44N9au7wOhrPS3iZqiDbRupzT10= github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= -github.com/shirou/gopsutil v3.21.3+incompatible h1:uenXGGa8ESCQq+dbgtl916dmg6PSAz2cXov0uORQ9v8= -github.com/shirou/gopsutil v3.21.3+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA= +github.com/shirou/gopsutil/v3 v3.21.12/go.mod h1:BToYZVTlSVlfazpDDYFnsVZLaoRG+g8ufT6fPQLdJzA= github.com/shirou/gopsutil/v3 v3.22.12 h1:oG0ns6poeUSxf78JtOsfygNWuEHYYz8hnnNg7P04TJs= github.com/shirou/gopsutil/v3 v3.22.12/go.mod h1:Xd7P1kwZcp5VW52+9XsirIKd/BROzbb2wdX3Kqlz9uI= github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 h1:mj/nMDAwTBiaCqMEs4cYCqF7pO6Np7vhy1D1wcQGz+E= @@ -442,6 +430,8 @@ github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072 h1:Txo4SXVJq/OgEjwgkWoxkMoTjGlcrgsQE/XSghjmu0w= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072/go.mod h1:+4nWMF0+CqEcU74SnX2NxaGqZ8zX4pcQ8Jcs77DbX5A= +github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= +github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= @@ -469,15 +459,14 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 h1:PyYN9JH5jY9j6av01SpfRMb+1DWg/i3MbGOKPxJ2wjM= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14/go.mod h1:gxQT6pBGRuIGunNf/+tSOB5OHvguWi8Tbt82WOkf35E= -github.com/swaggo/gin-swagger v1.2.0/go.mod h1:qlH2+W7zXGZkczuL+r2nEBR2JTT+/lX05Nn6vPhc7OI= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba h1:lUPlXKqgbqT2SVg2Y+eT9mu5wbqMnG+i/+Q9nK7C0Rs= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba/go.mod h1:O1lAbCgAAX/KZ80LM/OXwtWFI/5TvZlwxSg8Cq08PV0= -github.com/swaggo/swag v1.5.1/go.mod h1:1Bl9F/ZBpVWh22nY0zmYyASPO1lI/zIwRDrpZU+tv8Y= -github.com/swaggo/swag v1.6.3/go.mod h1:wcc83tB4Mb2aNiL/HP4MFeQdpHUrca+Rp/DRNgWAUio= +github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= +github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 h1:+iNTcqQJy0OZ5jk6a5NLib47eqXK8uYcPX+O4+cBpEM= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2/go.mod h1:lKJPbtWzJ9JhsTN1k1gZgleJWY/cqq0psdoMmaThG3w= +github.com/swaggo/http-swagger v1.2.6 h1:ihTjChUoSRMpFMjWw+0AkL1Ti4r6v8pCgVYLmQVRlRw= +github.com/swaggo/http-swagger v1.2.6/go.mod h1:CcoICgY3yVDk2u1LQUCMHbAj0fjlxIX+873psXlIKNA= +github.com/swaggo/swag v1.7.9/go.mod h1:gZ+TJ2w/Ve1RwQsA2IRoSOTidHz6DX+PIG8GWvbnoLU= github.com/swaggo/swag v1.8.3 h1:3pZSSCQ//gAH88lfmxM3Cd1+JCsxV8Md6f36b9hrZ5s= github.com/swaggo/swag v1.8.3/go.mod h1:jMLeXOOmYyjk8PvHTsXBdrubsNd9gUJTTCzL5iBnseg= github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 h1:1oFLiOyVl+W7bnBzGhf7BbIv9loSFQcieWWYIjLqcAw= @@ -488,27 +477,26 @@ github.com/tidwall/match v1.0.1 h1:PnKP62LPNxHKTwvHHZZzdOAOCtsJTjo6dZLCwpKm5xc= github.com/tidwall/match v1.0.1/go.mod h1:LujAq0jyVjBy028G1WhWfIzbpQfMO8bBZ6Tyb0+pL9E= github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4= github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= -github.com/tklauser/go-sysconf v0.3.4/go.mod h1:Cl2c8ZRWfHD5IrfHo9VN+FX9kCFjIOyVklgXycLB6ek= +github.com/tklauser/go-sysconf v0.3.9/go.mod h1:11DU/5sG7UexIrp/O6g35hrWzu0JxlwQ3LSFUzyeuhs= github.com/tklauser/go-sysconf v0.3.11 h1:89WgdJhk5SNwJfu+GKyYveZ4IaJ7xAkecBo+KdJV0CM= github.com/tklauser/go-sysconf v0.3.11/go.mod h1:GqXfhXY3kiPa0nAXPDIQIWzJbMCB7AmcWpGR8lSZfqI= -github.com/tklauser/numcpus v0.2.1/go.mod h1:9aU+wOc6WjUIZEwWMP62PL/41d65P+iks1gBkr4QyP8= +github.com/tklauser/numcpus v0.3.0/go.mod h1:yFGUr7TUHQRAhyqBcEg0Ge34zDBAsIvJJcyE6boqnA8= github.com/tklauser/numcpus v0.6.0 h1:kebhY2Qt+3U6RNK7UqpYNA+tJ23IBEGKkB7JQBfDYms= github.com/tklauser/numcpus v0.6.0/go.mod h1:FEZLMke0lhOUG6w2JadTzp0a+Nl8PF/GFkQ5UVIcaL4= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966 h1:j6JEOq5QWFker+d7mFQYOhjTZonQ7YkLTHm56dbn+yM= github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= -github.com/ugorji/go v1.1.5-pre/go.mod h1:FwP/aQVg39TXzItUBMwnWp9T9gPQnXw4Poh4/oBQZ/0= github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw= github.com/ugorji/go v1.2.7/go.mod h1:nF9osbDWLy6bDVv/Rtoh6QgnvNDpmCalQV5urGCCS6M= github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= -github.com/ugorji/go/codec v1.1.5-pre/go.mod h1:tULtS6Gy1AE1yCENaw4Vb//HLH5njI2tfCQDUqRd8fI= github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY= github.com/ugorji/go/codec v1.2.7 h1:YPXUKf7fYbp/y8xloBqZOw2qaVggbfwMlI8WM3wZUJ0= github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95kRgeqEY= github.com/unrolled/render v1.0.1 h1:VDDnQQVfBMsOsp3VaCJszSO0nkBIVEYoPWeRThk9spY= github.com/unrolled/render v1.0.1/go.mod h1:gN9T0NhL4Bfbwu8ann7Ry/TGHYfosul+J0obPf6NBdM= github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA= +github.com/urfave/cli/v2 v2.3.0/go.mod h1:LJmUH05zAU44vOAcrfzZQKsZbVcdbOG8rtL3/XcUArI= github.com/urfave/negroni v0.3.0 h1:PaXOb61mWeZJxc1Ji2xJjpVg9QfPo0rrB+lHyBxGNSU= github.com/urfave/negroni v0.3.0/go.mod h1:Meg73S6kFm/4PpbYdq35yYWoCZ9mS/YSx+lKnmiohz4= github.com/vmihailenco/msgpack/v5 v5.3.5 h1:5gO0H1iULLWGhs2H5tbAHIZTV8/cYafcFOr9znI5mJU= @@ -521,6 +509,8 @@ github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1: github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg= github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= @@ -568,14 +558,16 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20200204104054-c9f3fb736b72/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU= golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a h1:tlXy25amD5A7gOfbXdqCGN5k8ESEed/Ee1E5RcrYnqU= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= -golang.org/x/image v0.0.0-20200119044424-58c23975cae1 h1:5h3ngYt7+vXCDZCup/HkCQgW5XwmSvR/nA2JmJ0RErg= golang.org/x/image v0.0.0-20200119044424-58c23975cae1/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= +golang.org/x/image v0.5.0 h1:5JMiNunQeQw++mMOz48/ISeNu3Iweh/JaZU8ZLqHRrI= +golang.org/x/image v0.5.0/go.mod h1:FVC7BI/5Ym8R25iw5OLsgshdUBbT1h5jZTpA+mvAdZ4= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= @@ -587,7 +579,8 @@ golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.6.0 h1:b9gGHsz9/HhJ3HF5DHQytPpuwocVTChQJK3AvoLRD5I= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -598,10 +591,8 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= -golang.org/x/net v0.0.0-20190611141213-3f473d35a33a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= @@ -613,14 +604,16 @@ golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwY golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM= +golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.2.0 h1:sZfSu1wtKLGlWI4ZZayP0ck9Y73K1ynO6gqzTdBVdPU= -golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 h1:nt+Q6cXKz4MosCSpnbMtqiQ8Oz0pxTef2B4Vca2lvfk= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg= +golang.org/x/oauth2 v0.4.0 h1:NF0gk8LVPg1Ml7SSbGyySuoxdsXitj7TvgvuRxIMc/M= +golang.org/x/oauth2 v0.4.0/go.mod h1:RznEsdpjGAINPTOF0UH/t+xJ75L18YO3Ho6Pyn+uRec= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -629,6 +622,7 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180810173357-98c5dad5d1a0/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -639,10 +633,8 @@ golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181228144115-9a3f9b0469bb/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190610200419-93c9922d18ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -654,7 +646,6 @@ golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210217105451-b926d437f341/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210420072515-93ed5bcd2bfe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -663,19 +654,27 @@ golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210816074244-15123e1e1f71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211013075003-97ac67df715c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ= golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg= -golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.1.0 h1:xYY+Bajn2a7VBmTM5GikTmnK8ZuX8YgnQCqZpbBNtmA= @@ -685,15 +684,13 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190524210228-3d17549cdc6b/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.0.0-20190606050223-4d9ae51c2468/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190611222205-d73e1c7e250b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191030062658-86caa796c7ab/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191107010934-f79515f33823/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191108193012-7d206e10da11/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191114200427-caa0b0f7d508/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= @@ -703,8 +700,10 @@ golang.org/x/tools v0.0.0-20201125231158-b5590deeca9b/go.mod h1:emZCQorbCU4vsT4f golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210112230658-8b4aab62c064/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.2.0 h1:G6AHpWxTMGY1KyEYoAQ5WTtIekUUvDNjan3ugu60JvE= -golang.org/x/tools v0.2.0/go.mod h1:y4OqIKeOV/fWJetJ8bXPU1sEVniLMIyDAZWeHdV+NTA= +golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= diff --git a/tests/integrations/mcs/keyspace/tso_keyspace_group_test.go b/tests/integrations/mcs/keyspace/tso_keyspace_group_test.go new file mode 100644 index 00000000000..6d2c51e1dd0 --- /dev/null +++ b/tests/integrations/mcs/keyspace/tso_keyspace_group_test.go @@ -0,0 +1,360 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package keyspace_test + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "testing" + "time" + + "github.com/stretchr/testify/suite" + bs "github.com/tikv/pd/pkg/basicserver" + "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/utils/tempurl" + "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/server/apiv2/handlers" + "github.com/tikv/pd/tests" + "github.com/tikv/pd/tests/integrations/mcs" +) + +const ( + keyspaceGroupsPrefix = "/pd/api/v2/tso/keyspace-groups" +) + +type keyspaceGroupTestSuite struct { + suite.Suite + ctx context.Context + cleanupFunc testutil.CleanupFunc + cluster *tests.TestCluster + server *tests.TestServer + backendEndpoints string + dialClient *http.Client +} + +func TestKeyspaceGroupTestSuite(t *testing.T) { + suite.Run(t, new(keyspaceGroupTestSuite)) +} + +func (suite *keyspaceGroupTestSuite) SetupTest() { + ctx, cancel := context.WithCancel(context.Background()) + suite.ctx = ctx + cluster, err := tests.NewTestAPICluster(suite.ctx, 1) + suite.cluster = cluster + suite.NoError(err) + suite.NoError(cluster.RunInitialServers()) + suite.NotEmpty(cluster.WaitLeader()) + suite.server = cluster.GetServer(cluster.GetLeader()) + suite.NoError(suite.server.BootstrapCluster()) + suite.backendEndpoints = suite.server.GetAddr() + suite.dialClient = &http.Client{ + Transport: &http.Transport{ + DisableKeepAlives: true, + }, + } + suite.cleanupFunc = func() { + cancel() + } +} + +func (suite *keyspaceGroupTestSuite) TearDownTest() { + suite.cleanupFunc() + suite.cluster.Destroy() +} + +func (suite *keyspaceGroupTestSuite) TestAllocNodesUpdate() { + // add three nodes. + nodes := make(map[string]bs.Server) + for i := 0; i < utils.KeyspaceGroupDefaultReplicaCount+1; i++ { + s, cleanup := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, tempurl.Alloc()) + defer cleanup() + nodes[s.GetAddr()] = s + } + mcs.WaitForPrimaryServing(suite.Require(), nodes) + + // create a keyspace group. + kgs := &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Standard.String(), + }, + }} + code := suite.tryCreateKeyspaceGroup(kgs) + suite.Equal(http.StatusOK, code) + + // alloc nodes for the keyspace group. + id := 1 + params := &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount, + } + got, code := suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusOK, code) + suite.Equal(utils.KeyspaceGroupDefaultReplicaCount, len(got)) + oldMembers := make(map[string]struct{}) + for _, member := range got { + suite.Contains(nodes, member.Address) + oldMembers[member.Address] = struct{}{} + } + + // alloc node update to 3. + params.Replica = utils.KeyspaceGroupDefaultReplicaCount + 1 + got, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusOK, code) + suite.Equal(params.Replica, len(got)) + newMembers := make(map[string]struct{}) + for _, member := range got { + suite.Contains(nodes, member.Address) + newMembers[member.Address] = struct{}{} + } + for member := range oldMembers { + // old members should be in new members. + suite.Contains(newMembers, member) + } +} + +func (suite *keyspaceGroupTestSuite) TestAllocReplica() { + nodes := make(map[string]bs.Server) + for i := 0; i < utils.KeyspaceGroupDefaultReplicaCount; i++ { + s, cleanup := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, tempurl.Alloc()) + defer cleanup() + nodes[s.GetAddr()] = s + } + mcs.WaitForPrimaryServing(suite.Require(), nodes) + + // miss replica. + id := 1 + params := &handlers.AllocNodesForKeyspaceGroupParams{} + got, code := suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + suite.Empty(got) + + // replica is less than default replica. + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount - 1, + } + _, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // there is no any keyspace group. + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount, + } + _, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // create a keyspace group. + kgs := &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(id), + UserKind: endpoint.Standard.String(), + }, + }} + code = suite.tryCreateKeyspaceGroup(kgs) + suite.Equal(http.StatusOK, code) + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount, + } + got, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusOK, code) + for _, member := range got { + suite.Contains(nodes, member.Address) + } + + // the keyspace group is exist, but the replica is more than the num of nodes. + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount + 1, + } + _, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // the keyspace group is exist, the new replica is more than the old replica. + s2, cleanup2 := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, tempurl.Alloc()) + defer cleanup2() + nodes[s2.GetAddr()] = s2 + mcs.WaitForPrimaryServing(suite.Require(), nodes) + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount + 1, + } + got, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusOK, code) + for _, member := range got { + suite.Contains(nodes, member.Address) + } + + // the keyspace group is exist, the new replica is equal to the old replica. + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount + 1, + } + _, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // the keyspace group is exist, the new replica is less than the old replica. + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount, + } + _, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // the keyspace group is not exist. + id = 2 + params = &handlers.AllocNodesForKeyspaceGroupParams{ + Replica: utils.KeyspaceGroupDefaultReplicaCount, + } + _, code = suite.tryAllocNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) +} + +func (suite *keyspaceGroupTestSuite) TestSetNodes() { + nodes := make(map[string]bs.Server) + nodesList := []string{} + for i := 0; i < utils.KeyspaceGroupDefaultReplicaCount; i++ { + s, cleanup := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, tempurl.Alloc()) + defer cleanup() + nodes[s.GetAddr()] = s + nodesList = append(nodesList, s.GetAddr()) + } + mcs.WaitForPrimaryServing(suite.Require(), nodes) + + // the keyspace group is not exist. + id := 1 + params := &handlers.SetNodesForKeyspaceGroupParams{ + Nodes: nodesList, + } + _, code := suite.trySetNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // the keyspace group is exist. + kgs := &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(id), + UserKind: endpoint.Standard.String(), + }, + }} + code = suite.tryCreateKeyspaceGroup(kgs) + suite.Equal(http.StatusOK, code) + params = &handlers.SetNodesForKeyspaceGroupParams{ + Nodes: nodesList, + } + kg, code := suite.trySetNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusOK, code) + suite.Len(kg.Members, 2) + for _, member := range kg.Members { + suite.Contains(nodes, member.Address) + } + + // the keyspace group is exist, but the nodes is not exist. + params = &handlers.SetNodesForKeyspaceGroupParams{ + Nodes: append(nodesList, "pingcap.com:2379"), + } + _, code = suite.trySetNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // the keyspace group is exist, but the count of nodes is less than the default replica. + params = &handlers.SetNodesForKeyspaceGroupParams{ + Nodes: []string{nodesList[0]}, + } + _, code = suite.trySetNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) + + // the keyspace group is not exist. + id = 2 + params = &handlers.SetNodesForKeyspaceGroupParams{ + Nodes: nodesList, + } + _, code = suite.trySetNodesForKeyspaceGroup(id, params) + suite.Equal(http.StatusBadRequest, code) +} + +func (suite *keyspaceGroupTestSuite) TestDefaultKeyspaceGroup() { + nodes := make(map[string]bs.Server) + for i := 0; i < utils.KeyspaceGroupDefaultReplicaCount; i++ { + s, cleanup := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, tempurl.Alloc()) + defer cleanup() + nodes[s.GetAddr()] = s + } + mcs.WaitForPrimaryServing(suite.Require(), nodes) + + // the default keyspace group is exist. + time.Sleep(2 * time.Second) + kg, code := suite.tryGetKeyspaceGroup(utils.DefaultKeyspaceGroupID) + suite.Equal(http.StatusOK, code) + suite.Equal(utils.DefaultKeyspaceGroupID, kg.ID) + suite.Len(kg.Members, utils.KeyspaceGroupDefaultReplicaCount) + for _, member := range kg.Members { + suite.Contains(nodes, member.Address) + } +} + +func (suite *keyspaceGroupTestSuite) tryAllocNodesForKeyspaceGroup(id int, request *handlers.AllocNodesForKeyspaceGroupParams) ([]endpoint.KeyspaceGroupMember, int) { + data, err := json.Marshal(request) + suite.NoError(err) + httpReq, err := http.NewRequest(http.MethodPost, suite.server.GetAddr()+keyspaceGroupsPrefix+fmt.Sprintf("/%d/alloc", id), bytes.NewBuffer(data)) + suite.NoError(err) + resp, err := suite.dialClient.Do(httpReq) + suite.NoError(err) + defer resp.Body.Close() + nodes := make([]endpoint.KeyspaceGroupMember, 0) + if resp.StatusCode == http.StatusOK { + bodyBytes, err := io.ReadAll(resp.Body) + suite.NoError(err) + suite.NoError(json.Unmarshal(bodyBytes, &nodes)) + } + return nodes, resp.StatusCode +} + +func (suite *keyspaceGroupTestSuite) tryCreateKeyspaceGroup(request *handlers.CreateKeyspaceGroupParams) int { + data, err := json.Marshal(request) + suite.NoError(err) + httpReq, err := http.NewRequest(http.MethodPost, suite.server.GetAddr()+keyspaceGroupsPrefix, bytes.NewBuffer(data)) + suite.NoError(err) + resp, err := suite.dialClient.Do(httpReq) + suite.NoError(err) + defer resp.Body.Close() + return resp.StatusCode +} + +func (suite *keyspaceGroupTestSuite) tryGetKeyspaceGroup(id uint32) (*endpoint.KeyspaceGroup, int) { + httpReq, err := http.NewRequest(http.MethodGet, suite.server.GetAddr()+keyspaceGroupsPrefix+fmt.Sprintf("/%d", id), nil) + suite.NoError(err) + resp, err := suite.dialClient.Do(httpReq) + suite.NoError(err) + defer resp.Body.Close() + kg := &endpoint.KeyspaceGroup{} + if resp.StatusCode == http.StatusOK { + bodyBytes, err := io.ReadAll(resp.Body) + suite.NoError(err) + suite.NoError(json.Unmarshal(bodyBytes, kg)) + } + return kg, resp.StatusCode +} + +func (suite *keyspaceGroupTestSuite) trySetNodesForKeyspaceGroup(id int, request *handlers.SetNodesForKeyspaceGroupParams) (*endpoint.KeyspaceGroup, int) { + data, err := json.Marshal(request) + suite.NoError(err) + httpReq, err := http.NewRequest(http.MethodPost, suite.server.GetAddr()+keyspaceGroupsPrefix+fmt.Sprintf("/%d/nodes", id), bytes.NewBuffer(data)) + suite.NoError(err) + resp, err := suite.dialClient.Do(httpReq) + suite.NoError(err) + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, resp.StatusCode + } + return suite.tryGetKeyspaceGroup(uint32(id)) +} diff --git a/tests/mcs/resource_manager/resource_manager_test.go b/tests/integrations/mcs/resource_manager/resource_manager_test.go similarity index 100% rename from tests/mcs/resource_manager/resource_manager_test.go rename to tests/integrations/mcs/resource_manager/resource_manager_test.go diff --git a/tests/mcs/resource_manager/server_test.go b/tests/integrations/mcs/resource_manager/server_test.go similarity index 86% rename from tests/mcs/resource_manager/server_test.go rename to tests/integrations/mcs/resource_manager/server_test.go index f3aaabc6af7..5d94c2eb970 100644 --- a/tests/mcs/resource_manager/server_test.go +++ b/tests/integrations/mcs/resource_manager/server_test.go @@ -27,7 +27,7 @@ import ( "github.com/tikv/pd/client/grpcutil" "github.com/tikv/pd/pkg/utils/tempurl" "github.com/tikv/pd/tests" - "github.com/tikv/pd/tests/mcs" + "github.com/tikv/pd/tests/integrations/mcs" ) func TestResourceManagerServer(t *testing.T) { @@ -92,14 +92,15 @@ func TestResourceManagerServer(t *testing.T) { re.Equal("{\"name\":\"pingcap\",\"mode\":1,\"r_u_settings\":{\"r_u\":{\"state\":{\"initialized\":false}}}}", string(respString)) } - // Test metrics handler - { - resp, err := http.Get(addr + "/metrics") - re.NoError(err) - defer resp.Body.Close() - re.Equal(http.StatusOK, resp.StatusCode) - respBytes, err := io.ReadAll(resp.Body) - re.NoError(err) - re.Contains(string(respBytes), "resource_manager_server_info") - } + // Notes: We will skip resource manager test. + // // Test metrics handler + // { + // resp, err := http.Get(addr + "/metrics") + // re.NoError(err) + // defer resp.Body.Close() + // re.Equal(http.StatusOK, resp.StatusCode) + // respBytes, err := io.ReadAll(resp.Body) + // re.NoError(err) + // re.Contains(string(respBytes), "resource_manager_server_info") + // } } diff --git a/tests/integrations/mcs/testutil.go b/tests/integrations/mcs/testutil.go new file mode 100644 index 00000000000..9a3108067c9 --- /dev/null +++ b/tests/integrations/mcs/testutil.go @@ -0,0 +1,218 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package mcs + +import ( + "context" + "os" + "sync" + "time" + + "github.com/pingcap/log" + "github.com/stretchr/testify/require" + pd "github.com/tikv/pd/client" + bs "github.com/tikv/pd/pkg/basicserver" + rm "github.com/tikv/pd/pkg/mcs/resource_manager/server" + tso "github.com/tikv/pd/pkg/mcs/tso/server" + "github.com/tikv/pd/pkg/utils/logutil" + "github.com/tikv/pd/pkg/utils/testutil" + "github.com/tikv/pd/pkg/utils/tsoutil" +) + +var once sync.Once + +// InitLogger initializes the logger for test. +func InitLogger(cfg *tso.Config) (err error) { + once.Do(func() { + // Setup the logger. + err = logutil.SetupLogger(cfg.Log, &cfg.Logger, &cfg.LogProps, cfg.Security.RedactInfoLog) + if err != nil { + return + } + log.ReplaceGlobals(cfg.Logger, cfg.LogProps) + // Flushing any buffered log entries. + log.Sync() + }) + return err +} + +// SetupClientWithAPIContext creates a TSO client with api context name for test. +func SetupClientWithAPIContext( + ctx context.Context, re *require.Assertions, apiCtx pd.APIContext, endpoints []string, opts ...pd.ClientOption, +) pd.Client { + cli, err := pd.NewClientWithAPIContext(ctx, apiCtx, endpoints, pd.SecurityOption{}, opts...) + re.NoError(err) + return cli +} + +// SetupClientWithKeyspaceID creates a TSO client with the given keyspace id for test. +func SetupClientWithKeyspaceID( + ctx context.Context, re *require.Assertions, + keyspaceID uint32, endpoints []string, opts ...pd.ClientOption, +) pd.Client { + cli, err := pd.NewClientWithKeyspace(ctx, keyspaceID, endpoints, pd.SecurityOption{}, opts...) + re.NoError(err) + return cli +} + +// StartSingleResourceManagerTestServer creates and starts a resource manager server with default config for testing. +func StartSingleResourceManagerTestServer(ctx context.Context, re *require.Assertions, backendEndpoints, listenAddrs string) (*rm.Server, func()) { + cfg := rm.NewConfig() + cfg.BackendEndpoints = backendEndpoints + cfg.ListenAddr = listenAddrs + cfg, err := rm.GenerateConfig(cfg) + re.NoError(err) + + s, cleanup, err := rm.NewTestServer(ctx, re, cfg) + re.NoError(err) + testutil.Eventually(re, func() bool { + return !s.IsClosed() + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) + + return s, cleanup +} + +// StartSingleTSOTestServerWithoutCheck creates and starts a tso server with default config for testing. +func StartSingleTSOTestServerWithoutCheck(ctx context.Context, re *require.Assertions, backendEndpoints, listenAddrs string) (*tso.Server, func(), error) { + cfg := tso.NewConfig() + cfg.BackendEndpoints = backendEndpoints + cfg.ListenAddr = listenAddrs + cfg, err := tso.GenerateConfig(cfg) + re.NoError(err) + // Setup the logger. + err = InitLogger(cfg) + re.NoError(err) + return NewTSOTestServer(ctx, cfg) +} + +// StartSingleTSOTestServer creates and starts a tso server with default config for testing. +func StartSingleTSOTestServer(ctx context.Context, re *require.Assertions, backendEndpoints, listenAddrs string) (*tso.Server, func()) { + s, cleanup, err := StartSingleTSOTestServerWithoutCheck(ctx, re, backendEndpoints, listenAddrs) + re.NoError(err) + testutil.Eventually(re, func() bool { + return !s.IsClosed() + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) + + return s, cleanup +} + +// NewTSOTestServer creates a tso server with given config for testing. +func NewTSOTestServer(ctx context.Context, cfg *tso.Config) (*tso.Server, testutil.CleanupFunc, error) { + s := tso.CreateServer(ctx, cfg) + if err := s.Run(); err != nil { + return nil, nil, err + } + cleanup := func() { + s.Close() + os.RemoveAll(cfg.DataDir) + } + return s, cleanup, nil +} + +// WaitForPrimaryServing waits for one of servers being elected to be the primary/leader +func WaitForPrimaryServing(re *require.Assertions, serverMap map[string]bs.Server) string { + var primary string + testutil.Eventually(re, func() bool { + for name, s := range serverMap { + if s.IsServing() { + primary = name + return true + } + } + return false + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) + + return primary +} + +// WaitForTSOServiceAvailable waits for the pd client being served by the tso server side +func WaitForTSOServiceAvailable( + ctx context.Context, re *require.Assertions, client pd.Client, +) { + testutil.Eventually(re, func() bool { + _, _, err := client.GetTS(ctx) + return err == nil + }) +} + +// CheckMultiKeyspacesTSO checks the correctness of TSO for multiple keyspaces. +func CheckMultiKeyspacesTSO( + ctx context.Context, re *require.Assertions, + clients []pd.Client, parallelAct func(), +) { + ctx, cancel := context.WithCancel(ctx) + wg := sync.WaitGroup{} + wg.Add(len(clients)) + + for _, client := range clients { + go func(cli pd.Client) { + defer wg.Done() + var ts, lastTS uint64 + for { + select { + case <-ctx.Done(): + // Make sure the lastTS is not empty + re.NotEmpty(lastTS) + return + default: + } + physical, logical, err := cli.GetTS(ctx) + // omit the error check since there are many kinds of errors + if err != nil { + continue + } + ts = tsoutil.ComposeTS(physical, logical) + re.Less(lastTS, ts) + lastTS = ts + } + }(client) + } + + wg.Add(1) + go func() { + defer wg.Done() + parallelAct() + cancel() + }() + + wg.Wait() +} + +// WaitForMultiKeyspacesTSOAvailable waits for the given keyspaces being served by the tso server side +func WaitForMultiKeyspacesTSOAvailable( + ctx context.Context, re *require.Assertions, + keyspaceIDs []uint32, backendEndpoints []string, +) []pd.Client { + wg := sync.WaitGroup{} + wg.Add(len(keyspaceIDs)) + + clients := make([]pd.Client, 0, len(keyspaceIDs)) + for _, keyspaceID := range keyspaceIDs { + cli := SetupClientWithKeyspaceID(ctx, re, keyspaceID, backendEndpoints) + re.NotNil(cli) + clients = append(clients, cli) + + go func() { + defer wg.Done() + testutil.Eventually(re, func() bool { + _, _, err := cli.GetTS(ctx) + return err == nil + }) + }() + } + + wg.Wait() + return clients +} diff --git a/tests/integrations/mcs/tso/api_test.go b/tests/integrations/mcs/tso/api_test.go new file mode 100644 index 00000000000..e5dbcd7998c --- /dev/null +++ b/tests/integrations/mcs/tso/api_test.go @@ -0,0 +1,106 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tso + +import ( + "context" + "encoding/json" + "io" + "net/http" + "testing" + + "github.com/stretchr/testify/require" + "github.com/stretchr/testify/suite" + tso "github.com/tikv/pd/pkg/mcs/tso/server" + apis "github.com/tikv/pd/pkg/mcs/tso/server/apis/v1" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/tests" + "github.com/tikv/pd/tests/integrations/mcs" +) + +const ( + tsoKeyspaceGroupsPrefix = "/tso/api/v1/keyspace-groups" +) + +// dialClient used to dial http request. +var dialClient = &http.Client{ + Transport: &http.Transport{ + DisableKeepAlives: true, + }, +} + +type tsoAPITestSuite struct { + suite.Suite + ctx context.Context + cancel context.CancelFunc + pdCluster *tests.TestCluster + tsoCluster *mcs.TestTSOCluster +} + +func TestTSOAPI(t *testing.T) { + suite.Run(t, new(tsoAPITestSuite)) +} + +func (suite *tsoAPITestSuite) SetupTest() { + re := suite.Require() + + var err error + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + suite.pdCluster, err = tests.NewTestAPICluster(suite.ctx, 1) + re.NoError(err) + err = suite.pdCluster.RunInitialServers() + re.NoError(err) + leaderName := suite.pdCluster.WaitLeader() + pdLeaderServer := suite.pdCluster.GetServer(leaderName) + re.NoError(pdLeaderServer.BootstrapCluster()) + suite.tsoCluster, err = mcs.NewTestTSOCluster(suite.ctx, 1, pdLeaderServer.GetAddr()) + re.NoError(err) +} + +func (suite *tsoAPITestSuite) TearDownTest() { + suite.cancel() + suite.tsoCluster.Destroy() + suite.pdCluster.Destroy() +} + +func (suite *tsoAPITestSuite) TestGetKeyspaceGroupMembers() { + re := suite.Require() + + primary := suite.tsoCluster.WaitForDefaultPrimaryServing(re) + re.NotNil(primary) + members := mustGetKeyspaceGroupMembers(re, primary) + re.Len(members, 1) + defaultGroupMember := members[mcsutils.DefaultKeyspaceGroupID] + re.NotNil(defaultGroupMember) + re.Equal(mcsutils.DefaultKeyspaceGroupID, defaultGroupMember.Group.ID) + re.True(defaultGroupMember.IsPrimary) + primaryMember, err := primary.GetMember(mcsutils.DefaultKeyspaceID, mcsutils.DefaultKeyspaceGroupID) + re.NoError(err) + re.Equal(primaryMember.GetLeaderID(), defaultGroupMember.PrimaryID) +} + +func mustGetKeyspaceGroupMembers(re *require.Assertions, server *tso.Server) map[uint32]*apis.KeyspaceGroupMember { + httpReq, err := http.NewRequest(http.MethodGet, server.GetAddr()+tsoKeyspaceGroupsPrefix+"/members", nil) + re.NoError(err) + httpResp, err := dialClient.Do(httpReq) + re.NoError(err) + defer httpResp.Body.Close() + data, err := io.ReadAll(httpResp.Body) + re.NoError(err) + re.Equal(http.StatusOK, httpResp.StatusCode, string(data)) + var resp map[uint32]*apis.KeyspaceGroupMember + re.NoError(json.Unmarshal(data, &resp)) + return resp +} diff --git a/tests/integrations/mcs/tso/keyspace_group_manager_test.go b/tests/integrations/mcs/tso/keyspace_group_manager_test.go new file mode 100644 index 00000000000..4b650bf1e25 --- /dev/null +++ b/tests/integrations/mcs/tso/keyspace_group_manager_test.go @@ -0,0 +1,427 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tso + +import ( + "context" + "fmt" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/pingcap/kvproto/pkg/pdpb" + "github.com/stretchr/testify/require" + "github.com/stretchr/testify/suite" + pd "github.com/tikv/pd/client" + "github.com/tikv/pd/client/testutil" + "github.com/tikv/pd/pkg/election" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/member" + "github.com/tikv/pd/pkg/storage/endpoint" + tsopkg "github.com/tikv/pd/pkg/tso" + "github.com/tikv/pd/pkg/utils/tsoutil" + "github.com/tikv/pd/server/apiv2/handlers" + "github.com/tikv/pd/tests" + "github.com/tikv/pd/tests/integrations/mcs" + handlersutil "github.com/tikv/pd/tests/server/apiv2/handlers" +) + +type tsoKeyspaceGroupManagerTestSuite struct { + suite.Suite + + ctx context.Context + cancel context.CancelFunc + + // The PD cluster. + cluster *tests.TestCluster + // pdLeaderServer is the leader server of the PD cluster. + pdLeaderServer *tests.TestServer + // tsoCluster is the TSO service cluster. + tsoCluster *mcs.TestTSOCluster +} + +func TestTSOKeyspaceGroupManager(t *testing.T) { + suite.Run(t, &tsoKeyspaceGroupManagerTestSuite{}) +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) SetupSuite() { + re := suite.Require() + + var err error + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + suite.cluster, err = tests.NewTestAPICluster(suite.ctx, 1) + re.NoError(err) + err = suite.cluster.RunInitialServers() + re.NoError(err) + leaderName := suite.cluster.WaitLeader() + suite.pdLeaderServer = suite.cluster.GetServer(leaderName) + re.NoError(suite.pdLeaderServer.BootstrapCluster()) + suite.tsoCluster, err = mcs.NewTestTSOCluster(suite.ctx, 2, suite.pdLeaderServer.GetAddr()) + re.NoError(err) +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) TearDownSuite() { + suite.cancel() + suite.tsoCluster.Destroy() + suite.cluster.Destroy() +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) TearDownTest() { + cleanupKeyspaceGroups(suite.Require(), suite.pdLeaderServer) +} + +func cleanupKeyspaceGroups(re *require.Assertions, server *tests.TestServer) { + keyspaceGroups := handlersutil.MustLoadKeyspaceGroups(re, server, "0", "0") + for _, group := range keyspaceGroups { + // Do not delete default keyspace group. + if group.ID == mcsutils.DefaultKeyspaceGroupID { + continue + } + handlersutil.MustDeleteKeyspaceGroup(re, server, group.ID) + } +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) TestKeyspacesServedByDefaultKeyspaceGroup() { + // There is only default keyspace group. Any keyspace, which hasn't been assigned to + // a keyspace group before, will be served by the default keyspace group. + re := suite.Require() + testutil.Eventually(re, func() bool { + for _, keyspaceID := range []uint32{0, 1, 2} { + served := false + for _, server := range suite.tsoCluster.GetServers() { + if server.IsKeyspaceServing(keyspaceID, mcsutils.DefaultKeyspaceGroupID) { + tam, err := server.GetTSOAllocatorManager(mcsutils.DefaultKeyspaceGroupID) + re.NoError(err) + re.NotNil(tam) + served = true + break + } + } + if !served { + return false + } + } + return true + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) + + // Any keyspace that was assigned to a keyspace group before, except default keyspace, + // won't be served at this time. Default keyspace will be served by default keyspace group + // all the time. + for _, server := range suite.tsoCluster.GetServers() { + server.IsKeyspaceServing(mcsutils.DefaultKeyspaceID, mcsutils.DefaultKeyspaceGroupID) + for _, keyspaceGroupID := range []uint32{1, 2, 3} { + server.IsKeyspaceServing(mcsutils.DefaultKeyspaceID, keyspaceGroupID) + server.IsKeyspaceServing(mcsutils.DefaultKeyspaceID, keyspaceGroupID) + for _, keyspaceID := range []uint32{1, 2, 3} { + if server.IsKeyspaceServing(keyspaceID, keyspaceGroupID) { + tam, err := server.GetTSOAllocatorManager(keyspaceGroupID) + re.NoError(err) + re.NotNil(tam) + } + } + } + } + + // Create a client for each keyspace and make sure they can successfully discover the service + // provided by the default keyspace group. + keyspaceIDs := []uint32{0, 1, 2, 3, 1000} + clients := mcs.WaitForMultiKeyspacesTSOAvailable( + suite.ctx, re, keyspaceIDs, []string{suite.pdLeaderServer.GetAddr()}) + re.Equal(len(keyspaceIDs), len(clients)) + mcs.CheckMultiKeyspacesTSO(suite.ctx, re, clients, func() { + time.Sleep(3 * time.Second) + }) +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) TestKeyspacesServedByNonDefaultKeyspaceGroups() { + // Create multiple keyspace groups, and every keyspace should be served by one of them + // on a tso server. + re := suite.Require() + + // Create keyspace groups. + params := []struct { + keyspaceGroupID uint32 + keyspaceIDs []uint32 + }{ + {0, []uint32{0, 10}}, + {1, []uint32{1, 11}}, + {2, []uint32{2, 12}}, + } + + for _, param := range params { + if param.keyspaceGroupID == 0 { + // we have already created default keyspace group, so we can skip it. + // keyspace 10 isn't assigned to any keyspace group, so they will be + // served by default keyspace group. + continue + } + handlersutil.MustCreateKeyspaceGroup(re, suite.pdLeaderServer, &handlers.CreateKeyspaceGroupParams{ + KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: param.keyspaceGroupID, + UserKind: endpoint.Standard.String(), + Members: suite.tsoCluster.GetKeyspaceGroupMember(), + Keyspaces: param.keyspaceIDs, + }, + }, + }) + } + + // Wait until all keyspace groups are ready. + testutil.Eventually(re, func() bool { + for _, param := range params { + for _, keyspaceID := range param.keyspaceIDs { + served := false + for _, server := range suite.tsoCluster.GetServers() { + if server.IsKeyspaceServing(keyspaceID, param.keyspaceGroupID) { + am, err := server.GetTSOAllocatorManager(param.keyspaceGroupID) + re.NoError(err) + re.NotNil(am) + + // Make sure every keyspace group is using the right timestamp path + // for loading/saving timestamp from/to etcd. + var timestampPath string + clusterID := strconv.FormatUint(suite.pdLeaderServer.GetClusterID(), 10) + if param.keyspaceGroupID == mcsutils.DefaultKeyspaceGroupID { + timestampPath = fmt.Sprintf("/pd/%s/timestamp", clusterID) + } else { + timestampPath = fmt.Sprintf("/ms/%s/tso/%05d/gta/timestamp", + clusterID, param.keyspaceGroupID) + } + re.Equal(timestampPath, am.GetTimestampPath("")) + + served = true + } + } + if !served { + return false + } + } + } + return true + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) + + // Create a client for each keyspace and make sure they can successfully discover the service + // provided by the corresponding keyspace group. + keyspaceIDs := make([]uint32, 0) + for _, param := range params { + keyspaceIDs = append(keyspaceIDs, param.keyspaceIDs...) + } + + clients := mcs.WaitForMultiKeyspacesTSOAvailable( + suite.ctx, re, keyspaceIDs, []string{suite.pdLeaderServer.GetAddr()}) + re.Equal(len(keyspaceIDs), len(clients)) + mcs.CheckMultiKeyspacesTSO(suite.ctx, re, clients, func() { + time.Sleep(3 * time.Second) + }) +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) TestTSOKeyspaceGroupSplit() { + re := suite.Require() + // Create the keyspace group 1 with keyspaces [111, 222, 333]. + handlersutil.MustCreateKeyspaceGroup(re, suite.pdLeaderServer, &handlers.CreateKeyspaceGroupParams{ + KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: 1, + UserKind: endpoint.Standard.String(), + Members: suite.tsoCluster.GetKeyspaceGroupMember(), + Keyspaces: []uint32{111, 222, 333}, + }, + }, + }) + kg1 := handlersutil.MustLoadKeyspaceGroupByID(re, suite.pdLeaderServer, 1) + re.Equal(uint32(1), kg1.ID) + re.Equal([]uint32{111, 222, 333}, kg1.Keyspaces) + re.False(kg1.IsSplitting()) + // Get a TSO from the keyspace group 1. + var ( + ts pdpb.Timestamp + err error + ) + testutil.Eventually(re, func() bool { + ts, err = suite.requestTSO(re, 1, 222, 1) + return err == nil && tsoutil.CompareTimestamp(&ts, &pdpb.Timestamp{}) > 0 + }) + ts.Physical += time.Hour.Milliseconds() + // Set the TSO of the keyspace group 1 to a large value. + err = suite.tsoCluster.GetPrimaryServer(222, 1).GetHandler().ResetTS(tsoutil.GenerateTS(&ts), false, true, 1) + re.NoError(err) + // Split the keyspace group 1 to 2. + handlersutil.MustSplitKeyspaceGroup(re, suite.pdLeaderServer, 1, &handlers.SplitKeyspaceGroupByIDParams{ + NewID: 2, + Keyspaces: []uint32{222, 333}, + }) + kg2 := handlersutil.MustLoadKeyspaceGroupByID(re, suite.pdLeaderServer, 2) + re.Equal(uint32(2), kg2.ID) + re.Equal([]uint32{222, 333}, kg2.Keyspaces) + re.True(kg2.IsSplitTarget()) + // Check the split TSO from keyspace group 2. + var splitTS pdpb.Timestamp + testutil.Eventually(re, func() bool { + splitTS, err = suite.requestTSO(re, 1, 222, 2) + return err == nil && tsoutil.CompareTimestamp(&splitTS, &pdpb.Timestamp{}) > 0 + }) + splitTS, err = suite.requestTSO(re, 1, 222, 2) + re.Greater(tsoutil.CompareTimestamp(&splitTS, &ts), 0) +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) requestTSO( + re *require.Assertions, + count, keyspaceID, keyspaceGroupID uint32, +) (pdpb.Timestamp, error) { + primary := suite.tsoCluster.WaitForPrimaryServing(re, keyspaceID, keyspaceGroupID) + kgm := primary.GetKeyspaceGroupManager() + re.NotNil(kgm) + ts, _, err := kgm.HandleTSORequest(keyspaceID, keyspaceGroupID, tsopkg.GlobalDCLocation, count) + re.NoError(err) + return ts, err +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) TestTSOKeyspaceGroupSplitElection() { + re := suite.Require() + // Create the keyspace group 1 with keyspaces [111, 222, 333]. + handlersutil.MustCreateKeyspaceGroup(re, suite.pdLeaderServer, &handlers.CreateKeyspaceGroupParams{ + KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: 1, + UserKind: endpoint.Standard.String(), + Members: suite.tsoCluster.GetKeyspaceGroupMember(), + Keyspaces: []uint32{111, 222, 333}, + }, + }, + }) + kg1 := handlersutil.MustLoadKeyspaceGroupByID(re, suite.pdLeaderServer, 1) + re.Equal(uint32(1), kg1.ID) + re.Equal([]uint32{111, 222, 333}, kg1.Keyspaces) + re.False(kg1.IsSplitting()) + // Split the keyspace group 1 to 2. + handlersutil.MustSplitKeyspaceGroup(re, suite.pdLeaderServer, 1, &handlers.SplitKeyspaceGroupByIDParams{ + NewID: 2, + Keyspaces: []uint32{222, 333}, + }) + kg2 := handlersutil.MustLoadKeyspaceGroupByID(re, suite.pdLeaderServer, 2) + re.Equal(uint32(2), kg2.ID) + re.Equal([]uint32{222, 333}, kg2.Keyspaces) + re.True(kg2.IsSplitTarget()) + // Check the leadership. + member1, err := suite.tsoCluster.WaitForPrimaryServing(re, 111, 1).GetMember(111, 1) + re.NoError(err) + re.NotNil(member1) + member2, err := suite.tsoCluster.WaitForPrimaryServing(re, 222, 2).GetMember(222, 2) + re.NoError(err) + re.NotNil(member2) + // Wait for the leader of the keyspace group 1 and 2 to be elected. + testutil.Eventually(re, func() bool { + return len(member1.GetLeaderListenUrls()) > 0 && len(member2.GetLeaderListenUrls()) > 0 + }) + // Check if the leader of the keyspace group 1 and 2 are the same. + re.Equal(member1.GetLeaderListenUrls(), member2.GetLeaderListenUrls()) + // Resign and block the leader of the keyspace group 1 from being elected. + member1.(*member.Participant).SetCampaignChecker(func(*election.Leadership) bool { + return false + }) + member1.ResetLeader() + // The leader of the keyspace group 2 should be resigned also. + testutil.Eventually(re, func() bool { + return member2.IsLeader() == false + }) + // Check if the leader of the keyspace group 1 and 2 are the same again. + member1.(*member.Participant).SetCampaignChecker(nil) + testutil.Eventually(re, func() bool { + return len(member1.GetLeaderListenUrls()) > 0 && len(member2.GetLeaderListenUrls()) > 0 + }) + re.Equal(member1.GetLeaderListenUrls(), member2.GetLeaderListenUrls()) + // Finish the split. + handlersutil.MustFinishSplitKeyspaceGroup(re, suite.pdLeaderServer, 2) +} + +func (suite *tsoKeyspaceGroupManagerTestSuite) TestTSOKeyspaceGroupSplitClient() { + re := suite.Require() + // Create the keyspace group 1 with keyspaces [111, 222, 333]. + handlersutil.MustCreateKeyspaceGroup(re, suite.pdLeaderServer, &handlers.CreateKeyspaceGroupParams{ + KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: 1, + UserKind: endpoint.Standard.String(), + Members: suite.tsoCluster.GetKeyspaceGroupMember(), + Keyspaces: []uint32{111, 222, 333}, + }, + }, + }) + kg1 := handlersutil.MustLoadKeyspaceGroupByID(re, suite.pdLeaderServer, 1) + re.Equal(uint32(1), kg1.ID) + re.Equal([]uint32{111, 222, 333}, kg1.Keyspaces) + re.False(kg1.IsSplitting()) + // Prepare the client for keyspace 222. + var tsoClient pd.TSOClient + tsoClient, err := pd.NewClientWithKeyspace(suite.ctx, 222, []string{suite.pdLeaderServer.GetAddr()}, pd.SecurityOption{}) + re.NoError(err) + re.NotNil(tsoClient) + // Request the TSO for keyspace 222 concurrently. + var ( + wg sync.WaitGroup + ctx, cancel = context.WithCancel(suite.ctx) + lastPhysical, lastLogical int64 + ) + wg.Add(1) + go func() { + defer wg.Done() + for { + select { + case <-ctx.Done(): + // Make sure at least one TSO request is successful. + re.NotEmpty(lastPhysical) + return + default: + } + physical, logical, err := tsoClient.GetTS(ctx) + if err != nil { + errMsg := err.Error() + // Ignore the errors caused by the split and context cancellation. + if strings.Contains(errMsg, "context canceled") || + strings.Contains(errMsg, "not leader") || + strings.Contains(errMsg, "not served") || + strings.Contains(errMsg, "ErrKeyspaceNotAssigned") { + continue + } + re.FailNow(errMsg) + } + if physical == lastPhysical { + re.Greater(logical, lastLogical) + } else { + re.Greater(physical, lastPhysical) + } + lastPhysical, lastLogical = physical, logical + } + }() + // Split the keyspace group 1 to 2. + handlersutil.MustSplitKeyspaceGroup(re, suite.pdLeaderServer, 1, &handlers.SplitKeyspaceGroupByIDParams{ + NewID: 2, + Keyspaces: []uint32{222, 333}, + }) + kg2 := handlersutil.MustLoadKeyspaceGroupByID(re, suite.pdLeaderServer, 2) + re.Equal(uint32(2), kg2.ID) + re.Equal([]uint32{222, 333}, kg2.Keyspaces) + re.True(kg2.IsSplitTarget()) + // Finish the split. + handlersutil.MustFinishSplitKeyspaceGroup(re, suite.pdLeaderServer, 2) + // Wait for a while to make sure the client has received the new TSO. + time.Sleep(time.Second) + // Stop the client. + cancel() + wg.Wait() +} diff --git a/tests/mcs/tso/server_test.go b/tests/integrations/mcs/tso/server_test.go similarity index 51% rename from tests/mcs/tso/server_test.go rename to tests/integrations/mcs/tso/server_test.go index 99366240612..4dce8313034 100644 --- a/tests/mcs/tso/server_test.go +++ b/tests/integrations/mcs/tso/server_test.go @@ -16,9 +16,11 @@ package tso import ( "context" + "encoding/json" "fmt" "io" "net/http" + "strconv" "strings" "testing" "time" @@ -28,18 +30,18 @@ import ( "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" pd "github.com/tikv/pd/client" - bs "github.com/tikv/pd/pkg/basicserver" "github.com/tikv/pd/pkg/core" "github.com/tikv/pd/pkg/mcs/discovery" tso "github.com/tikv/pd/pkg/mcs/tso/server" tsoapi "github.com/tikv/pd/pkg/mcs/tso/server/apis/v1" "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/pkg/utils/etcdutil" "github.com/tikv/pd/pkg/utils/tempurl" "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/pkg/utils/tsoutil" "github.com/tikv/pd/tests" - "github.com/tikv/pd/tests/mcs" + "github.com/tikv/pd/tests/integrations/mcs" "go.etcd.io/etcd/clientv3" "go.uber.org/goleak" "google.golang.org/grpc" @@ -76,6 +78,7 @@ func (suite *tsoServerTestSuite) SetupSuite() { leaderName := suite.cluster.WaitLeader() suite.pdLeader = suite.cluster.GetServer(leaderName) suite.backendEndpoints = suite.pdLeader.GetAddr() + suite.NoError(suite.pdLeader.BootstrapCluster()) } func (suite *tsoServerTestSuite) TearDownSuite() { @@ -122,7 +125,7 @@ func (suite *tsoServerTestSuite) TestTSOServerStartAndStopNormally() { } } -func (suite *tsoServerTestSuite) TestPariticipantStartWithAdvertiseListenAddr() { +func (suite *tsoServerTestSuite) TestParticipantStartWithAdvertiseListenAddr() { re := suite.Require() cfg := tso.NewConfig() @@ -132,14 +135,16 @@ func (suite *tsoServerTestSuite) TestPariticipantStartWithAdvertiseListenAddr() cfg, err := tso.GenerateConfig(cfg) re.NoError(err) - s, cleanup, err := tso.NewTSOTestServer(suite.ctx, re, cfg) + s, cleanup, err := mcs.NewTSOTestServer(suite.ctx, cfg) re.NoError(err) defer cleanup() testutil.Eventually(re, func() bool { return s.IsServing() }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) - re.Equal(cfg.AdvertiseListenAddr, s.GetMemberName()) + member, err := s.GetMember(utils.DefaultKeyspaceID, utils.DefaultKeyspaceGroupID) + re.NoError(err) + re.Equal(fmt.Sprintf("%s-%05d", cfg.AdvertiseListenAddr, utils.DefaultKeyspaceGroupID), member.Name()) } func TestTSOPath(t *testing.T) { @@ -168,6 +173,7 @@ func checkTSOPath(re *require.Assertions, isAPIServiceMode bool) { re.NoError(err) leaderName := cluster.WaitLeader() pdLeader := cluster.GetServer(leaderName) + re.NoError(pdLeader.BootstrapCluster()) backendEndpoints := pdLeader.GetAddr() client := pdLeader.GetEtcdClient() if isAPIServiceMode { @@ -179,7 +185,7 @@ func checkTSOPath(re *require.Assertions, isAPIServiceMode bool) { _, cleanup := mcs.StartSingleTSOTestServer(ctx, re, backendEndpoints, tempurl.Alloc()) defer cleanup() - cli := mcs.SetupTSOClient(ctx, re, []string{backendEndpoints}) + cli := mcs.SetupClientWithAPIContext(ctx, re, pd.NewAPIContextV2(""), []string{backendEndpoints}) physical, logical, err := cli.GetTS(ctx) re.NoError(err) ts := tsoutil.ComposeTS(physical, logical) @@ -188,65 +194,68 @@ func checkTSOPath(re *require.Assertions, isAPIServiceMode bool) { re.Equal(1, getEtcdTimestampKeyNum(re, client)) } -func TestWaitAPIServiceReady(t *testing.T) { - re := require.New(t) - checkWaitAPIServiceReady(re, true /*isAPIServiceMode*/) - checkWaitAPIServiceReady(re, false /*isAPIServiceMode*/) +func getEtcdTimestampKeyNum(re *require.Assertions, client *clientv3.Client) int { + resp, err := etcdutil.EtcdKVGet(client, "/", clientv3.WithPrefix()) + re.NoError(err) + var count int + for _, kv := range resp.Kvs { + key := strings.TrimSpace(string(kv.Key)) + if !strings.HasSuffix(key, "timestamp") { + continue + } + count++ + } + return count } -func checkWaitAPIServiceReady(re *require.Assertions, isAPIServiceMode bool) { - var ( - cluster *tests.TestCluster - err error - ) +func TestWaitAPIServiceReady(t *testing.T) { + re := require.New(t) ctx, cancel := context.WithCancel(context.Background()) defer cancel() - if isAPIServiceMode { - cluster, err = tests.NewTestAPICluster(ctx, 1) - } else { - cluster, err = tests.NewTestCluster(ctx, 1) + + startCluster := func(isAPIServiceMode bool) (cluster *tests.TestCluster, backendEndpoints string) { + var err error + if isAPIServiceMode { + cluster, err = tests.NewTestAPICluster(ctx, 1) + } else { + cluster, err = tests.NewTestCluster(ctx, 1) + } + re.NoError(err) + err = cluster.RunInitialServers() + re.NoError(err) + leaderName := cluster.WaitLeader() + pdLeader := cluster.GetServer(leaderName) + return cluster, pdLeader.GetAddr() } - re.NoError(err) - defer cluster.Destroy() - err = cluster.RunInitialServers() - re.NoError(err) - leaderName := cluster.WaitLeader() - pdLeader := cluster.GetServer(leaderName) - backendEndpoints := pdLeader.GetAddr() + + // tso server cannot be started because the pd server is not ready as api service. + cluster, backendEndpoints := startCluster(false /*isAPIServiceMode*/) sctx, scancel := context.WithTimeout(ctx, time.Second*10) defer scancel() - s, cleanup := mcs.StartSingleTSOTestServerWithoutCheck(sctx, re, backendEndpoints, tempurl.Alloc()) + s, _, err := mcs.StartSingleTSOTestServerWithoutCheck(sctx, re, backendEndpoints, tempurl.Alloc()) + re.Error(err) + re.Nil(s) + cluster.Destroy() + + // tso server can be started because the pd server is ready as api service. + cluster, backendEndpoints = startCluster(true /*isAPIServiceMode*/) + sctx, scancel = context.WithTimeout(ctx, time.Second*10) + defer scancel() + s, cleanup, err := mcs.StartSingleTSOTestServerWithoutCheck(sctx, re, backendEndpoints, tempurl.Alloc()) + re.NoError(err) + defer cluster.Destroy() + defer cleanup() + for i := 0; i < 12; i++ { select { case <-time.After(time.Second): case <-sctx.Done(): return } - if isAPIServiceMode { - if s != nil && s.IsServing() { - break - } - } - } - if isAPIServiceMode { - cleanup() - } else { - re.Nil(s) - } -} - -func getEtcdTimestampKeyNum(re *require.Assertions, client *clientv3.Client) int { - resp, err := etcdutil.EtcdKVGet(client, "/", clientv3.WithPrefix()) - re.NoError(err) - var count int - for _, kv := range resp.Kvs { - key := strings.TrimSpace(string(kv.Key)) - if !strings.HasSuffix(key, "timestamp") { - continue + if s != nil && s.IsServing() { + break } - count++ } - return count } type APIServerForwardTestSuite struct { @@ -263,11 +272,11 @@ func TestAPIServerForwardTestSuite(t *testing.T) { suite.Run(t, new(APIServerForwardTestSuite)) } -func (suite *APIServerForwardTestSuite) SetupSuite() { +func (suite *APIServerForwardTestSuite) SetupTest() { var err error re := suite.Require() suite.ctx, suite.cancel = context.WithCancel(context.Background()) - suite.cluster, err = tests.NewTestAPICluster(suite.ctx, 1) + suite.cluster, err = tests.NewTestAPICluster(suite.ctx, 3) re.NoError(err) err = suite.cluster.RunInitialServers() @@ -279,76 +288,181 @@ func (suite *APIServerForwardTestSuite) SetupSuite() { suite.NoError(suite.pdLeader.BootstrapCluster()) suite.addRegions() + suite.NoError(failpoint.Enable("github.com/tikv/pd/client/usePDServiceMode", "return(true)")) suite.pdClient, err = pd.NewClientWithContext(context.Background(), []string{suite.backendEndpoints}, pd.SecurityOption{}, pd.WithMaxErrorRetry(1)) suite.NoError(err) } -func (suite *APIServerForwardTestSuite) TearDownSuite() { +func (suite *APIServerForwardTestSuite) TearDownTest() { suite.pdClient.Close() etcdClient := suite.pdLeader.GetEtcdClient() - endpoints, err := discovery.Discover(etcdClient, utils.TSOServiceName) + clusterID := strconv.FormatUint(suite.pdLeader.GetClusterID(), 10) + endpoints, err := discovery.Discover(etcdClient, clusterID, utils.TSOServiceName) suite.NoError(err) if len(endpoints) != 0 { - endpoints, err = discovery.Discover(etcdClient, utils.TSOServiceName) + endpoints, err = discovery.Discover(etcdClient, clusterID, utils.TSOServiceName) suite.NoError(err) suite.Empty(endpoints) } suite.cluster.Destroy() suite.cancel() + suite.NoError(failpoint.Disable("github.com/tikv/pd/client/usePDServiceMode")) } func (suite *APIServerForwardTestSuite) TestForwardTSORelated() { // Unable to use the tso-related interface without tso server suite.checkUnavailableTSO() - // can use the tso-related interface with tso server - s, cleanup := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, tempurl.Alloc()) - serverMap := make(map[string]bs.Server) - serverMap[s.GetAddr()] = s - mcs.WaitForPrimaryServing(suite.Require(), serverMap) + tc, err := mcs.NewTestTSOCluster(suite.ctx, 1, suite.backendEndpoints) + suite.NoError(err) + defer tc.Destroy() + tc.WaitForDefaultPrimaryServing(suite.Require()) suite.checkAvailableTSO() - cleanup() } func (suite *APIServerForwardTestSuite) TestForwardTSOWhenPrimaryChanged() { - serverMap := make(map[string]bs.Server) - for i := 0; i < 3; i++ { - s, cleanup := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, tempurl.Alloc()) - defer cleanup() - serverMap[s.GetAddr()] = s - } - mcs.WaitForPrimaryServing(suite.Require(), serverMap) + re := suite.Require() - // can use the tso-related interface with new primary + tc, err := mcs.NewTestTSOCluster(suite.ctx, 2, suite.backendEndpoints) + re.NoError(err) + defer tc.Destroy() + tc.WaitForDefaultPrimaryServing(re) + + // can use the tso-related interface with old primary oldPrimary, exist := suite.pdLeader.GetServer().GetServicePrimaryAddr(suite.ctx, utils.TSOServiceName) - suite.True(exist) - serverMap[oldPrimary].Close() - delete(serverMap, oldPrimary) + re.True(exist) + suite.checkAvailableTSO() + + // can use the tso-related interface with new primary + tc.DestroyServer(oldPrimary) time.Sleep(time.Duration(utils.DefaultLeaderLease) * time.Second) // wait for leader lease timeout - mcs.WaitForPrimaryServing(suite.Require(), serverMap) + tc.WaitForDefaultPrimaryServing(re) primary, exist := suite.pdLeader.GetServer().GetServicePrimaryAddr(suite.ctx, utils.TSOServiceName) - suite.True(exist) - suite.NotEqual(oldPrimary, primary) + re.True(exist) + re.NotEqual(oldPrimary, primary) suite.checkAvailableTSO() // can use the tso-related interface with old primary again - s, cleanup := mcs.StartSingleTSOTestServer(suite.ctx, suite.Require(), suite.backendEndpoints, oldPrimary) - defer cleanup() - serverMap[oldPrimary] = s + tc.AddServer(oldPrimary) suite.checkAvailableTSO() - for addr, s := range serverMap { + for addr := range tc.GetServers() { if addr != oldPrimary { - s.Close() - delete(serverMap, addr) + tc.DestroyServer(addr) } } - mcs.WaitForPrimaryServing(suite.Require(), serverMap) + tc.WaitForDefaultPrimaryServing(re) time.Sleep(time.Duration(utils.DefaultLeaderLease) * time.Second) // wait for leader lease timeout primary, exist = suite.pdLeader.GetServer().GetServicePrimaryAddr(suite.ctx, utils.TSOServiceName) - suite.True(exist) - suite.Equal(oldPrimary, primary) + re.True(exist) + re.Equal(oldPrimary, primary) + suite.checkAvailableTSO() +} + +func (suite *APIServerForwardTestSuite) TestResignTSOPrimaryForward() { + // TODO: test random kill primary with 3 nodes + re := suite.Require() + + tc, err := mcs.NewTestTSOCluster(suite.ctx, 2, suite.backendEndpoints) + re.NoError(err) + defer tc.Destroy() + tc.WaitForDefaultPrimaryServing(re) + + for j := 0; j < 10; j++ { + tc.ResignPrimary(utils.DefaultKeyspaceID, utils.DefaultKeyspaceGroupID) + tc.WaitForDefaultPrimaryServing(re) + var err error + for i := 0; i < 3; i++ { // try 3 times + _, _, err = suite.pdClient.GetTS(suite.ctx) + if err == nil { + break + } + time.Sleep(100 * time.Millisecond) + } + suite.NoError(err) + suite.checkAvailableTSO() + } +} + +func (suite *APIServerForwardTestSuite) TestResignAPIPrimaryForward() { + re := suite.Require() + + tc, err := mcs.NewTestTSOCluster(suite.ctx, 2, suite.backendEndpoints) + re.NoError(err) + defer tc.Destroy() + tc.WaitForDefaultPrimaryServing(re) + + for j := 0; j < 10; j++ { + suite.pdLeader.ResignLeader() + suite.pdLeader = suite.cluster.GetServer(suite.cluster.WaitLeader()) + suite.backendEndpoints = suite.pdLeader.GetAddr() + _, _, err = suite.pdClient.GetTS(suite.ctx) + suite.NoError(err) + } +} + +func (suite *APIServerForwardTestSuite) TestForwardTSOUnexpectedToFollower1() { + suite.checkForwardTSOUnexpectedToFollower(func() { + // unary call will retry internally + // try to update gc safe point + min, err := suite.pdClient.UpdateServiceGCSafePoint(context.Background(), "a", 1000, 1) + suite.NoError(err) + suite.Equal(uint64(0), min) + }) +} + +func (suite *APIServerForwardTestSuite) TestForwardTSOUnexpectedToFollower2() { + suite.checkForwardTSOUnexpectedToFollower(func() { + // unary call will retry internally + // try to set external ts + ts, err := suite.pdClient.GetExternalTimestamp(suite.ctx) + suite.NoError(err) + err = suite.pdClient.SetExternalTimestamp(suite.ctx, ts+1) + suite.NoError(err) + }) +} + +func (suite *APIServerForwardTestSuite) TestForwardTSOUnexpectedToFollower3() { + suite.checkForwardTSOUnexpectedToFollower(func() { + _, _, err := suite.pdClient.GetTS(suite.ctx) + suite.Error(err) + }) +} + +func (suite *APIServerForwardTestSuite) checkForwardTSOUnexpectedToFollower(checkTSO func()) { + re := suite.Require() + tc, err := mcs.NewTestTSOCluster(suite.ctx, 2, suite.backendEndpoints) + re.NoError(err) + tc.WaitForDefaultPrimaryServing(re) + + // get follower's address + servers := tc.GetServers() + oldPrimary := tc.GetPrimaryServer(utils.DefaultKeyspaceID, utils.DefaultKeyspaceGroupID).GetAddr() + var follower string + for addr := range servers { + if addr != oldPrimary { + follower = addr + break + } + } + re.NotEmpty(follower) + + // write follower's address to cache to simulate cache is not updated. + suite.pdLeader.GetServer().SetServicePrimaryAddr(utils.TSOServiceName, follower) + errorAddr, ok := suite.pdLeader.GetServer().GetServicePrimaryAddr(suite.ctx, utils.TSOServiceName) + suite.True(ok) + suite.Equal(follower, errorAddr) + + // test tso request + checkTSO() + + // test tso request will success after cache is updated suite.checkAvailableTSO() + newPrimary, exist2 := suite.pdLeader.GetServer().GetServicePrimaryAddr(suite.ctx, utils.TSOServiceName) + suite.True(exist2) + suite.NotEqual(errorAddr, newPrimary) + suite.Equal(oldPrimary, newPrimary) + tc.Destroy() } func (suite *APIServerForwardTestSuite) addRegions() { @@ -377,65 +491,85 @@ func (suite *APIServerForwardTestSuite) checkUnavailableTSO() { } func (suite *APIServerForwardTestSuite) checkAvailableTSO() { - err := mcs.WaitForTSOServiceAvailable(suite.ctx, suite.pdClient) - suite.NoError(err) + re := suite.Require() + mcs.WaitForTSOServiceAvailable(suite.ctx, re, suite.pdClient) // try to get ts - _, _, err = suite.pdClient.GetTS(suite.ctx) - suite.NoError(err) + _, _, err := suite.pdClient.GetTS(suite.ctx) + re.NoError(err) // try to update gc safe point min, err := suite.pdClient.UpdateServiceGCSafePoint(context.Background(), "a", 1000, 1) - suite.NoError(err) - suite.Equal(uint64(0), min) + re.NoError(err) + re.Equal(uint64(0), min) // try to set external ts ts, err := suite.pdClient.GetExternalTimestamp(suite.ctx) - suite.NoError(err) + re.NoError(err) err = suite.pdClient.SetExternalTimestamp(suite.ctx, ts+1) - suite.NoError(err) + re.NoError(err) } -func TestAdvertiseAddr(t *testing.T) { - re := require.New(t) +type CommonTestSuite struct { + suite.Suite + ctx context.Context + cancel context.CancelFunc + cluster *tests.TestCluster + tsoCluster *mcs.TestTSOCluster + pdLeader *tests.TestServer + // tsoDefaultPrimaryServer is the primary server of the default keyspace group + tsoDefaultPrimaryServer *tso.Server + backendEndpoints string +} - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - cluster, err := tests.NewTestAPICluster(ctx, 1) - defer cluster.Destroy() - re.NoError(err) +func TestCommonTestSuite(t *testing.T) { + suite.Run(t, new(CommonTestSuite)) +} - err = cluster.RunInitialServers() +func (suite *CommonTestSuite) SetupSuite() { + var err error + re := suite.Require() + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + suite.cluster, err = tests.NewTestAPICluster(suite.ctx, 1) re.NoError(err) - leaderName := cluster.WaitLeader() - leader := cluster.GetServer(leaderName) + err = suite.cluster.RunInitialServers() + re.NoError(err) - u := tempurl.Alloc() - s, cleanup := mcs.StartSingleTSOTestServer(ctx, re, leader.GetAddr(), u) - defer cleanup() + leaderName := suite.cluster.WaitLeader() + suite.pdLeader = suite.cluster.GetServer(leaderName) + suite.backendEndpoints = suite.pdLeader.GetAddr() + suite.NoError(suite.pdLeader.BootstrapCluster()) - tsoServerConf := s.GetConfig() - re.Equal(u, tsoServerConf.AdvertiseListenAddr) + suite.tsoCluster, err = mcs.NewTestTSOCluster(suite.ctx, 1, suite.backendEndpoints) + suite.NoError(err) + suite.tsoCluster.WaitForDefaultPrimaryServing(re) + suite.tsoDefaultPrimaryServer = suite.tsoCluster.GetPrimaryServer(utils.DefaultKeyspaceID, utils.DefaultKeyspaceGroupID) } -func TestMetrics(t *testing.T) { - re := require.New(t) - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - cluster, err := tests.NewTestAPICluster(ctx, 1) - defer cluster.Destroy() - re.NoError(err) +func (suite *CommonTestSuite) TearDownSuite() { + suite.tsoCluster.Destroy() + etcdClient := suite.pdLeader.GetEtcdClient() + clusterID := strconv.FormatUint(suite.pdLeader.GetClusterID(), 10) + endpoints, err := discovery.Discover(etcdClient, clusterID, utils.TSOServiceName) + suite.NoError(err) + if len(endpoints) != 0 { + endpoints, err = discovery.Discover(etcdClient, clusterID, utils.TSOServiceName) + suite.NoError(err) + suite.Empty(endpoints) + } + suite.cluster.Destroy() + suite.cancel() +} - err = cluster.RunInitialServers() - re.NoError(err) +func (suite *CommonTestSuite) TestAdvertiseAddr() { + re := suite.Require() - leaderName := cluster.WaitLeader() - leader := cluster.GetServer(leaderName) + conf := suite.tsoDefaultPrimaryServer.GetConfig() + re.Equal(conf.GetListenAddr(), conf.GetAdvertiseListenAddr()) +} - u := tempurl.Alloc() - s, cleanup := mcs.StartSingleTSOTestServer(ctx, re, leader.GetAddr(), u) - defer cleanup() +func (suite *CommonTestSuite) TestMetrics() { + re := suite.Require() - resp, err := http.Get(s.GetConfig().AdvertiseListenAddr + "/metrics") + resp, err := http.Get(suite.tsoDefaultPrimaryServer.GetConfig().GetAdvertiseListenAddr() + "/metrics") re.NoError(err) defer resp.Body.Close() re.Equal(http.StatusOK, resp.StatusCode) @@ -443,3 +577,37 @@ func TestMetrics(t *testing.T) { re.NoError(err) re.Contains(string(respBytes), "tso_server_info") } + +func (suite *CommonTestSuite) TestBootstrapDefaultKeyspaceGroup() { + re := suite.Require() + + // check the default keyspace group + check := func() { + resp, err := http.Get(suite.pdLeader.GetServer().GetConfig().AdvertiseClientUrls + "/pd/api/v2/tso/keyspace-groups") + re.NoError(err) + defer resp.Body.Close() + re.Equal(http.StatusOK, resp.StatusCode) + respString, err := io.ReadAll(resp.Body) + re.NoError(err) + var kgs []*endpoint.KeyspaceGroup + re.NoError(json.Unmarshal(respString, &kgs)) + re.Len(kgs, 1) + re.Equal(utils.DefaultKeyspaceGroupID, kgs[0].ID) + re.Equal(endpoint.Basic.String(), kgs[0].UserKind) + re.Empty(kgs[0].SplitState) + re.Empty(kgs[0].Members) + re.Empty(kgs[0].KeyspaceLookupTable) + } + check() + + s, err := suite.cluster.JoinAPIServer(suite.ctx) + re.NoError(err) + re.NoError(s.Run()) + + // transfer leader to the new server + suite.pdLeader.ResignLeader() + suite.pdLeader = suite.cluster.GetServer(suite.cluster.WaitLeader()) + check() + suite.pdLeader.ResignLeader() + suite.pdLeader = suite.cluster.GetServer(suite.cluster.WaitLeader()) +} diff --git a/tests/tso/Makefile b/tests/integrations/tso/Makefile similarity index 60% rename from tests/tso/Makefile rename to tests/integrations/tso/Makefile index 655cc0e62d1..820bbf33ccf 100644 --- a/tests/tso/Makefile +++ b/tests/integrations/tso/Makefile @@ -12,34 +12,35 @@ # See the License for the specific language governing permissions and # limitations under the License. -GO_TOOLS_BIN_PATH := $(shell pwd)/../../.tools/bin +ROOT_PATH := ../../.. +GO_TOOLS_BIN_PATH := $(ROOT_PATH)/.tools/bin PATH := $(GO_TOOLS_BIN_PATH):$(PATH) SHELL := env PATH='$(PATH)' GOBIN='$(GO_TOOLS_BIN_PATH)' $(shell which bash) static: install-tools @ gofmt -s -l -d . 2>&1 | awk '{ print } END { if (NR > 0) { exit 1 } }' @ golangci-lint run ./... - @ revive -formatter friendly -config ../../revive.toml . + @ revive -formatter friendly -config $(ROOT_PATH)/revive.toml . tidy: @ go mod tidy git diff go.mod go.sum | cat git diff --quiet go.mod go.sum -test: enable-codegen - CGO_ENABLED=1 go test ./... -tags deadlock -race -cover || { $(MAKE) disable-codegen && exit 1; } - $(MAKE) disable-codegen +test: failpoint-enable + CGO_ENABLED=1 go test -v -tags deadlock -race -cover || { $(MAKE) failpoint-disable && exit 1; } + $(MAKE) failpoint-disable -ci-test-job: enable-codegen - CGO_ENABLED=1 go test ./... -tags deadlock -race -covermode=atomic -coverprofile=covprofile -coverpkg=../../... github.com/tikv/pd/tests/tso +ci-test-job: + CGO_ENABLED=1 go test -tags deadlock -race -covermode=atomic -coverprofile=covprofile -coverpkg=$(ROOT_PATH)/... github.com/tikv/pd/tests/integrations/tso install-tools: - cd ../../ && $(MAKE) install-tools + cd $(ROOT_PATH) && $(MAKE) install-tools -enable-codegen: - cd ../../ && $(MAKE) failpoint-enable +failpoint-enable: + cd $(ROOT_PATH) && $(MAKE) failpoint-enable go mod tidy -disable-codegen: - cd ../../ && $(MAKE) failpoint-disable +failpoint-disable: + cd $(ROOT_PATH) && $(MAKE) failpoint-disable go mod tidy diff --git a/tests/integrations/tso/client_test.go b/tests/integrations/tso/client_test.go new file mode 100644 index 00000000000..9a21c0f921a --- /dev/null +++ b/tests/integrations/tso/client_test.go @@ -0,0 +1,480 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package tso + +import ( + "context" + "fmt" + "math" + "math/rand" + "strings" + "sync" + "testing" + "time" + + "github.com/pingcap/failpoint" + "github.com/stretchr/testify/require" + "github.com/stretchr/testify/suite" + pd "github.com/tikv/pd/client" + "github.com/tikv/pd/client/testutil" + bs "github.com/tikv/pd/pkg/basicserver" + mcsutils "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/slice" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/utils/tempurl" + "github.com/tikv/pd/pkg/utils/tsoutil" + "github.com/tikv/pd/server/apiv2/handlers" + "github.com/tikv/pd/tests" + "github.com/tikv/pd/tests/integrations/mcs" + handlersutil "github.com/tikv/pd/tests/server/apiv2/handlers" +) + +var r = rand.New(rand.NewSource(time.Now().UnixNano())) + +type tsoClientTestSuite struct { + suite.Suite + legacy bool + + ctx context.Context + cancel context.CancelFunc + // The PD cluster. + cluster *tests.TestCluster + // pdLeaderServer is the leader server of the PD cluster. + pdLeaderServer *tests.TestServer + // The TSO service in microservice mode. + tsoCluster *mcs.TestTSOCluster + + keyspaceGroups []struct { + keyspaceGroupID uint32 + keyspaceIDs []uint32 + } + + backendEndpoints string + keyspaceIDs []uint32 + clients []pd.Client +} + +func TestLegacyTSOClient(t *testing.T) { + suite.Run(t, &tsoClientTestSuite{ + legacy: true, + }) +} + +func TestMicroserviceTSOClient(t *testing.T) { + suite.Run(t, &tsoClientTestSuite{ + legacy: false, + }) +} + +func (suite *tsoClientTestSuite) SetupSuite() { + re := suite.Require() + + var err error + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + if suite.legacy { + suite.cluster, err = tests.NewTestCluster(suite.ctx, serverCount) + } else { + suite.cluster, err = tests.NewTestAPICluster(suite.ctx, serverCount) + } + re.NoError(err) + err = suite.cluster.RunInitialServers() + re.NoError(err) + leaderName := suite.cluster.WaitLeader() + suite.pdLeaderServer = suite.cluster.GetServer(leaderName) + re.NoError(suite.pdLeaderServer.BootstrapCluster()) + suite.backendEndpoints = suite.pdLeaderServer.GetAddr() + suite.keyspaceIDs = make([]uint32, 0) + + if suite.legacy { + client, err := pd.NewClientWithContext(suite.ctx, strings.Split(suite.backendEndpoints, ","), pd.SecurityOption{}) + re.NoError(err) + innerClient, ok := client.(interface{ GetServiceDiscovery() pd.ServiceDiscovery }) + re.True(ok) + re.Equal(mcsutils.NullKeyspaceID, innerClient.GetServiceDiscovery().GetKeyspaceID()) + re.Equal(mcsutils.DefaultKeyspaceGroupID, innerClient.GetServiceDiscovery().GetKeyspaceGroupID()) + mcs.WaitForTSOServiceAvailable(suite.ctx, re, client) + suite.clients = make([]pd.Client, 0) + suite.clients = append(suite.clients, client) + } else { + suite.tsoCluster, err = mcs.NewTestTSOCluster(suite.ctx, 3, suite.backendEndpoints) + re.NoError(err) + + suite.keyspaceGroups = []struct { + keyspaceGroupID uint32 + keyspaceIDs []uint32 + }{ + {0, []uint32{mcsutils.DefaultKeyspaceID, 10}}, + {1, []uint32{1, 11}}, + {2, []uint32{2}}, + } + + for _, keyspaceGroup := range suite.keyspaceGroups { + suite.keyspaceIDs = append(suite.keyspaceIDs, keyspaceGroup.keyspaceIDs...) + } + + for _, param := range suite.keyspaceGroups { + if param.keyspaceGroupID == 0 { + // we have already created default keyspace group, so we can skip it. + // keyspace 10 isn't assigned to any keyspace group, so they will be + // served by default keyspace group. + continue + } + handlersutil.MustCreateKeyspaceGroup(re, suite.pdLeaderServer, &handlers.CreateKeyspaceGroupParams{ + KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: param.keyspaceGroupID, + UserKind: endpoint.Standard.String(), + Members: suite.tsoCluster.GetKeyspaceGroupMember(), + Keyspaces: param.keyspaceIDs, + }, + }, + }) + } + + suite.waitForAllKeyspaceGroupsInServing(re) + } +} + +func (suite *tsoClientTestSuite) waitForAllKeyspaceGroupsInServing(re *require.Assertions) { + // The tso servers are loading keyspace groups asynchronously. Make sure all keyspace groups + // are available for serving tso requests from corresponding keyspaces by querying + // IsKeyspaceServing(keyspaceID, the Desired KeyspaceGroupID). if use default keyspace group id + // in the query, it will always return true as the keyspace will be served by default keyspace + // group before the keyspace groups are loaded. + testutil.Eventually(re, func() bool { + for _, keyspaceGroup := range suite.keyspaceGroups { + for _, keyspaceID := range keyspaceGroup.keyspaceIDs { + served := false + for _, server := range suite.tsoCluster.GetServers() { + if server.IsKeyspaceServing(keyspaceID, keyspaceGroup.keyspaceGroupID) { + served = true + break + } + } + if !served { + return false + } + } + } + return true + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) + + // Create clients and make sure they all have discovered the tso service. + suite.clients = mcs.WaitForMultiKeyspacesTSOAvailable( + suite.ctx, re, suite.keyspaceIDs, strings.Split(suite.backendEndpoints, ",")) + re.Equal(len(suite.keyspaceIDs), len(suite.clients)) +} + +func (suite *tsoClientTestSuite) TearDownSuite() { + suite.cancel() + if !suite.legacy { + suite.tsoCluster.Destroy() + } + suite.cluster.Destroy() +} + +func (suite *tsoClientTestSuite) TestGetTS() { + var wg sync.WaitGroup + wg.Add(tsoRequestConcurrencyNumber * len(suite.clients)) + for i := 0; i < tsoRequestConcurrencyNumber; i++ { + for _, client := range suite.clients { + go func(client pd.Client) { + defer wg.Done() + var lastTS uint64 + for j := 0; j < tsoRequestRound; j++ { + physical, logical, err := client.GetTS(suite.ctx) + suite.NoError(err) + ts := tsoutil.ComposeTS(physical, logical) + suite.Less(lastTS, ts) + lastTS = ts + } + }(client) + } + } + wg.Wait() +} + +func (suite *tsoClientTestSuite) TestGetTSAsync() { + var wg sync.WaitGroup + wg.Add(tsoRequestConcurrencyNumber * len(suite.clients)) + for i := 0; i < tsoRequestConcurrencyNumber; i++ { + for _, client := range suite.clients { + go func(client pd.Client) { + defer wg.Done() + tsFutures := make([]pd.TSFuture, tsoRequestRound) + for j := range tsFutures { + tsFutures[j] = client.GetTSAsync(suite.ctx) + } + var lastTS uint64 = math.MaxUint64 + for j := len(tsFutures) - 1; j >= 0; j-- { + physical, logical, err := tsFutures[j].Wait() + suite.NoError(err) + ts := tsoutil.ComposeTS(physical, logical) + suite.Greater(lastTS, ts) + lastTS = ts + } + }(client) + } + } + wg.Wait() +} + +func (suite *tsoClientTestSuite) TestDiscoverTSOServiceWithLegacyPath() { + re := suite.Require() + keyspaceID := uint32(1000000) + // Make sure this keyspace ID is not in use somewhere. + re.False(slice.Contains(suite.keyspaceIDs, keyspaceID)) + failpointValue := fmt.Sprintf(`return(%d)`, keyspaceID) + // Simulate the case that the server has lower version than the client and returns no tso addrs + // in the GetClusterInfo RPC. + re.NoError(failpoint.Enable("github.com/tikv/pd/client/serverReturnsNoTSOAddrs", `return(true)`)) + re.NoError(failpoint.Enable("github.com/tikv/pd/client/unexpectedCallOfFindGroupByKeyspaceID", failpointValue)) + defer func() { + re.NoError(failpoint.Disable("github.com/tikv/pd/client/serverReturnsNoTSOAddrs")) + re.NoError(failpoint.Disable("github.com/tikv/pd/client/unexpectedCallOfFindGroupByKeyspaceID")) + }() + + ctx, cancel := context.WithCancel(suite.ctx) + defer cancel() + client := mcs.SetupClientWithKeyspaceID( + ctx, re, keyspaceID, strings.Split(suite.backendEndpoints, ",")) + var lastTS uint64 + for j := 0; j < tsoRequestRound; j++ { + physical, logical, err := client.GetTS(ctx) + suite.NoError(err) + ts := tsoutil.ComposeTS(physical, logical) + suite.Less(lastTS, ts) + lastTS = ts + } +} + +// TestGetMinTS tests the correctness of GetMinTS. +func (suite *tsoClientTestSuite) TestGetMinTS() { + re := suite.Require() + suite.waitForAllKeyspaceGroupsInServing(re) + + var wg sync.WaitGroup + wg.Add(tsoRequestConcurrencyNumber * len(suite.clients)) + for i := 0; i < tsoRequestConcurrencyNumber; i++ { + for _, client := range suite.clients { + go func(client pd.Client) { + defer wg.Done() + var lastMinTS uint64 + for j := 0; j < tsoRequestRound; j++ { + physical, logical, err := client.GetMinTS(suite.ctx) + re.NoError(err) + minTS := tsoutil.ComposeTS(physical, logical) + re.Less(lastMinTS, minTS) + lastMinTS = minTS + + // Now we check whether the returned ts is the minimum one + // among all keyspace groups, i.e., the returned ts is + // less than the new timestamps of all keyspace groups. + for _, client := range suite.clients { + physical, logical, err := client.GetTS(suite.ctx) + re.NoError(err) + ts := tsoutil.ComposeTS(physical, logical) + re.Less(minTS, ts) + } + } + }(client) + } + } + wg.Wait() +} + +// More details can be found in this issue: https://github.com/tikv/pd/issues/4884 +func (suite *tsoClientTestSuite) TestUpdateAfterResetTSO() { + re := suite.Require() + ctx, cancel := context.WithCancel(suite.ctx) + defer cancel() + + for i := 0; i < len(suite.clients); i++ { + client := suite.clients[i] + testutil.Eventually(re, func() bool { + _, _, err := client.GetTS(ctx) + return err == nil + }) + // Resign leader to trigger the TSO resetting. + re.NoError(failpoint.Enable("github.com/tikv/pd/server/updateAfterResetTSO", "return(true)")) + oldLeaderName := suite.cluster.WaitLeader() + err := suite.cluster.GetServer(oldLeaderName).ResignLeader() + re.NoError(err) + re.NoError(failpoint.Disable("github.com/tikv/pd/server/updateAfterResetTSO")) + newLeaderName := suite.cluster.WaitLeader() + re.NotEqual(oldLeaderName, newLeaderName) + // Request a new TSO. + testutil.Eventually(re, func() bool { + _, _, err := client.GetTS(ctx) + return err == nil + }) + // Transfer leader back. + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/tso/delaySyncTimestamp", `return(true)`)) + err = suite.cluster.GetServer(newLeaderName).ResignLeader() + re.NoError(err) + // Should NOT panic here. + testutil.Eventually(re, func() bool { + _, _, err := client.GetTS(ctx) + return err == nil + }) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/tso/delaySyncTimestamp")) + } +} + +func (suite *tsoClientTestSuite) TestRandomResignLeader() { + re := suite.Require() + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/tso/fastUpdatePhysicalInterval", "return(true)")) + defer re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/tso/fastUpdatePhysicalInterval")) + + parallelAct := func() { + // After https://github.com/tikv/pd/issues/6376 is fixed, we can use a smaller number here. + // currently, the time to discover tso service is usually a little longer than 1s, compared + // to the previous time taken < 1s. + n := r.Intn(2) + 3 + time.Sleep(time.Duration(n) * time.Second) + if !suite.legacy { + wg := sync.WaitGroup{} + // Select the first keyspace from all keyspace groups. We need to make sure the selected + // keyspaces are from different keyspace groups, otherwise multiple goroutines below could + // try to resign the primary of the same keyspace group and cause race condition. + keyspaceIDs := make([]uint32, 0) + for _, keyspaceGroup := range suite.keyspaceGroups { + if len(keyspaceGroup.keyspaceIDs) > 0 { + keyspaceIDs = append(keyspaceIDs, keyspaceGroup.keyspaceIDs[0]) + } + } + wg.Add(len(keyspaceIDs)) + for _, keyspaceID := range keyspaceIDs { + go func(keyspaceID uint32) { + defer wg.Done() + err := suite.tsoCluster.ResignPrimary(keyspaceID, mcsutils.DefaultKeyspaceGroupID) + re.NoError(err) + suite.tsoCluster.WaitForPrimaryServing(re, keyspaceID, 0) + }(keyspaceID) + } + wg.Wait() + } else { + err := suite.cluster.ResignLeader() + re.NoError(err) + suite.cluster.WaitLeader() + } + time.Sleep(time.Duration(n) * time.Second) + } + + mcs.CheckMultiKeyspacesTSO(suite.ctx, re, suite.clients, parallelAct) +} + +func (suite *tsoClientTestSuite) TestRandomShutdown() { + re := suite.Require() + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/tso/fastUpdatePhysicalInterval", "return(true)")) + defer re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/tso/fastUpdatePhysicalInterval")) + + parallelAct := func() { + // After https://github.com/tikv/pd/issues/6376 is fixed, we can use a smaller number here. + // currently, the time to discover tso service is usually a little longer than 1s, compared + // to the previous time taken < 1s. + n := r.Intn(2) + 3 + time.Sleep(time.Duration(n) * time.Second) + if !suite.legacy { + suite.tsoCluster.WaitForDefaultPrimaryServing(re).Close() + } else { + suite.cluster.GetServer(suite.cluster.GetLeader()).GetServer().Close() + } + time.Sleep(time.Duration(n) * time.Second) + } + + mcs.CheckMultiKeyspacesTSO(suite.ctx, re, suite.clients, parallelAct) + suite.TearDownSuite() + suite.SetupSuite() +} + +// When we upgrade the PD cluster, there may be a period of time that the old and new PDs are running at the same time. +func TestMixedTSODeployment(t *testing.T) { + re := require.New(t) + + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/tso/fastUpdatePhysicalInterval", "return(true)")) + re.NoError(failpoint.Enable("github.com/tikv/pd/client/skipUpdateServiceMode", "return(true)")) + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/mcs/tso/server/skipWaitAPIServiceReady", "return(true)")) + defer func() { + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/tso/fastUpdatePhysicalInterval")) + re.NoError(failpoint.Disable("github.com/tikv/pd/client/skipUpdateServiceMode")) + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/mcs/tso/server/skipWaitAPIServiceReady")) + }() + + ctx, cancel := context.WithCancel(context.Background()) + cluster, err := tests.NewTestCluster(ctx, 1) + re.NoError(err) + defer cancel() + defer cluster.Destroy() + + err = cluster.RunInitialServers() + re.NoError(err) + + leaderServer := cluster.GetServer(cluster.WaitLeader()) + backendEndpoints := leaderServer.GetAddr() + + apiSvr, err := cluster.JoinAPIServer(ctx) + re.NoError(err) + err = apiSvr.Run() + re.NoError(err) + + s, cleanup := mcs.StartSingleTSOTestServer(ctx, re, backendEndpoints, tempurl.Alloc()) + defer cleanup() + mcs.WaitForPrimaryServing(re, map[string]bs.Server{s.GetAddr(): s}) + + ctx1, cancel1 := context.WithCancel(context.Background()) + var wg sync.WaitGroup + checkTSO(ctx1, re, &wg, backendEndpoints) + wg.Add(1) + go func() { + defer wg.Done() + for i := 0; i < 2; i++ { + n := r.Intn(2) + 1 + time.Sleep(time.Duration(n) * time.Second) + leaderServer.ResignLeader() + leaderServer = cluster.GetServer(cluster.WaitLeader()) + } + cancel1() + }() + wg.Wait() +} + +func checkTSO(ctx context.Context, re *require.Assertions, wg *sync.WaitGroup, backendEndpoints string) { + wg.Add(tsoRequestConcurrencyNumber) + for i := 0; i < tsoRequestConcurrencyNumber; i++ { + go func() { + defer wg.Done() + cli := mcs.SetupClientWithAPIContext(ctx, re, pd.NewAPIContextV1(), strings.Split(backendEndpoints, ",")) + var ts, lastTS uint64 + for { + select { + case <-ctx.Done(): + // Make sure the lastTS is not empty + re.NotEmpty(lastTS) + return + default: + } + physical, logical, err := cli.GetTS(ctx) + // omit the error check since there are many kinds of errors + if err != nil { + continue + } + ts = tsoutil.ComposeTS(physical, logical) + re.Less(lastTS, ts) + lastTS = ts + } + }() + } +} diff --git a/tests/tso/consistency_test.go b/tests/integrations/tso/consistency_test.go similarity index 99% rename from tests/tso/consistency_test.go rename to tests/integrations/tso/consistency_test.go index b4222ff6abb..70000d2abd4 100644 --- a/tests/tso/consistency_test.go +++ b/tests/integrations/tso/consistency_test.go @@ -30,7 +30,7 @@ import ( pd "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/pkg/utils/tsoutil" "github.com/tikv/pd/tests" - "github.com/tikv/pd/tests/mcs" + "github.com/tikv/pd/tests/integrations/mcs" "google.golang.org/grpc" ) diff --git a/tests/tso/go.mod b/tests/integrations/tso/go.mod similarity index 89% rename from tests/tso/go.mod rename to tests/integrations/tso/go.mod index c789853a1e2..b640c8f7d86 100644 --- a/tests/tso/go.mod +++ b/tests/integrations/tso/go.mod @@ -1,11 +1,11 @@ -module github.com/tikv/pd/tests/tso +module github.com/tikv/pd/tests/integrations/tso go 1.20 replace ( - github.com/tikv/pd => ../../ - github.com/tikv/pd/client => ../../client - github.com/tikv/pd/tests/mcs => ../mcs + github.com/tikv/pd => ../../../ + github.com/tikv/pd/client => ../../../client + github.com/tikv/pd/tests/integrations/mcs => ../mcs ) // reset grpc and protobuf deps in order to import client and server at the same time @@ -13,11 +13,11 @@ replace google.golang.org/grpc v1.51.0 => google.golang.org/grpc v1.26.0 require ( github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 - github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba + github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 github.com/stretchr/testify v1.8.2 github.com/tikv/pd v0.0.0-00010101000000-000000000000 github.com/tikv/pd/client v0.0.0-00010101000000-000000000000 - github.com/tikv/pd/tests/mcs v0.0.0-20230320064440-220dbed05897 + github.com/tikv/pd/tests/integrations/mcs v0.0.0-00010101000000-000000000000 google.golang.org/grpc v1.51.0 ) @@ -28,7 +28,6 @@ require ( github.com/PuerkitoBio/purell v1.1.1 // indirect github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect github.com/ReneKroon/ttlcache/v2 v2.3.0 // indirect - github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect github.com/VividCortex/mysqlerr v1.0.0 // indirect github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502 // indirect github.com/aws/aws-sdk-go v1.35.3 // indirect @@ -105,7 +104,7 @@ require ( github.com/pingcap/errcode v0.3.0 // indirect github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c // indirect github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 // indirect - github.com/pingcap/sysutil v0.0.0-20211208032423-041a72e5860d // indirect + github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21 // indirect github.com/pingcap/tidb-dashboard v0.0.0-20230209052558-a58fc2a7e924 // indirect github.com/pingcap/tipb v0.0.0-20220718022156-3e2483c20a9e // indirect github.com/pkg/errors v0.9.1 // indirect @@ -118,7 +117,6 @@ require ( github.com/rs/cors v1.7.0 // indirect github.com/samber/lo v1.37.0 // indirect github.com/sasha-s/go-deadlock v0.2.0 // indirect - github.com/shirou/gopsutil v3.21.3+incompatible // indirect github.com/shirou/gopsutil/v3 v3.22.12 // indirect github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 // indirect github.com/sirupsen/logrus v1.6.0 // indirect @@ -127,8 +125,8 @@ require ( github.com/spf13/cobra v1.0.0 // indirect github.com/spf13/pflag v1.0.5 // indirect github.com/stretchr/objx v0.5.0 // indirect - github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 // indirect - github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba // indirect + github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 // indirect + github.com/swaggo/http-swagger v1.2.6 // indirect github.com/swaggo/swag v1.8.3 // indirect github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 // indirect github.com/tklauser/go-sysconf v0.3.11 // indirect @@ -151,14 +149,14 @@ require ( go.uber.org/zap v1.20.0 // indirect golang.org/x/crypto v0.1.0 // indirect golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a // indirect - golang.org/x/image v0.0.0-20200119044424-58c23975cae1 // indirect - golang.org/x/net v0.2.0 // indirect - golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 // indirect + golang.org/x/image v0.5.0 // indirect + golang.org/x/net v0.9.0 // indirect + golang.org/x/oauth2 v0.4.0 // indirect golang.org/x/sync v0.1.0 // indirect - golang.org/x/sys v0.3.0 // indirect - golang.org/x/text v0.4.0 // indirect + golang.org/x/sys v0.7.0 // indirect + golang.org/x/text v0.9.0 // indirect golang.org/x/time v0.1.0 // indirect - golang.org/x/tools v0.2.0 // indirect + golang.org/x/tools v0.6.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/genproto v0.0.0-20221202195650-67e5cbc046fd // indirect google.golang.org/protobuf v1.28.1 // indirect diff --git a/tests/tso/go.sum b/tests/integrations/tso/go.sum similarity index 92% rename from tests/tso/go.sum rename to tests/integrations/tso/go.sum index 640842c5a7d..c2718e7c0b0 100644 --- a/tests/tso/go.sum +++ b/tests/integrations/tso/go.sum @@ -7,19 +7,17 @@ github.com/KyleBanks/depth v1.2.1/go.mod h1:jzSb9d0L43HxTQfT+oSA1EEp2q+ne2uh6Xge github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= -github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= github.com/ReneKroon/ttlcache/v2 v2.3.0 h1:qZnUjRKIrbKHH6vF5T7Y9Izn5ObfTZfyYpGhvz2BKPo= github.com/ReneKroon/ttlcache/v2 v2.3.0/go.mod h1:zbo6Pv/28e21Z8CzzqgYRArQYGYtjONRxaAKGxzQvG4= -github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk= -github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg= github.com/VividCortex/mysqlerr v1.0.0 h1:5pZ2TZA+YnzPgzBfiUWGqWmKDVNBdrkf9g+DNe1Tiq8= github.com/VividCortex/mysqlerr v1.0.0/go.mod h1:xERx8E4tBhLvpjzdUyQiSfUxeMcATEQrflDAfXsqcAE= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502 h1:L8IbaI/W6h5Cwgh0n4zGeZpVK78r/jBf9ASurHo9+/o= github.com/Xeoncross/go-aesctr-with-hmac v0.0.0-20200623134604-12b17a7ff502/go.mod h1:pmnBM9bxWSiHvC/gSWunUIyDvGn33EkP2CUjxFKtTTM= +github.com/agiledragon/gomonkey/v2 v2.3.1/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= @@ -74,6 +72,7 @@ github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbp github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/corona10/goimagehash v1.0.2 h1:pUfB0LnsJASMPGEZLj7tGY251vF+qLGqOgEP4rUs6kA= github.com/corona10/goimagehash v1.0.2/go.mod h1:/l9umBhvcHQXVtQO1V6Gp1yD20STawkhRnnX0D1bvVI= +github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/creack/pty v1.1.11/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= @@ -104,37 +103,27 @@ github.com/gin-contrib/cors v1.4.0/go.mod h1:bs9pNM0x/UsmHPBWT2xZz9ROh8xYjYkiURU github.com/gin-contrib/gzip v0.0.1 h1:ezvKOL6jH+jlzdHNE4h9h8q8uMpDQjyl0NN0Jd7jozc= github.com/gin-contrib/gzip v0.0.1/go.mod h1:fGBJBCdt6qCZuCAOwWuFhBB4OOq9EFqlo5dEaFhhu5w= github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= -github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE= github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI= github.com/gin-gonic/gin v1.3.0/go.mod h1:7cKuhb5qV2ggCFctp2fJQ+ErvciLZrIeoOSOm6mUr7Y= -github.com/gin-gonic/gin v1.4.0/go.mod h1:OW2EZn3DO8Ln9oIKOvM++LBO+5UPHJJDH72/q/3rZdM= github.com/gin-gonic/gin v1.6.3/go.mod h1:75u5sXoLsGZoRN5Sgbi1eraJ4GU3++wFwWzhwvtwp4M= github.com/gin-gonic/gin v1.8.1 h1:4+fr/el88TOO3ewCmQr8cx/CtZ/umlIRIs5M4NTNjf8= github.com/gin-gonic/gin v1.8.1/go.mod h1:ji8BvRH1azfM+SYow9zQ6SZMvR8qOMZHmsCuWR9tTTk= -github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAUcHlgXVRs= -github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY= github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= -github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM= github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= -github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= -github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= -github.com/go-openapi/jsonreference v0.19.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= github.com/go-openapi/jsonreference v0.19.6 h1:UBIxjkht+AWIgYzCDSv2GN+E/togfwXUJFRTWhl2Jjs= github.com/go-openapi/jsonreference v0.19.6/go.mod h1:diGHMEHg2IqXZGKxqyvWdfWU/aim5Dprw5bqpKkTvns= -github.com/go-openapi/spec v0.19.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= github.com/go-openapi/spec v0.20.4 h1:O8hJrt0UMnhHcluhIdUgCLRWyM2x7QkBXRvOs7m+O1M= github.com/go-openapi/spec v0.20.4/go.mod h1:faYFR1CvsJZ0mNsmsphTMSoRrNV3TEDoAM7FOEWeq8I= -github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= github.com/go-openapi/swag v0.19.15 h1:D2NRCBzS9/pEY3gP9Nl8aDqGUcPFrwG2p+CNFrLyrCM= github.com/go-openapi/swag v0.19.15/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= @@ -216,7 +205,7 @@ github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= +github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/mux v1.7.4 h1:VuZ8uybHlWmqV03+zRzdwKL4tUnIp1MAQtp1mIFE1bc= github.com/gorilla/mux v1.7.4/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= @@ -279,6 +268,7 @@ github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/ github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= @@ -302,15 +292,12 @@ github.com/leodido/go-urn v1.2.1/go.mod h1:zt4jvISO2HfUBqxjfIshjdMTYS56ZS/qv49ic github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4= github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= -github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA= github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= -github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= -github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= @@ -348,6 +335,11 @@ github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1Cpa github.com/onsi/gomega v1.20.1 h1:PA/3qinGoukvymdIDV8pii6tiZgC8kbmJO6Z5+b002Q= github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs= github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc= +github.com/otiai10/copy v1.7.0/go.mod h1:rmRl6QPdJj6EiUqXQ/4Nn2lLXoNQjFCQbbNrxgc/t3U= +github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE= +github.com/otiai10/curr v1.0.0/go.mod h1:LskTG5wDwr8Rs+nNQ+1LlxRjAtTZZjtJW4rMXl6j4vs= +github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo= +github.com/otiai10/mint v1.3.3/go.mod h1:/yxELlJQ0ufhjUwhshSj+wFjZ78CnZ48/1wtmBH1OTc= github.com/pascaldekloe/name v0.0.0-20180628100202-0fd16699aae1/go.mod h1:eD5JxqMiuNYyFNmyY9rkJ/slN8y59oEu4Ei7F8OoKWQ= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml/v2 v2.0.1 h1:8e3L2cCQzLFi2CR4g7vGFuFxX7Jl1kKX8gW+iV0GUKU= @@ -356,8 +348,6 @@ github.com/petermattis/goid v0.0.0-20211229010228-4d14c490ee36 h1:64bxqeTEN0/xoE github.com/petermattis/goid v0.0.0-20211229010228-4d14c490ee36/go.mod h1:pxMtw7cyUw6B2bRH0ZBANSPg+AoSud1I1iyJHI69jH4= github.com/phf/go-queue v0.0.0-20170504031614-9abe38d0371d h1:U+PMnTlV2tu7RuMK5etusZG3Cf+rpow5hqQByeCzJ2g= github.com/phf/go-queue v0.0.0-20170504031614-9abe38d0371d/go.mod h1:lXfE4PvvTW5xOjO6Mba8zDPyw8M93B6AQ7frTGnMlA8= -github.com/pingcap/check v0.0.0-20190102082844-67f458068fc8/go.mod h1:B1+S9LNcuMyLH/4HMTViQOJevkGiik3wW2AN9zb2fNQ= -github.com/pingcap/check v0.0.0-20191107115940-caf2b9e6ccf4/go.mod h1:PYMCGwN0JHjoqGr3HrZoD+b8Tgx8bKnArhSq8YVzUMc= github.com/pingcap/check v0.0.0-20191216031241-8a5a85928f12 h1:rfD9v3+ppLPzoQBgZev0qYCpegrwyFx/BUpkApEiKdY= github.com/pingcap/errcode v0.3.0 h1:IF6LC/4+b1KNwrMlr2rBTUrojFPMexXBcDWZSpNwxjg= github.com/pingcap/errcode v0.3.0/go.mod h1:4b2X8xSqxIroj/IZ9MX/VGZhAwc11wB9wRIzHvz6SeM= @@ -369,14 +359,13 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ue github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 h1:C3N3itkduZXDZFh4N3vQ5HEtld3S+Y+StULhWVvumU0= github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00/go.mod h1:4qGtCB0QK0wBzKtFEGDhxXnSnbQApw1gc9siScUl8ew= github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba h1:7g2yM0llENlRqtjboBKFBJ8N9SE01hPDpKuTwxBLpLM= -github.com/pingcap/kvproto v0.0.0-20230321060725-1841520d34ba/go.mod h1:RjuuhxITxwATlt5adgTedg3ehKk01M03L1U4jNHdeeQ= -github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 h1:VXQ6Du/nKZ9IQnI9NWMzKbftWu8NV5pQkSLKIRzzGN4= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1/go.mod h1:guCyM5N+o+ru0TsoZ1hi9lDjUMs2sIBjW3ARTEpVbnk= github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= -github.com/pingcap/sysutil v0.0.0-20211208032423-041a72e5860d h1:k3/APKZjXOyJrFy8VyYwRlZhMelpD3qBLJNsw3bPl/g= -github.com/pingcap/sysutil v0.0.0-20211208032423-041a72e5860d/go.mod h1:7j18ezaWTao2LHOyMlsc2Dg1vW+mDY9dEbPzVyOlaeM= +github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21 h1:QV6jqlfOkh8hqvEAgwBZa+4bSgO0EeKC7s5c6Luam2I= +github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21/go.mod h1:QYnjfA95ZaMefyl1NO8oPtKeb8pYUdnDVhQgf+qdpjM= github.com/pingcap/tidb-dashboard v0.0.0-20230209052558-a58fc2a7e924 h1:49x3JR5zEYqjVqONKV9r/nrv0Rh5QU8ivIhktoLvP4g= github.com/pingcap/tidb-dashboard v0.0.0-20230209052558-a58fc2a7e924/go.mod h1:OUzFMMVjR1GKlf4LWLqza9QNKjCrYJ7stVn/3PN0djM= github.com/pingcap/tipb v0.0.0-20220718022156-3e2483c20a9e h1:FBaTXU8C3xgt/drM58VHxojHo/QoG1oPsgWTGvaSpO4= @@ -428,8 +417,7 @@ github.com/samber/lo v1.37.0/go.mod h1:9vaz2O4o8oOnK23pd2TrXufcbdbJIa3b6cstBWKpo github.com/sasha-s/go-deadlock v0.2.0 h1:lMqc+fUb7RrFS3gQLtoQsJ7/6TV/pAIFvBsqX73DK8Y= github.com/sasha-s/go-deadlock v0.2.0/go.mod h1:StQn567HiB1fF2yJ44N9au7wOhrPS3iZqiDbRupzT10= github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= -github.com/shirou/gopsutil v3.21.3+incompatible h1:uenXGGa8ESCQq+dbgtl916dmg6PSAz2cXov0uORQ9v8= -github.com/shirou/gopsutil v3.21.3+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA= +github.com/shirou/gopsutil/v3 v3.21.12/go.mod h1:BToYZVTlSVlfazpDDYFnsVZLaoRG+g8ufT6fPQLdJzA= github.com/shirou/gopsutil/v3 v3.22.12 h1:oG0ns6poeUSxf78JtOsfygNWuEHYYz8hnnNg7P04TJs= github.com/shirou/gopsutil/v3 v3.22.12/go.mod h1:Xd7P1kwZcp5VW52+9XsirIKd/BROzbb2wdX3Kqlz9uI= github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 h1:mj/nMDAwTBiaCqMEs4cYCqF7pO6Np7vhy1D1wcQGz+E= @@ -441,6 +429,8 @@ github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072 h1:Txo4SXVJq/OgEjwgkWoxkMoTjGlcrgsQE/XSghjmu0w= github.com/smallnest/chanx v0.0.0-20221229104322-eb4c998d2072/go.mod h1:+4nWMF0+CqEcU74SnX2NxaGqZ8zX4pcQ8Jcs77DbX5A= +github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= +github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= @@ -471,13 +461,11 @@ github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 h1:PyYN9JH5jY9j6av01SpfRMb+1DWg/i3MbGOKPxJ2wjM= -github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14/go.mod h1:gxQT6pBGRuIGunNf/+tSOB5OHvguWi8Tbt82WOkf35E= -github.com/swaggo/gin-swagger v1.2.0/go.mod h1:qlH2+W7zXGZkczuL+r2nEBR2JTT+/lX05Nn6vPhc7OI= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba h1:lUPlXKqgbqT2SVg2Y+eT9mu5wbqMnG+i/+Q9nK7C0Rs= -github.com/swaggo/http-swagger v0.0.0-20200308142732-58ac5e232fba/go.mod h1:O1lAbCgAAX/KZ80LM/OXwtWFI/5TvZlwxSg8Cq08PV0= -github.com/swaggo/swag v1.5.1/go.mod h1:1Bl9F/ZBpVWh22nY0zmYyASPO1lI/zIwRDrpZU+tv8Y= -github.com/swaggo/swag v1.6.3/go.mod h1:wcc83tB4Mb2aNiL/HP4MFeQdpHUrca+Rp/DRNgWAUio= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 h1:+iNTcqQJy0OZ5jk6a5NLib47eqXK8uYcPX+O4+cBpEM= +github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2/go.mod h1:lKJPbtWzJ9JhsTN1k1gZgleJWY/cqq0psdoMmaThG3w= +github.com/swaggo/http-swagger v1.2.6 h1:ihTjChUoSRMpFMjWw+0AkL1Ti4r6v8pCgVYLmQVRlRw= +github.com/swaggo/http-swagger v1.2.6/go.mod h1:CcoICgY3yVDk2u1LQUCMHbAj0fjlxIX+873psXlIKNA= +github.com/swaggo/swag v1.7.9/go.mod h1:gZ+TJ2w/Ve1RwQsA2IRoSOTidHz6DX+PIG8GWvbnoLU= github.com/swaggo/swag v1.8.3 h1:3pZSSCQ//gAH88lfmxM3Cd1+JCsxV8Md6f36b9hrZ5s= github.com/swaggo/swag v1.8.3/go.mod h1:jMLeXOOmYyjk8PvHTsXBdrubsNd9gUJTTCzL5iBnseg= github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 h1:1oFLiOyVl+W7bnBzGhf7BbIv9loSFQcieWWYIjLqcAw= @@ -488,27 +476,26 @@ github.com/tidwall/match v1.0.1 h1:PnKP62LPNxHKTwvHHZZzdOAOCtsJTjo6dZLCwpKm5xc= github.com/tidwall/match v1.0.1/go.mod h1:LujAq0jyVjBy028G1WhWfIzbpQfMO8bBZ6Tyb0+pL9E= github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4= github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= -github.com/tklauser/go-sysconf v0.3.4/go.mod h1:Cl2c8ZRWfHD5IrfHo9VN+FX9kCFjIOyVklgXycLB6ek= +github.com/tklauser/go-sysconf v0.3.9/go.mod h1:11DU/5sG7UexIrp/O6g35hrWzu0JxlwQ3LSFUzyeuhs= github.com/tklauser/go-sysconf v0.3.11 h1:89WgdJhk5SNwJfu+GKyYveZ4IaJ7xAkecBo+KdJV0CM= github.com/tklauser/go-sysconf v0.3.11/go.mod h1:GqXfhXY3kiPa0nAXPDIQIWzJbMCB7AmcWpGR8lSZfqI= -github.com/tklauser/numcpus v0.2.1/go.mod h1:9aU+wOc6WjUIZEwWMP62PL/41d65P+iks1gBkr4QyP8= +github.com/tklauser/numcpus v0.3.0/go.mod h1:yFGUr7TUHQRAhyqBcEg0Ge34zDBAsIvJJcyE6boqnA8= github.com/tklauser/numcpus v0.6.0 h1:kebhY2Qt+3U6RNK7UqpYNA+tJ23IBEGKkB7JQBfDYms= github.com/tklauser/numcpus v0.6.0/go.mod h1:FEZLMke0lhOUG6w2JadTzp0a+Nl8PF/GFkQ5UVIcaL4= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966 h1:j6JEOq5QWFker+d7mFQYOhjTZonQ7YkLTHm56dbn+yM= github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= -github.com/ugorji/go v1.1.5-pre/go.mod h1:FwP/aQVg39TXzItUBMwnWp9T9gPQnXw4Poh4/oBQZ/0= github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw= github.com/ugorji/go v1.2.7/go.mod h1:nF9osbDWLy6bDVv/Rtoh6QgnvNDpmCalQV5urGCCS6M= github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= -github.com/ugorji/go/codec v1.1.5-pre/go.mod h1:tULtS6Gy1AE1yCENaw4Vb//HLH5njI2tfCQDUqRd8fI= github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY= github.com/ugorji/go/codec v1.2.7 h1:YPXUKf7fYbp/y8xloBqZOw2qaVggbfwMlI8WM3wZUJ0= github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95kRgeqEY= github.com/unrolled/render v1.0.1 h1:VDDnQQVfBMsOsp3VaCJszSO0nkBIVEYoPWeRThk9spY= github.com/unrolled/render v1.0.1/go.mod h1:gN9T0NhL4Bfbwu8ann7Ry/TGHYfosul+J0obPf6NBdM= github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA= +github.com/urfave/cli/v2 v2.3.0/go.mod h1:LJmUH05zAU44vOAcrfzZQKsZbVcdbOG8rtL3/XcUArI= github.com/urfave/negroni v0.3.0 h1:PaXOb61mWeZJxc1Ji2xJjpVg9QfPo0rrB+lHyBxGNSU= github.com/urfave/negroni v0.3.0/go.mod h1:Meg73S6kFm/4PpbYdq35yYWoCZ9mS/YSx+lKnmiohz4= github.com/vmihailenco/msgpack/v5 v5.3.5 h1:5gO0H1iULLWGhs2H5tbAHIZTV8/cYafcFOr9znI5mJU= @@ -521,6 +508,8 @@ github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1: github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg= github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= @@ -568,14 +557,16 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20200204104054-c9f3fb736b72/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU= golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a h1:tlXy25amD5A7gOfbXdqCGN5k8ESEed/Ee1E5RcrYnqU= golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= -golang.org/x/image v0.0.0-20200119044424-58c23975cae1 h1:5h3ngYt7+vXCDZCup/HkCQgW5XwmSvR/nA2JmJ0RErg= golang.org/x/image v0.0.0-20200119044424-58c23975cae1/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= +golang.org/x/image v0.5.0 h1:5JMiNunQeQw++mMOz48/ISeNu3Iweh/JaZU8ZLqHRrI= +golang.org/x/image v0.5.0/go.mod h1:FVC7BI/5Ym8R25iw5OLsgshdUBbT1h5jZTpA+mvAdZ4= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= @@ -587,7 +578,8 @@ golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.6.0 h1:b9gGHsz9/HhJ3HF5DHQytPpuwocVTChQJK3AvoLRD5I= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -598,10 +590,8 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= -golang.org/x/net v0.0.0-20190611141213-3f473d35a33a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= @@ -613,14 +603,16 @@ golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwY golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM= +golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.2.0 h1:sZfSu1wtKLGlWI4ZZayP0ck9Y73K1ynO6gqzTdBVdPU= -golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 h1:nt+Q6cXKz4MosCSpnbMtqiQ8Oz0pxTef2B4Vca2lvfk= -golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg= +golang.org/x/oauth2 v0.4.0 h1:NF0gk8LVPg1Ml7SSbGyySuoxdsXitj7TvgvuRxIMc/M= +golang.org/x/oauth2 v0.4.0/go.mod h1:RznEsdpjGAINPTOF0UH/t+xJ75L18YO3Ho6Pyn+uRec= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -629,6 +621,7 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -638,10 +631,8 @@ golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181228144115-9a3f9b0469bb/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190610200419-93c9922d18ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -653,7 +644,6 @@ golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210217105451-b926d437f341/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210420072515-93ed5bcd2bfe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -662,19 +652,27 @@ golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210816074244-15123e1e1f71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211013075003-97ac67df715c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ= golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg= -golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.1.0 h1:xYY+Bajn2a7VBmTM5GikTmnK8ZuX8YgnQCqZpbBNtmA= @@ -684,15 +682,13 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190524210228-3d17549cdc6b/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.0.0-20190606050223-4d9ae51c2468/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190611222205-d73e1c7e250b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191030062658-86caa796c7ab/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191107010934-f79515f33823/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191108193012-7d206e10da11/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191114200427-caa0b0f7d508/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= @@ -702,8 +698,10 @@ golang.org/x/tools v0.0.0-20201125231158-b5590deeca9b/go.mod h1:emZCQorbCU4vsT4f golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210112230658-8b4aab62c064/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.2.0 h1:G6AHpWxTMGY1KyEYoAQ5WTtIekUUvDNjan3ugu60JvE= -golang.org/x/tools v0.2.0/go.mod h1:y4OqIKeOV/fWJetJ8bXPU1sEVniLMIyDAZWeHdV+NTA= +golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= diff --git a/tests/tso/server_test.go b/tests/integrations/tso/server_test.go similarity index 92% rename from tests/tso/server_test.go rename to tests/integrations/tso/server_test.go index edf9585e6a8..96fc5d334b0 100644 --- a/tests/tso/server_test.go +++ b/tests/integrations/tso/server_test.go @@ -28,7 +28,7 @@ import ( "github.com/tikv/pd/pkg/utils/tempurl" pd "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/tests" - "github.com/tikv/pd/tests/mcs" + "github.com/tikv/pd/tests/integrations/mcs" "google.golang.org/grpc" ) @@ -107,9 +107,9 @@ func (suite *tsoServerTestSuite) getClusterID() uint64 { func (suite *tsoServerTestSuite) resetTS(ts uint64, ignoreSmaller, skipUpperBoundCheck bool) { var err error if suite.legacy { - err = suite.pdLeaderServer.GetServer().GetHandler().ResetTS(ts, ignoreSmaller, skipUpperBoundCheck) + err = suite.pdLeaderServer.GetServer().GetHandler().ResetTS(ts, ignoreSmaller, skipUpperBoundCheck, 0) } else { - err = suite.tsoServer.GetHandler().ResetTS(ts, ignoreSmaller, skipUpperBoundCheck) + err = suite.tsoServer.GetHandler().ResetTS(ts, ignoreSmaller, skipUpperBoundCheck, 0) } // Only this error is acceptable. if err != nil { @@ -154,7 +154,10 @@ func (suite *tsoServerTestSuite) TestConcurrentlyReset() { go func() { defer wg.Done() for j := 0; j <= 100; j++ { - physical := now.Add(time.Duration(2*j)*time.Minute).UnixNano() / int64(time.Millisecond) + // Get a copy of now then call base.add, because now is shared by all goroutines + // and now.add() will add to itself which isn't atomic and multi-goroutine safe. + base := now + physical := base.Add(time.Duration(2*j)*time.Minute).UnixNano() / int64(time.Millisecond) ts := uint64(physical << 18) suite.resetTS(ts, false, false) } diff --git a/tests/tso/testutil.go b/tests/integrations/tso/testutil.go similarity index 97% rename from tests/tso/testutil.go rename to tests/integrations/tso/testutil.go index 55e24fd1e58..2a4e5eabd90 100644 --- a/tests/tso/testutil.go +++ b/tests/integrations/tso/testutil.go @@ -22,7 +22,7 @@ import ( const ( serverCount = 3 tsoRequestConcurrencyNumber = 5 - tsoRequestRound = 30 + tsoRequestRound = 300 tsoCount = 10 ) diff --git a/tests/mcs/testutil.go b/tests/mcs/testutil.go deleted file mode 100644 index 13362e9b146..00000000000 --- a/tests/mcs/testutil.go +++ /dev/null @@ -1,114 +0,0 @@ -// Copyright 2023 TiKV Project Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package mcs - -import ( - "context" - "time" - - "github.com/pkg/errors" - rm "github.com/tikv/pd/pkg/mcs/resource_manager/server" - tso "github.com/tikv/pd/pkg/mcs/tso/server" - "github.com/tikv/pd/pkg/mcs/utils" - "github.com/tikv/pd/pkg/utils/testutil" - - "github.com/stretchr/testify/require" - pd "github.com/tikv/pd/client" - bs "github.com/tikv/pd/pkg/basicserver" -) - -// SetupTSOClient creates a TSO client for test. -func SetupTSOClient(ctx context.Context, re *require.Assertions, endpoints []string, opts ...pd.ClientOption) pd.Client { - cli, err := pd.NewClientWithKeyspace(ctx, utils.DefaultKeyspaceID, endpoints, pd.SecurityOption{}, opts...) - re.NoError(err) - return cli -} - -// StartSingleResourceManagerTestServer creates and starts a resource manager server with default config for testing. -func StartSingleResourceManagerTestServer(ctx context.Context, re *require.Assertions, backendEndpoints, listenAddrs string) (*rm.Server, func()) { - cfg := rm.NewConfig() - cfg.BackendEndpoints = backendEndpoints - cfg.ListenAddr = listenAddrs - cfg, err := rm.GenerateConfig(cfg) - re.NoError(err) - - s, cleanup, err := rm.NewTestServer(ctx, re, cfg) - re.NoError(err) - testutil.Eventually(re, func() bool { - return !s.IsClosed() - }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) - - return s, cleanup -} - -// StartSingleTSOTestServer creates and starts a tso server with default config for testing. -func StartSingleTSOTestServer(ctx context.Context, re *require.Assertions, backendEndpoints, listenAddrs string) (*tso.Server, func()) { - cfg := tso.NewConfig() - cfg.BackendEndpoints = backendEndpoints - cfg.ListenAddr = listenAddrs - cfg, err := tso.GenerateConfig(cfg) - re.NoError(err) - - s, cleanup, err := tso.NewTSOTestServer(ctx, re, cfg) - re.NoError(err) - testutil.Eventually(re, func() bool { - return !s.IsClosed() - }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) - - return s, cleanup -} - -// StartSingleTSOTestServerWithoutCheck creates and starts a tso server with default config for testing. -func StartSingleTSOTestServerWithoutCheck(ctx context.Context, re *require.Assertions, backendEndpoints, listenAddrs string) (*tso.Server, func()) { - cfg := tso.NewConfig() - cfg.BackendEndpoints = backendEndpoints - cfg.ListenAddr = listenAddrs - cfg, err := tso.GenerateConfig(cfg) - re.NoError(err) - s, cleanup, _ := tso.NewTSOTestServer(ctx, re, cfg) - return s, cleanup -} - -// WaitForPrimaryServing waits for one of servers being elected to be the primary/leader -func WaitForPrimaryServing(re *require.Assertions, serverMap map[string]bs.Server) string { - var primary string - testutil.Eventually(re, func() bool { - for name, s := range serverMap { - if s.IsServing() { - primary = name - return true - } - } - return false - }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(50*time.Millisecond)) - - return primary -} - -// WaitForTSOServiceAvailable waits for the pd client being served by the tso server side -func WaitForTSOServiceAvailable(ctx context.Context, pdClient pd.Client) error { - var err error - for i := 0; i < 30; i++ { - if _, _, err := pdClient.GetTS(ctx); err == nil { - return nil - } - select { - case <-ctx.Done(): - return err - case <-time.After(100 * time.Millisecond): - } - } - return errors.WithStack(err) -} diff --git a/tests/pdctl/keyspace/keyspace_group_test.go b/tests/pdctl/keyspace/keyspace_group_test.go new file mode 100644 index 00000000000..7c0862a5d96 --- /dev/null +++ b/tests/pdctl/keyspace/keyspace_group_test.go @@ -0,0 +1,84 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package keyspace_test + +import ( + "context" + "encoding/json" + "fmt" + "testing" + + "github.com/stretchr/testify/require" + "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/server/apiv2/handlers" + "github.com/tikv/pd/tests" + "github.com/tikv/pd/tests/pdctl" + handlersutil "github.com/tikv/pd/tests/server/apiv2/handlers" + pdctlCmd "github.com/tikv/pd/tools/pd-ctl/pdctl" +) + +func TestKeyspaceGroup(t *testing.T) { + re := require.New(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + tc, err := tests.NewTestAPICluster(ctx, 1) + re.NoError(err) + err = tc.RunInitialServers() + re.NoError(err) + tc.WaitLeader() + leaderServer := tc.GetServer(tc.GetLeader()) + re.NoError(leaderServer.BootstrapCluster()) + pdAddr := tc.GetConfig().GetClientURL() + cmd := pdctlCmd.GetRootCmd() + + // Show keyspace group information. + defaultKeyspaceGroupID := fmt.Sprintf("%d", utils.DefaultKeyspaceGroupID) + args := []string{"-u", pdAddr, "keyspace-group"} + output, err := pdctl.ExecuteCommand(cmd, append(args, defaultKeyspaceGroupID)...) + re.NoError(err) + var keyspaceGroup endpoint.KeyspaceGroup + err = json.Unmarshal(output, &keyspaceGroup) + re.NoError(err) + re.Equal(utils.DefaultKeyspaceGroupID, keyspaceGroup.ID) + re.Contains(keyspaceGroup.Keyspaces, utils.DefaultKeyspaceID) + // Split keyspace group. + handlersutil.MustCreateKeyspaceGroup(re, leaderServer, &handlers.CreateKeyspaceGroupParams{ + KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: 1, + UserKind: endpoint.Standard.String(), + Members: make([]endpoint.KeyspaceGroupMember, utils.KeyspaceGroupDefaultReplicaCount), + Keyspaces: []uint32{111, 222, 333}, + }, + }, + }) + _, err = pdctl.ExecuteCommand(cmd, append(args, "split", "1", "2", "222", "333")...) + re.NoError(err) + output, err = pdctl.ExecuteCommand(cmd, append(args, "1")...) + re.NoError(err) + keyspaceGroup = endpoint.KeyspaceGroup{} + err = json.Unmarshal(output, &keyspaceGroup) + re.NoError(err) + re.Equal(uint32(1), keyspaceGroup.ID) + re.Equal(keyspaceGroup.Keyspaces, []uint32{111}) + output, err = pdctl.ExecuteCommand(cmd, append(args, "2")...) + re.NoError(err) + keyspaceGroup = endpoint.KeyspaceGroup{} + err = json.Unmarshal(output, &keyspaceGroup) + re.NoError(err) + re.Equal(uint32(2), keyspaceGroup.ID) + re.Equal(keyspaceGroup.Keyspaces, []uint32{222, 333}) +} diff --git a/tests/server/apiv2/handlers/keyspace_test.go b/tests/server/apiv2/handlers/keyspace_test.go index e13e13737d7..ad456bd5be6 100644 --- a/tests/server/apiv2/handlers/keyspace_test.go +++ b/tests/server/apiv2/handlers/keyspace_test.go @@ -12,36 +12,24 @@ // See the License for the specific language governing permissions and // limitations under the License. -package handlers_test +package handlers import ( - "bytes" "context" - "encoding/json" "fmt" - "io" - "net/http" "testing" + "github.com/pingcap/failpoint" "github.com/pingcap/kvproto/pkg/keyspacepb" "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" + "github.com/tikv/pd/pkg/mcs/utils" "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/server/apiv2/handlers" - "github.com/tikv/pd/server/keyspace" "github.com/tikv/pd/tests" "go.uber.org/goleak" ) -const keyspacesPrefix = "/pd/api/v2/keyspaces" - -// dialClient used to dial http request. -var dialClient = &http.Client{ - Transport: &http.Transport{ - DisableKeepAlives: true, - }, -} - func TestMain(m *testing.M) { goleak.VerifyTestMain(m, testutil.LeakOptions...) } @@ -67,11 +55,13 @@ func (suite *keyspaceTestSuite) SetupTest() { suite.NotEmpty(cluster.WaitLeader()) suite.server = cluster.GetServer(cluster.GetLeader()) suite.NoError(suite.server.BootstrapCluster()) + suite.NoError(failpoint.Enable("github.com/tikv/pd/pkg/keyspace/skipSplitRegion", "return(true)")) } func (suite *keyspaceTestSuite) TearDownTest() { suite.cleanup() suite.cluster.Destroy() + suite.NoError(failpoint.Disable("github.com/tikv/pd/pkg/keyspace/skipSplitRegion")) } func (suite *keyspaceTestSuite) TestCreateLoadKeyspace() { @@ -81,8 +71,8 @@ func (suite *keyspaceTestSuite) TestCreateLoadKeyspace() { loaded := mustLoadKeyspaces(re, suite.server, created.Name) re.Equal(created, loaded) } - defaultKeyspace := mustLoadKeyspaces(re, suite.server, keyspace.DefaultKeyspaceName) - re.Equal(keyspace.DefaultKeyspaceName, defaultKeyspace.Name) + defaultKeyspace := mustLoadKeyspaces(re, suite.server, utils.DefaultKeyspaceName) + re.Equal(utils.DefaultKeyspaceName, defaultKeyspace.Name) re.Equal(keyspacepb.KeyspaceState_ENABLED, defaultKeyspace.State) } @@ -133,7 +123,7 @@ func (suite *keyspaceTestSuite) TestUpdateKeyspaceState() { re.Equal(keyspacepb.KeyspaceState_TOMBSTONE, tombstone.State) } // Changing default keyspace's state is NOT allowed. - success, _ := sendUpdateStateRequest(re, suite.server, keyspace.DefaultKeyspaceName, &handlers.UpdateStateParam{State: "disabled"}) + success, _ := sendUpdateStateRequest(re, suite.server, utils.DefaultKeyspaceName, &handlers.UpdateStateParam{State: "disabled"}) re.False(success) } @@ -147,48 +137,10 @@ func (suite *keyspaceTestSuite) TestLoadRangeKeyspace() { for i, created := range keyspaces { re.Equal(created, loadResponse.Keyspaces[i+1].KeyspaceMeta) } - re.Equal(keyspace.DefaultKeyspaceName, loadResponse.Keyspaces[0].Name) + re.Equal(utils.DefaultKeyspaceName, loadResponse.Keyspaces[0].Name) re.Equal(keyspacepb.KeyspaceState_ENABLED, loadResponse.Keyspaces[0].State) } -func sendLoadRangeRequest(re *require.Assertions, server *tests.TestServer, token, limit string) *handlers.LoadAllKeyspacesResponse { - // Construct load range request. - httpReq, err := http.NewRequest(http.MethodGet, server.GetAddr()+keyspacesPrefix, nil) - re.NoError(err) - query := httpReq.URL.Query() - query.Add("page_token", token) - query.Add("limit", limit) - httpReq.URL.RawQuery = query.Encode() - // Send request. - httpResp, err := dialClient.Do(httpReq) - re.NoError(err) - defer httpResp.Body.Close() - re.Equal(http.StatusOK, httpResp.StatusCode) - // Receive & decode response. - data, err := io.ReadAll(httpResp.Body) - re.NoError(err) - resp := &handlers.LoadAllKeyspacesResponse{} - re.NoError(json.Unmarshal(data, resp)) - return resp -} - -func sendUpdateStateRequest(re *require.Assertions, server *tests.TestServer, name string, request *handlers.UpdateStateParam) (bool, *keyspacepb.KeyspaceMeta) { - data, err := json.Marshal(request) - re.NoError(err) - httpReq, err := http.NewRequest(http.MethodPut, server.GetAddr()+keyspacesPrefix+"/"+name+"/state", bytes.NewBuffer(data)) - re.NoError(err) - httpResp, err := dialClient.Do(httpReq) - re.NoError(err) - defer httpResp.Body.Close() - if httpResp.StatusCode != http.StatusOK { - return false, nil - } - data, err = io.ReadAll(httpResp.Body) - re.NoError(err) - meta := &handlers.KeyspaceMeta{} - re.NoError(json.Unmarshal(data, meta)) - return true, meta.KeyspaceMeta -} func mustMakeTestKeyspaces(re *require.Assertions, server *tests.TestServer, count int) []*keyspacepb.KeyspaceMeta { testConfig := map[string]string{ "config1": "100", @@ -197,7 +149,7 @@ func mustMakeTestKeyspaces(re *require.Assertions, server *tests.TestServer, cou resultMeta := make([]*keyspacepb.KeyspaceMeta, count) for i := 0; i < count; i++ { createRequest := &handlers.CreateKeyspaceParams{ - Name: fmt.Sprintf("test_keyspace%d", i), + Name: fmt.Sprintf("test_keyspace_%d", i), Config: testConfig, } resultMeta[i] = mustCreateKeyspace(re, server, createRequest) @@ -205,58 +157,6 @@ func mustMakeTestKeyspaces(re *require.Assertions, server *tests.TestServer, cou return resultMeta } -func mustCreateKeyspace(re *require.Assertions, server *tests.TestServer, request *handlers.CreateKeyspaceParams) *keyspacepb.KeyspaceMeta { - data, err := json.Marshal(request) - re.NoError(err) - httpReq, err := http.NewRequest(http.MethodPost, server.GetAddr()+keyspacesPrefix, bytes.NewBuffer(data)) - re.NoError(err) - resp, err := dialClient.Do(httpReq) - re.NoError(err) - defer resp.Body.Close() - re.Equal(http.StatusOK, resp.StatusCode) - data, err = io.ReadAll(resp.Body) - re.NoError(err) - meta := &handlers.KeyspaceMeta{} - re.NoError(json.Unmarshal(data, meta)) - checkCreateRequest(re, request, meta.KeyspaceMeta) - return meta.KeyspaceMeta -} - -func mustUpdateKeyspaceConfig(re *require.Assertions, server *tests.TestServer, name string, request *handlers.UpdateConfigParams) *keyspacepb.KeyspaceMeta { - data, err := json.Marshal(request) - re.NoError(err) - httpReq, err := http.NewRequest(http.MethodPatch, server.GetAddr()+keyspacesPrefix+"/"+name+"/config", bytes.NewBuffer(data)) - re.NoError(err) - resp, err := dialClient.Do(httpReq) - re.NoError(err) - defer resp.Body.Close() - re.Equal(http.StatusOK, resp.StatusCode) - data, err = io.ReadAll(resp.Body) - re.NoError(err) - meta := &handlers.KeyspaceMeta{} - re.NoError(json.Unmarshal(data, meta)) - return meta.KeyspaceMeta -} - -func mustLoadKeyspaces(re *require.Assertions, server *tests.TestServer, name string) *keyspacepb.KeyspaceMeta { - resp, err := dialClient.Get(server.GetAddr() + keyspacesPrefix + "/" + name) - re.NoError(err) - defer resp.Body.Close() - re.Equal(http.StatusOK, resp.StatusCode) - data, err := io.ReadAll(resp.Body) - re.NoError(err) - meta := &handlers.KeyspaceMeta{} - re.NoError(json.Unmarshal(data, meta)) - return meta.KeyspaceMeta -} - -// checkCreateRequest verifies a keyspace meta matches a create request. -func checkCreateRequest(re *require.Assertions, request *handlers.CreateKeyspaceParams, meta *keyspacepb.KeyspaceMeta) { - re.Equal(request.Name, meta.Name) - re.Equal(keyspacepb.KeyspaceState_ENABLED, meta.State) - re.Equal(request.Config, meta.Config) -} - // checkUpdateRequest verifies a keyspace meta matches a update request. func checkUpdateRequest(re *require.Assertions, request *handlers.UpdateConfigParams, oldConfig, newConfig map[string]string) { expected := map[string]string{} diff --git a/tests/server/apiv2/handlers/testutil.go b/tests/server/apiv2/handlers/testutil.go new file mode 100644 index 00000000000..3ab41bfb0cc --- /dev/null +++ b/tests/server/apiv2/handlers/testutil.go @@ -0,0 +1,233 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package handlers + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + + "github.com/pingcap/kvproto/pkg/keyspacepb" + "github.com/stretchr/testify/require" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/server/apiv2/handlers" + "github.com/tikv/pd/tests" +) + +const ( + keyspacesPrefix = "/pd/api/v2/keyspaces" + keyspaceGroupsPrefix = "/pd/api/v2/tso/keyspace-groups" +) + +// dialClient used to dial http request. +var dialClient = &http.Client{ + Transport: &http.Transport{ + DisableKeepAlives: true, + }, +} + +func sendLoadRangeRequest(re *require.Assertions, server *tests.TestServer, token, limit string) *handlers.LoadAllKeyspacesResponse { + // Construct load range request. + httpReq, err := http.NewRequest(http.MethodGet, server.GetAddr()+keyspacesPrefix, nil) + re.NoError(err) + query := httpReq.URL.Query() + query.Add("page_token", token) + query.Add("limit", limit) + httpReq.URL.RawQuery = query.Encode() + // Send request. + httpResp, err := dialClient.Do(httpReq) + re.NoError(err) + defer httpResp.Body.Close() + re.Equal(http.StatusOK, httpResp.StatusCode) + // Receive & decode response. + data, err := io.ReadAll(httpResp.Body) + re.NoError(err) + resp := &handlers.LoadAllKeyspacesResponse{} + re.NoError(json.Unmarshal(data, resp)) + return resp +} + +func sendUpdateStateRequest(re *require.Assertions, server *tests.TestServer, name string, request *handlers.UpdateStateParam) (bool, *keyspacepb.KeyspaceMeta) { + data, err := json.Marshal(request) + re.NoError(err) + httpReq, err := http.NewRequest(http.MethodPut, server.GetAddr()+keyspacesPrefix+"/"+name+"/state", bytes.NewBuffer(data)) + re.NoError(err) + httpResp, err := dialClient.Do(httpReq) + re.NoError(err) + defer httpResp.Body.Close() + if httpResp.StatusCode != http.StatusOK { + return false, nil + } + data, err = io.ReadAll(httpResp.Body) + re.NoError(err) + meta := &handlers.KeyspaceMeta{} + re.NoError(json.Unmarshal(data, meta)) + return true, meta.KeyspaceMeta +} + +func mustCreateKeyspace(re *require.Assertions, server *tests.TestServer, request *handlers.CreateKeyspaceParams) *keyspacepb.KeyspaceMeta { + data, err := json.Marshal(request) + re.NoError(err) + httpReq, err := http.NewRequest(http.MethodPost, server.GetAddr()+keyspacesPrefix, bytes.NewBuffer(data)) + re.NoError(err) + resp, err := dialClient.Do(httpReq) + re.NoError(err) + defer resp.Body.Close() + re.Equal(http.StatusOK, resp.StatusCode) + data, err = io.ReadAll(resp.Body) + re.NoError(err) + meta := &handlers.KeyspaceMeta{} + re.NoError(json.Unmarshal(data, meta)) + checkCreateRequest(re, request, meta.KeyspaceMeta) + return meta.KeyspaceMeta +} + +// checkCreateRequest verifies a keyspace meta matches a create request. +func checkCreateRequest(re *require.Assertions, request *handlers.CreateKeyspaceParams, meta *keyspacepb.KeyspaceMeta) { + re.Equal(request.Name, meta.Name) + re.Equal(keyspacepb.KeyspaceState_ENABLED, meta.State) + re.Equal(request.Config, meta.Config) +} + +func mustUpdateKeyspaceConfig(re *require.Assertions, server *tests.TestServer, name string, request *handlers.UpdateConfigParams) *keyspacepb.KeyspaceMeta { + data, err := json.Marshal(request) + re.NoError(err) + httpReq, err := http.NewRequest(http.MethodPatch, server.GetAddr()+keyspacesPrefix+"/"+name+"/config", bytes.NewBuffer(data)) + re.NoError(err) + resp, err := dialClient.Do(httpReq) + re.NoError(err) + defer resp.Body.Close() + re.Equal(http.StatusOK, resp.StatusCode) + data, err = io.ReadAll(resp.Body) + re.NoError(err) + meta := &handlers.KeyspaceMeta{} + re.NoError(json.Unmarshal(data, meta)) + return meta.KeyspaceMeta +} + +func mustLoadKeyspaces(re *require.Assertions, server *tests.TestServer, name string) *keyspacepb.KeyspaceMeta { + resp, err := dialClient.Get(server.GetAddr() + keyspacesPrefix + "/" + name) + re.NoError(err) + defer resp.Body.Close() + re.Equal(http.StatusOK, resp.StatusCode) + data, err := io.ReadAll(resp.Body) + re.NoError(err) + meta := &handlers.KeyspaceMeta{} + re.NoError(json.Unmarshal(data, meta)) + return meta.KeyspaceMeta +} + +// MustLoadKeyspaceGroups loads all keyspace groups from the server. +func MustLoadKeyspaceGroups(re *require.Assertions, server *tests.TestServer, token, limit string) []*endpoint.KeyspaceGroup { + // Construct load range request. + httpReq, err := http.NewRequest(http.MethodGet, server.GetAddr()+keyspaceGroupsPrefix, nil) + re.NoError(err) + query := httpReq.URL.Query() + query.Add("page_token", token) + query.Add("limit", limit) + httpReq.URL.RawQuery = query.Encode() + // Send request. + httpResp, err := dialClient.Do(httpReq) + re.NoError(err) + defer httpResp.Body.Close() + data, err := io.ReadAll(httpResp.Body) + re.NoError(err) + re.Equal(http.StatusOK, httpResp.StatusCode, string(data)) + var resp []*endpoint.KeyspaceGroup + re.NoError(json.Unmarshal(data, &resp)) + return resp +} + +func tryCreateKeyspaceGroup(re *require.Assertions, server *tests.TestServer, request *handlers.CreateKeyspaceGroupParams) (int, string) { + data, err := json.Marshal(request) + re.NoError(err) + httpReq, err := http.NewRequest(http.MethodPost, server.GetAddr()+keyspaceGroupsPrefix, bytes.NewBuffer(data)) + re.NoError(err) + resp, err := dialClient.Do(httpReq) + re.NoError(err) + defer resp.Body.Close() + data, err = io.ReadAll(resp.Body) + re.NoError(err) + return resp.StatusCode, string(data) +} + +// MustLoadKeyspaceGroupByID loads the keyspace group by ID with HTTP API. +func MustLoadKeyspaceGroupByID(re *require.Assertions, server *tests.TestServer, id uint32) *endpoint.KeyspaceGroup { + httpReq, err := http.NewRequest(http.MethodGet, server.GetAddr()+keyspaceGroupsPrefix+fmt.Sprintf("/%d", id), nil) + re.NoError(err) + resp, err := dialClient.Do(httpReq) + re.NoError(err) + defer resp.Body.Close() + data, err := io.ReadAll(resp.Body) + re.NoError(err) + re.Equal(http.StatusOK, resp.StatusCode, string(data)) + var kg endpoint.KeyspaceGroup + re.NoError(json.Unmarshal(data, &kg)) + return &kg +} + +// MustCreateKeyspaceGroup creates a keyspace group with HTTP API. +func MustCreateKeyspaceGroup(re *require.Assertions, server *tests.TestServer, request *handlers.CreateKeyspaceGroupParams) { + code, data := tryCreateKeyspaceGroup(re, server, request) + re.Equal(http.StatusOK, code, data) +} + +// FailCreateKeyspaceGroupWithCode fails to create a keyspace group with HTTP API. +func FailCreateKeyspaceGroupWithCode(re *require.Assertions, server *tests.TestServer, request *handlers.CreateKeyspaceGroupParams, expect int) { + code, data := tryCreateKeyspaceGroup(re, server, request) + re.Equal(expect, code, data) +} + +// MustDeleteKeyspaceGroup deletes a keyspace group with HTTP API. +func MustDeleteKeyspaceGroup(re *require.Assertions, server *tests.TestServer, id uint32) { + httpReq, err := http.NewRequest(http.MethodDelete, server.GetAddr()+keyspaceGroupsPrefix+fmt.Sprintf("/%d", id), nil) + re.NoError(err) + resp, err := dialClient.Do(httpReq) + re.NoError(err) + defer resp.Body.Close() + data, err := io.ReadAll(resp.Body) + re.NoError(err) + re.Equal(http.StatusOK, resp.StatusCode, string(data)) +} + +// MustSplitKeyspaceGroup updates a keyspace group with HTTP API. +func MustSplitKeyspaceGroup(re *require.Assertions, server *tests.TestServer, id uint32, request *handlers.SplitKeyspaceGroupByIDParams) { + data, err := json.Marshal(request) + re.NoError(err) + httpReq, err := http.NewRequest(http.MethodPost, server.GetAddr()+keyspaceGroupsPrefix+fmt.Sprintf("/%d/split", id), bytes.NewBuffer(data)) + re.NoError(err) + // Send request. + resp, err := dialClient.Do(httpReq) + re.NoError(err) + defer resp.Body.Close() + data, err = io.ReadAll(resp.Body) + re.NoError(err) + re.Equal(http.StatusOK, resp.StatusCode, string(data)) +} + +// MustFinishSplitKeyspaceGroup finishes a keyspace group split with HTTP API. +func MustFinishSplitKeyspaceGroup(re *require.Assertions, server *tests.TestServer, id uint32) { + httpReq, err := http.NewRequest(http.MethodDelete, server.GetAddr()+keyspaceGroupsPrefix+fmt.Sprintf("/%d/split", id), nil) + re.NoError(err) + // Send request. + resp, err := dialClient.Do(httpReq) + re.NoError(err) + defer resp.Body.Close() + data, err := io.ReadAll(resp.Body) + re.NoError(err) + re.Equal(http.StatusOK, resp.StatusCode, string(data)) +} diff --git a/tests/server/apiv2/handlers/tso_keyspace_group_test.go b/tests/server/apiv2/handlers/tso_keyspace_group_test.go new file mode 100644 index 00000000000..630b745adb1 --- /dev/null +++ b/tests/server/apiv2/handlers/tso_keyspace_group_test.go @@ -0,0 +1,174 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package handlers + +import ( + "context" + "net/http" + "testing" + + "github.com/stretchr/testify/suite" + "github.com/tikv/pd/pkg/mcs/utils" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/server/apiv2/handlers" + "github.com/tikv/pd/tests" +) + +type keyspaceGroupTestSuite struct { + suite.Suite + ctx context.Context + cancel context.CancelFunc + cluster *tests.TestCluster + server *tests.TestServer +} + +func TestKeyspaceGroupTestSuite(t *testing.T) { + suite.Run(t, new(keyspaceGroupTestSuite)) +} + +func (suite *keyspaceGroupTestSuite) SetupTest() { + suite.ctx, suite.cancel = context.WithCancel(context.Background()) + cluster, err := tests.NewTestAPICluster(suite.ctx, 1) + suite.cluster = cluster + suite.NoError(err) + suite.NoError(cluster.RunInitialServers()) + suite.NotEmpty(cluster.WaitLeader()) + suite.server = cluster.GetServer(cluster.GetLeader()) + suite.NoError(suite.server.BootstrapCluster()) +} + +func (suite *keyspaceGroupTestSuite) TearDownTest() { + suite.cancel() + suite.cluster.Destroy() +} + +func (suite *keyspaceGroupTestSuite) TestCreateKeyspaceGroups() { + re := suite.Require() + kgs := &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Standard.String(), + }, + { + ID: uint32(2), + UserKind: endpoint.Standard.String(), + }, + }} + MustCreateKeyspaceGroup(re, suite.server, kgs) + + // miss user kind, use default value. + kgs = &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(3), + }, + }} + MustCreateKeyspaceGroup(re, suite.server, kgs) + + // invalid user kind. + kgs = &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(4), + UserKind: "invalid", + }, + }} + FailCreateKeyspaceGroupWithCode(re, suite.server, kgs, http.StatusBadRequest) + + // miss ID. + kgs = &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + UserKind: endpoint.Standard.String(), + }, + }} + FailCreateKeyspaceGroupWithCode(re, suite.server, kgs, http.StatusInternalServerError) + + // invalid ID. + kgs = &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: utils.MaxKeyspaceGroupCount + 1, + UserKind: endpoint.Standard.String(), + }, + }} + FailCreateKeyspaceGroupWithCode(re, suite.server, kgs, http.StatusBadRequest) + + // repeated ID. + kgs = &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(2), + UserKind: endpoint.Standard.String(), + }, + }} + FailCreateKeyspaceGroupWithCode(re, suite.server, kgs, http.StatusInternalServerError) +} + +func (suite *keyspaceGroupTestSuite) TestLoadKeyspaceGroup() { + re := suite.Require() + kgs := &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Standard.String(), + }, + { + ID: uint32(2), + UserKind: endpoint.Standard.String(), + }, + }} + + MustCreateKeyspaceGroup(re, suite.server, kgs) + resp := MustLoadKeyspaceGroups(re, suite.server, "0", "0") + re.Len(resp, 3) +} + +func (suite *keyspaceGroupTestSuite) TestSplitKeyspaceGroup() { + re := suite.Require() + kgs := &handlers.CreateKeyspaceGroupParams{KeyspaceGroups: []*endpoint.KeyspaceGroup{ + { + ID: uint32(1), + UserKind: endpoint.Standard.String(), + Keyspaces: []uint32{111, 222, 333}, + Members: make([]endpoint.KeyspaceGroupMember, utils.KeyspaceGroupDefaultReplicaCount), + }, + }} + + MustCreateKeyspaceGroup(re, suite.server, kgs) + resp := MustLoadKeyspaceGroups(re, suite.server, "0", "0") + re.Len(resp, 2) + MustSplitKeyspaceGroup(re, suite.server, 1, &handlers.SplitKeyspaceGroupByIDParams{ + NewID: uint32(2), + Keyspaces: []uint32{111, 222}, + }) + resp = MustLoadKeyspaceGroups(re, suite.server, "0", "0") + re.Len(resp, 3) + // Check keyspace group 1. + kg1 := MustLoadKeyspaceGroupByID(re, suite.server, 1) + re.Equal(uint32(1), kg1.ID) + re.Equal([]uint32{333}, kg1.Keyspaces) + re.True(kg1.IsSplitSource()) + re.Equal(kg1.ID, kg1.SplitSource()) + // Check keyspace group 2. + kg2 := MustLoadKeyspaceGroupByID(re, suite.server, 2) + re.Equal(uint32(2), kg2.ID) + re.Equal([]uint32{111, 222}, kg2.Keyspaces) + re.True(kg2.IsSplitTarget()) + re.Equal(kg1.ID, kg2.SplitSource()) + // They should have the same user kind and members. + re.Equal(kg1.UserKind, kg2.UserKind) + re.Equal(kg1.Members, kg2.Members) + // Finish the split and check the split state. + MustFinishSplitKeyspaceGroup(re, suite.server, 2) + kg1 = MustLoadKeyspaceGroupByID(re, suite.server, 1) + re.False(kg1.IsSplitting()) + kg2 = MustLoadKeyspaceGroupByID(re, suite.server, 2) + re.False(kg2.IsSplitting()) +} diff --git a/tests/server/cluster/cluster_test.go b/tests/server/cluster/cluster_test.go index 527f54c6fcc..c8d2ca823f0 100644 --- a/tests/server/cluster/cluster_test.go +++ b/tests/server/cluster/cluster_test.go @@ -813,7 +813,7 @@ func TestLoadClusterInfo(t *testing.T) { rc := cluster.NewRaftCluster(ctx, svr.ClusterID(), syncer.NewRegionSyncer(svr), svr.GetClient(), svr.GetHTTPClient()) // Cluster is not bootstrapped. - rc.InitCluster(svr.GetAllocator(), svr.GetPersistOptions(), svr.GetStorage(), svr.GetBasicCluster()) + rc.InitCluster(svr.GetAllocator(), svr.GetPersistOptions(), svr.GetStorage(), svr.GetBasicCluster(), svr.GetKeyspaceGroupManager()) raftCluster, err := rc.LoadClusterInfo() re.NoError(err) re.Nil(raftCluster) @@ -852,7 +852,7 @@ func TestLoadClusterInfo(t *testing.T) { re.NoError(testStorage.Flush()) raftCluster = cluster.NewRaftCluster(ctx, svr.ClusterID(), syncer.NewRegionSyncer(svr), svr.GetClient(), svr.GetHTTPClient()) - raftCluster.InitCluster(mockid.NewIDAllocator(), opt, testStorage, basicCluster) + raftCluster.InitCluster(mockid.NewIDAllocator(), opt, testStorage, basicCluster, svr.GetKeyspaceGroupManager()) raftCluster, err = raftCluster.LoadClusterInfo() re.NoError(err) re.NotNil(raftCluster) @@ -1440,7 +1440,7 @@ func TestTransferLeaderBack(t *testing.T) { leaderServer := tc.GetServer(tc.GetLeader()) svr := leaderServer.GetServer() rc := cluster.NewRaftCluster(ctx, svr.ClusterID(), syncer.NewRegionSyncer(svr), svr.GetClient(), svr.GetHTTPClient()) - rc.InitCluster(svr.GetAllocator(), svr.GetPersistOptions(), svr.GetStorage(), svr.GetBasicCluster()) + rc.InitCluster(svr.GetAllocator(), svr.GetPersistOptions(), svr.GetStorage(), svr.GetBasicCluster(), svr.GetKeyspaceGroupManager()) storage := rc.GetStorage() meta := &metapb.Cluster{Id: 123} re.NoError(storage.SaveMeta(meta)) diff --git a/tests/server/keyspace/keyspace_test.go b/tests/server/keyspace/keyspace_test.go index 5ade284c2be..47be22f3ff1 100644 --- a/tests/server/keyspace/keyspace_test.go +++ b/tests/server/keyspace/keyspace_test.go @@ -27,9 +27,9 @@ import ( "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" "github.com/tikv/pd/pkg/codec" + "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/server/config" - "github.com/tikv/pd/server/keyspace" "github.com/tikv/pd/server/schedule/labeler" "github.com/tikv/pd/tests" ) @@ -81,8 +81,9 @@ func (suite *keyspaceTestSuite) TestRegionLabeler() { var err error for i := 0; i < count; i++ { keyspaces[i], err = manager.CreateKeyspace(&keyspace.CreateKeyspaceRequest{ - Name: fmt.Sprintf("test_keyspace%d", i), - Now: now, + Name: fmt.Sprintf("test_keyspace_%d", i), + CreateTime: now, + IsPreAlloc: true, // skip wait region split }) re.NoError(err) } diff --git a/tests/server/tso/common_test.go b/tests/server/tso/common_test.go index f528103db84..877fcb10982 100644 --- a/tests/server/tso/common_test.go +++ b/tests/server/tso/common_test.go @@ -28,7 +28,7 @@ import ( ) const ( - tsoRequestConcurrencyNumber = 5 + tsoRequestConcurrencyNumber = 3 tsoRequestRound = 30 tsoCount = 10 ) diff --git a/tests/server/tso/tso_test.go b/tests/server/tso/tso_test.go index 7b318ad7d36..48df02a6c27 100644 --- a/tests/server/tso/tso_test.go +++ b/tests/server/tso/tso_test.go @@ -70,7 +70,7 @@ func TestLoadTimestamp(t *testing.T) { re.Greater(newTS.GetPhysical()-lastTS.GetPhysical(), int64(0)) } - failpoint.Disable("github.com/tikv/pd/pkg/tso/systemTimeSlow") + re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/tso/systemTimeSlow")) } func requestLocalTSOs(re *require.Assertions, cluster *tests.TestCluster, dcLocationConfig map[string]string) map[string]*pdpb.Timestamp { diff --git a/tests/tso/client_test.go b/tests/tso/client_test.go deleted file mode 100644 index 8e38e0fce1b..00000000000 --- a/tests/tso/client_test.go +++ /dev/null @@ -1,170 +0,0 @@ -// Copyright 2023 TiKV Project Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package tso - -import ( - "context" - "math" - "strings" - "sync" - "testing" - - "github.com/pingcap/failpoint" - "github.com/stretchr/testify/suite" - pd "github.com/tikv/pd/client" - "github.com/tikv/pd/client/testutil" - tso "github.com/tikv/pd/pkg/mcs/tso/server" - "github.com/tikv/pd/pkg/utils/tempurl" - "github.com/tikv/pd/pkg/utils/tsoutil" - "github.com/tikv/pd/tests" - "github.com/tikv/pd/tests/mcs" -) - -type tsoClientTestSuite struct { - suite.Suite - legacy bool - - ctx context.Context - cancel context.CancelFunc - // The PD cluster. - cluster *tests.TestCluster - // The TSO service in microservice mode. - tsoServer *tso.Server - tsoServerCleanup func() - - client pd.TSOClient -} - -func TestLegacyTSOClient(t *testing.T) { - suite.Run(t, &tsoClientTestSuite{ - legacy: true, - }) -} - -func TestMicroserviceTSOClient(t *testing.T) { - suite.Run(t, &tsoClientTestSuite{ - legacy: false, - }) -} - -func (suite *tsoClientTestSuite) SetupSuite() { - re := suite.Require() - - var err error - suite.ctx, suite.cancel = context.WithCancel(context.Background()) - if suite.legacy { - suite.cluster, err = tests.NewTestCluster(suite.ctx, serverCount) - } else { - suite.cluster, err = tests.NewTestAPICluster(suite.ctx, serverCount) - } - re.NoError(err) - err = suite.cluster.RunInitialServers() - re.NoError(err) - leaderName := suite.cluster.WaitLeader() - pdLeader := suite.cluster.GetServer(leaderName) - backendEndpoints := pdLeader.GetAddr() - if suite.legacy { - suite.client, err = pd.NewClientWithContext(suite.ctx, strings.Split(backendEndpoints, ","), pd.SecurityOption{}) - re.NoError(err) - } else { - suite.tsoServer, suite.tsoServerCleanup = mcs.StartSingleTSOTestServer(suite.ctx, re, backendEndpoints, tempurl.Alloc()) - suite.client = mcs.SetupTSOClient(suite.ctx, re, strings.Split(backendEndpoints, ",")) - } -} - -func (suite *tsoClientTestSuite) TearDownSuite() { - suite.cancel() - if !suite.legacy { - suite.tsoServerCleanup() - } - suite.cluster.Destroy() -} - -func (suite *tsoClientTestSuite) TestGetTS() { - var wg sync.WaitGroup - wg.Add(tsoRequestConcurrencyNumber) - for i := 0; i < tsoRequestConcurrencyNumber; i++ { - go func() { - defer wg.Done() - var lastTS uint64 - for i := 0; i < tsoRequestRound; i++ { - physical, logical, err := suite.client.GetTS(suite.ctx) - suite.NoError(err) - ts := tsoutil.ComposeTS(physical, logical) - suite.Less(lastTS, ts) - lastTS = ts - } - }() - } - wg.Wait() -} - -func (suite *tsoClientTestSuite) TestGetTSAsync() { - var wg sync.WaitGroup - wg.Add(tsoRequestConcurrencyNumber) - for i := 0; i < tsoRequestConcurrencyNumber; i++ { - go func() { - defer wg.Done() - tsFutures := make([]pd.TSFuture, tsoRequestRound) - for i := range tsFutures { - tsFutures[i] = suite.client.GetTSAsync(suite.ctx) - } - var lastTS uint64 = math.MaxUint64 - for i := len(tsFutures) - 1; i >= 0; i-- { - physical, logical, err := tsFutures[i].Wait() - suite.NoError(err) - ts := tsoutil.ComposeTS(physical, logical) - suite.Greater(lastTS, ts) - lastTS = ts - } - }() - } - wg.Wait() -} - -// More details can be found in this issue: https://github.com/tikv/pd/issues/4884 -func (suite *tsoClientTestSuite) TestUpdateAfterResetTSO() { - re := suite.Require() - ctx, cancel := context.WithCancel(suite.ctx) - defer cancel() - - testutil.Eventually(re, func() bool { - _, _, err := suite.client.GetTS(ctx) - return err == nil - }) - // Transfer leader to trigger the TSO resetting. - re.NoError(failpoint.Enable("github.com/tikv/pd/server/updateAfterResetTSO", "return(true)")) - oldLeaderName := suite.cluster.WaitLeader() - err := suite.cluster.GetServer(oldLeaderName).ResignLeader() - re.NoError(err) - re.NoError(failpoint.Disable("github.com/tikv/pd/server/updateAfterResetTSO")) - newLeaderName := suite.cluster.WaitLeader() - re.NotEqual(oldLeaderName, newLeaderName) - // Request a new TSO. - testutil.Eventually(re, func() bool { - _, _, err := suite.client.GetTS(ctx) - return err == nil - }) - // Transfer leader back. - re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/tso/delaySyncTimestamp", `return(true)`)) - err = suite.cluster.GetServer(newLeaderName).ResignLeader() - re.NoError(err) - // Should NOT panic here. - testutil.Eventually(re, func() bool { - _, _, err := suite.client.GetTS(ctx) - return err == nil - }) - re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/tso/delaySyncTimestamp")) -} diff --git a/tools/pd-ctl/pdctl/command/cluster_command.go b/tools/pd-ctl/pdctl/command/cluster_command.go index 60d2b786e3f..43722d4e58d 100644 --- a/tools/pd-ctl/pdctl/command/cluster_command.go +++ b/tools/pd-ctl/pdctl/command/cluster_command.go @@ -20,8 +20,10 @@ import ( "github.com/spf13/cobra" ) -const clusterPrefix = "pd/api/v1/cluster" -const clusterStatusPrefix = "pd/api/v1/cluster/status" +const ( + clusterPrefix = "pd/api/v1/cluster" + clusterStatusPrefix = "pd/api/v1/cluster/status" +) // NewClusterCommand return a cluster subcommand of rootCmd func NewClusterCommand() *cobra.Command { diff --git a/tools/pd-ctl/pdctl/command/keyspace_group_command.go b/tools/pd-ctl/pdctl/command/keyspace_group_command.go new file mode 100644 index 00000000000..6fbb8170461 --- /dev/null +++ b/tools/pd-ctl/pdctl/command/keyspace_group_command.go @@ -0,0 +1,83 @@ +// Copyright 2023 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package command + +import ( + "fmt" + "net/http" + "strconv" + + "github.com/spf13/cobra" +) + +const keyspaceGroupsPrefix = "pd/api/v2/tso/keyspace-groups" + +// NewKeyspaceGroupCommand return a keyspace group subcommand of rootCmd +func NewKeyspaceGroupCommand() *cobra.Command { + cmd := &cobra.Command{ + Use: "keyspace-group ", + Short: "show keyspace group information with the given ID", + Run: showKeyspaceGroupCommandFunc, + } + cmd.AddCommand(newSplitKeyspaceGroupCommand()) + return cmd +} + +func newSplitKeyspaceGroupCommand() *cobra.Command { + r := &cobra.Command{ + Use: "split []", + Short: "split the keyspace group with the given ID and transfer the keyspaces into the newly split one", + Run: splitKeyspaceGroupCommandFunc, + } + return r +} + +func showKeyspaceGroupCommandFunc(cmd *cobra.Command, args []string) { + if len(args) < 1 { + cmd.Usage() + return + } + r, err := doRequest(cmd, fmt.Sprintf("%s/%s", keyspaceGroupsPrefix, args[0]), http.MethodGet, http.Header{}) + if err != nil { + cmd.Printf("Failed to get the keyspace groups information: %s\n", err) + return + } + cmd.Println(r) +} + +func splitKeyspaceGroupCommandFunc(cmd *cobra.Command, args []string) { + if len(args) < 3 { + cmd.Usage() + return + } + newID, err := strconv.ParseUint(args[1], 10, 32) + if err != nil { + cmd.Printf("Failed to parse the new keyspace group ID: %s\n", err) + return + } + keyspaces := make([]uint32, 0, len(args)-2) + for _, arg := range args[2:] { + id, err := strconv.ParseUint(arg, 10, 32) + if err != nil { + cmd.Printf("Failed to parse the keyspace ID: %s\n", err) + return + } + keyspaces = append(keyspaces, uint32(id)) + } + postJSON(cmd, fmt.Sprintf("%s/%s/split", keyspaceGroupsPrefix, args[0]), map[string]interface{}{ + "new-id": uint32(newID), + "keyspaces": keyspaces, + }) +} diff --git a/tools/pd-ctl/pdctl/command/tso_command.go b/tools/pd-ctl/pdctl/command/tso_command.go index 2eedf9a2a3e..689420854ee 100644 --- a/tools/pd-ctl/pdctl/command/tso_command.go +++ b/tools/pd-ctl/pdctl/command/tso_command.go @@ -21,7 +21,7 @@ import ( "github.com/tikv/pd/pkg/utils/tsoutil" ) -// NewTSOCommand return a ping subcommand of rootCmd +// NewTSOCommand return a TSO subcommand of rootCmd func NewTSOCommand() *cobra.Command { cmd := &cobra.Command{ Use: "tso ", diff --git a/tools/pd-ctl/pdctl/ctl.go b/tools/pd-ctl/pdctl/ctl.go index 252c7471eaa..8fc3a454d7a 100644 --- a/tools/pd-ctl/pdctl/ctl.go +++ b/tools/pd-ctl/pdctl/ctl.go @@ -65,6 +65,7 @@ func GetRootCmd() *cobra.Command { command.NewMinResolvedTSCommand(), command.NewCompletionCommand(), command.NewUnsafeCommand(), + command.NewKeyspaceGroupCommand(), ) rootCmd.Flags().ParseErrorsWhitelist.UnknownFlags = true diff --git a/tools/pd-tso-bench/go.sum b/tools/pd-tso-bench/go.sum index ea7e8eb9d70..6d2e5e6bbb4 100644 --- a/tools/pd-tso-bench/go.sum +++ b/tools/pd-tso-bench/go.sum @@ -562,8 +562,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c h1:xpW9bvK+HuuTm github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ueLEUZgtx2cIogM0v4Zj5uvvzhuuiu7Pn8HzMPg= github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00 h1:C3N3itkduZXDZFh4N3vQ5HEtld3S+Y+StULhWVvumU0= github.com/pingcap/failpoint v0.0.0-20210918120811-547c13e3eb00/go.mod h1:4qGtCB0QK0wBzKtFEGDhxXnSnbQApw1gc9siScUl8ew= -github.com/pingcap/kvproto v0.0.0-20230317010544-b47a4830141f h1:P4MWntrAwXARSLRVgnJ8W2zqIhHWvOSSLK4DjNyiN4A= -github.com/pingcap/kvproto v0.0.0-20230317010544-b47a4830141f/go.mod h1:KUrW1FGoznGMMTssYBu0czfAhn6vQcIrHyZoSC6T990= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1 h1:VXQ6Du/nKZ9IQnI9NWMzKbftWu8NV5pQkSLKIRzzGN4= +github.com/pingcap/kvproto v0.0.0-20230511011722-6e0e8a7deaa1/go.mod h1:guCyM5N+o+ru0TsoZ1hi9lDjUMs2sIBjW3ARTEpVbnk= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= @@ -575,8 +575,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= -github.com/prometheus/client_golang v1.11.1 h1:+4eQaD7vAZ6DsfsxB15hbE0odUjGI5ARs9yskGu1v4s= -github.com/prometheus/client_golang v1.11.1/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= +github.com/prometheus/client_golang v1.11.0 h1:HNkLOAEQMIDv/K+04rukrLx6ch7msSRwf3/SASFAGtQ= +github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= @@ -734,8 +734,8 @@ golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug golang.org/x/net v0.0.0-20220909164309-bea034e7d591/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= golang.org/x/net v0.0.0-20221012135044-0b7e1fb9d458/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= golang.org/x/net v0.0.0-20221014081412-f15817d10f9b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= -golang.org/x/net v0.2.0 h1:sZfSu1wtKLGlWI4ZZayP0ck9Y73K1ynO6gqzTdBVdPU= -golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= +golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= +golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -847,11 +847,11 @@ golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220624220833-87e55d714810/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.2.0 h1:ljd4t30dBnAvMZaQCevtY0xLLD0A+bRZXbgLMLU1F/A= -golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -862,8 +862,9 @@ golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= -golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=