Merge TiKV 6.0 into Proxy #55

CalvinNeo · 2022-04-20T06:15:30Z

What problem does this PR solve?

Issue Number: close pingcap/tiflash#4618

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

…1213) close tikv#11159 Signed-off-by: Wenbo Zhang <[email protected]>

* reduce extra next/pre in row_scan close tikv#11572 Signed-off-by: Jayice <[email protected]> * reduce extra next/pre in row_scan close tikv#11572 Signed-off-by: Jayice <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#11593 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Yilin Chen <[email protected]>

* add snap-generator-pool-size for configurable snapshot generator thread pool Signed-off-by: iosmanthus <[email protected]> * rename config macro Signed-off-by: iosmanthus <[email protected]> Co-authored-by: Andy Lok <[email protected]>

* add prompt for ctl close tikv#11514 Signed-off-by: Jayice <[email protected]> * format code close tikv#11514 Signed-off-by: Jayice <[email protected]> * format code close tikv#11514 Signed-off-by: Jayice <[email protected]> * format code close tikv#11514 Signed-off-by: Jayice <[email protected]> * optimize duplicate codes:pd check close tikv#11514 Signed-off-by: Jayice <[email protected]> * move pd check in get_pd_rpc_client close tikv#11514 Signed-off-by: Jayice <[email protected]> * simplify the codes check host|data-dir close tikv#11514 Signed-off-by: Jayice <[email protected]>

…1269) * ctl: should check column name validation when scan data using ctl close tikv#11268 Signed-off-by: yuqi1129 <[email protected]> * fix format problem Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]>

…onsistency (tikv#10977) * Add mutation checking logic Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * Fix usages of mutation Signed-off-by: MyonKeminta <[email protected]> * Update kvproto and renaming Signed-off-by: MyonKeminta <[email protected]> * Add a simple test Signed-off-by: MyonKeminta <[email protected]> * Fix incorrect assertion Signed-off-by: MyonKeminta <[email protected]> * Support check existance during pessimistic lock Signed-off-by: MyonKeminta <[email protected]> * Add assertion level support Signed-off-by: MyonKeminta <[email protected]> * Add tests Signed-off-by: MyonKeminta <[email protected]> * Add more test cases Signed-off-by: MyonKeminta <[email protected]> * Add test for check_existence of pessimistic_lock request Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * ref tikv#11542 Signed-off-by: MyonKeminta <[email protected]> * Add a metric for counting different way of checking assertion, for reference about the performance impact Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: Shuaipeng Yu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> * close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…kv#11216) * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * fix enable_ttl Signed-off-by: andylokandy <[email protected]> * simplify test Signed-off-by: andylokandy <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> Co-authored-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: qupeng <[email protected]>

* close tikv#11642 Signed-off-by: qupeng <[email protected]> * a more comment Signed-off-by: qupeng <[email protected]>

* ctl: split into multiple modules ref tikv#10938 Signed-off-by: andylokandy <[email protected]> * refactor if let to match Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* close tikv#11676 Signed-off-by: devillove084 <[email protected]>

* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref tikv#11678 ```release-note NONE ``` Signed-off-by: Harold Dost <[email protected]>

Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#11690 Signed-off-by: mornyx <[email protected]>

* file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * fix clippy Signed-off-by: MuZhou233 <[email protected]> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <[email protected]> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * Integrate thread_io Signed-off-by: MuZhou233 <[email protected]> * record bytes in request Signed-off-by: MuZhou233 <[email protected]> * fix logical bug Signed-off-by: MuZhou233 <[email protected]> * Update test write_bytes Signed-off-by: MuZhou233 <[email protected]> * Delete unused & Rename func Signed-off-by: MuZhou233 <[email protected]> * Add test for read & Update test for write Signed-off-by: MuZhou233 <[email protected]> * Add buffered thread io Signed-off-by: MuZhou233 <[email protected]> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <[email protected]> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <[email protected]> * Change fetch logic. close tikv#10867 Signed-off-by: MuZhou233 <[email protected]> * Add AtomicIOBytes Signed-off-by: MuZhou233 <[email protected]> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <[email protected]> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <[email protected]> * Adjust code style & fix Signed-off-by: MuZhou233 <[email protected]> * improve write_bytes test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…ocks (tikv#11591) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]>

* add datasink trait Signed-off-by: Zhenchi <[email protected]> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build ref tikv#11691 Signed-off-by: Zhenchi <[email protected]> * datasink -> data_sink Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref tikv#4301 Signed-off-by: Harold Dost <[email protected]> * Test Revert of Iterations ref tikv#4301 Signed-off-by: Harold Dost <[email protected]> * clippy: Enable check for rust-2018-idioms ref tikv#4301 Signed-off-by: Harold Dost <[email protected]>

ref tikv#11691 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: linning <[email protected]>

* Use generic for api version to reduce runtime branching ref tikv#10938 Signed-off-by: andylokandy <[email protected]> * fix test Signed-off-by: andylokandy <[email protected]> * add test Signed-off-by: andylokandy <[email protected]> * fix clippy Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * fmt Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * rename vars Signed-off-by: andylokandy <[email protected]> * improve vars Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* sst_importer: check api version when importing ref tikv#10938 Signed-off-by: Peng Guanwen <[email protected]> * Fix bugs Signed-off-by: Peng Guanwen <[email protected]> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <[email protected]> * remove redundant log Signed-off-by: Peng Guanwen <[email protected]> * Show the entire key Signed-off-by: Peng Guanwen <[email protected]> * Fix lint issue Signed-off-by: Peng Guanwen <[email protected]> * resolve conflict Signed-off-by: andylokandy <[email protected]> Co-authored-by: Andy Lok <[email protected]>

* br: sperate io and scan threads Signed-off-by: Yu Juncen <[email protected]> * br/stream: make clippy happy Signed-off-by: yujuncen <[email protected]> * br/stream: fix tests Signed-off-by: Yu Juncen <[email protected]> * backup: fix a dummy bug Signed-off-by: Yu Juncen <[email protected]> * backup: add config of io-threads Signed-off-by: yujuncen <[email protected]> * backup: ref tikv#11350: some minior change Signed-off-by: yujuncen <[email protected]> * backup: added some metrics Signed-off-by: yujuncen <[email protected]> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <[email protected]> * backup: added some slow log and metrics Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: fix build Signed-off-by: yujuncen <[email protected]> * Revert "backup: fix build" This reverts commit 74537be. Signed-off-by: yujuncen <[email protected]> * Revert "backup: address comments" This reverts commit 77d7575. Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: always set IO type to Export Signed-off-by: yujuncen <[email protected]> * backup:make clippy happy Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]>

ref tikv#11662 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* Update kvproto Signed-off-by: Peng Guanwen <[email protected]> * Return api-version for br ref tikv#10938 Signed-off-by: Peng Guanwen <[email protected]> * Reformat code Signed-off-by: Peng Guanwen <[email protected]> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * format code Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]>

* limit thread_io target os Signed-off-by: MuZhou233 <[email protected]> * limit thread_io target os. typo. close tikv#11698 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Yilin Chen <[email protected]>

Close tikv#10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (danburkert/fs2-rs#42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <[email protected]>

Signed-off-by: CalvinNeo <[email protected]>

…kv#12315) close tikv#12278, ref tikv#12309 Fix test_delegate_subscribe_unsubscribe flaking test. Use the corresponding id, otherwise there will be failures due to concurrent runs of the test. (We will assign downstream ids concurrently). Signed-off-by: ti-srebot <[email protected]> Co-authored-by: 二手掉包工程师 <[email protected]> Co-authored-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

CLAassistant · 2022-04-20T06:15:58Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 7 committers have signed the CLA.

✅ mornyx
✅ CalvinNeo
✅ purelind
✅ breezewish
✅ tier-cap
❌ ti-srebot
❌ tabokie
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Signed-off-by: CalvinNeo <[email protected]>

…6.0-try

Signed-off-by: CalvinNeo <[email protected]>

Signed-off-by: Wish <[email protected]>

…raftstore-proxy-6.0-try

Signed-off-by: mornyx <[email protected]>

…g on ARM (pingcap#61) Signed-off-by: mornyx <[email protected]>

Signed-off-by: CalvinNeo <[email protected]>

close tikv#11379, close tikv#11598, close tikv#12107 when memory usage is high water, add flow control on the append log entries. disable log entry cache evict functions. Signed-off-by: tier-cap <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Signed-off-by: CalvinNeo <[email protected]>

…ine-ext into raftstore-proxy-6.0-try

Signed-off-by: CalvinNeo <[email protected]>

ethercflow and others added 30 commits December 11, 2021 02:01

raftstore: support dynamically scaling pool size (tikv#11159) (tikv#1…

7eeff52

…1213) close tikv#11159 Signed-off-by: Wenbo Zhang <[email protected]>

*: remove some unnecessary dependencies (tikv#11592)

c121e22

close tikv#11593 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Yilin Chen <[email protected]>

metrics: fix memory metrics (tikv#11656)

c5c030d

Signed-off-by: Shuaipeng Yu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

coprocesser: refactor encoding and add gbk_chinese_ci map (tikv#11661)

444edc0

* close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> * close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

cdc: load uninlined value more effectively (tikv#11615)

45fac77

Signed-off-by: qupeng <[email protected]>

cdc: load old value with prefix seek if possible (tikv#11643)

1b0edd0

* close tikv#11642 Signed-off-by: qupeng <[email protected]> * a more comment Signed-off-by: qupeng <[email protected]>

Fix build issues on non-linux platforms (tikv#11671)

ed82712

* close tikv#11676 Signed-off-by: devillove084 <[email protected]>

all: Migrate from std:: types to primitives. (tikv#11677)

c2ca032

* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref tikv#11678 ```release-note NONE ``` Signed-off-by: Harold Dost <[email protected]>

metrics alert: fix apply cpu alert (tikv#11672)

186bca2

Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

resrc_mtr: introduce scanned keys statistics (tikv#11085)

8c24715

close tikv#11690 Signed-off-by: mornyx <[email protected]>

*: check epoch and validness of lock table when writing pessimistic l…

a249ec7

…ocks (tikv#11591) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]>

rsmeter: add centralized place to construct protobuf data (tikv#11696)

849a01d

ref tikv#11691 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

tikv_util: update procfs to 0.12.0 (tikv#11703)

92da184

Signed-off-by: linning <[email protected]>

metrics: fix the metrics cannot be displayed (tikv#11710)

205e1b8

ref tikv#11662 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

file_system: limit thread_io target os (tikv#11715)

b62e381

* limit thread_io target os Signed-off-by: MuZhou233 <[email protected]> * limit thread_io target os. typo. close tikv#11698 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Yilin Chen <[email protected]>

CalvinNeo and others added 9 commits April 2, 2022 13:28

Fix test

309bfd5

Signed-off-by: CalvinNeo <[email protected]>

f

f110ef5

Signed-off-by: CalvinNeo <[email protected]>

fix linux

e284710

Signed-off-by: CalvinNeo <[email protected]>

fix

e394d60

Signed-off-by: CalvinNeo <[email protected]>

fix some files

5e86f8c

Signed-off-by: CalvinNeo <[email protected]>

fix

782c299

Signed-off-by: CalvinNeo <[email protected]>

fix

762f08f

Signed-off-by: CalvinNeo <[email protected]>

ignore some unsupported tests

317086a

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo added 5 commits April 20, 2022 15:08

fix fmt

8035118

Signed-off-by: CalvinNeo <[email protected]>

disable more disk_full tests

a5ff078

Signed-off-by: CalvinNeo <[email protected]>

f

1e31964

Signed-off-by: CalvinNeo <[email protected]>

disable

03b8215

Signed-off-by: CalvinNeo <[email protected]>

remove stale peer

77923ca

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo requested a review from solotzg April 22, 2022 05:41

CalvinNeo mentioned this pull request Apr 25, 2022

update TiFlash proxy to raftstore-proxy-6.0 for up(down)grade test pingcap/tiflash#4748

Merged

12 tasks

CalvinNeo and others added 4 commits May 5, 2022 11:36

Merge remote-tracking branch 'tikv/release-6.0' into raftstore-proxy-…

a03434d

…6.0-try

pick pr 12301

88168c6

Signed-off-by: CalvinNeo <[email protected]>

Fix build when openssl v1.1 and v3 are both installed (pingcap#59)

0ef80c7

Signed-off-by: Wish <[email protected]>

Merge remote-tracking branch 'upstream/raftstore-proxy-6.0-try' into …

6cb40b3

…raftstore-proxy-6.0-try

solotzg approved these changes May 11, 2022

View reviewed changes

mornyx and others added 7 commits May 12, 2022 12:33

Enable frame pointer (pingcap#60)

2cf6ff7

Signed-off-by: mornyx <[email protected]>

Upgrade pprof-rs to v0.9 (use frame-pointer), and enable CPU Profilin…

e46e887

…g on ARM (pingcap#61) Signed-off-by: mornyx <[email protected]>

reduce log

f307053

Signed-off-by: CalvinNeo <[email protected]>

Merge branch 'raftstore-proxy-6.0-try' of github.com:pingcap/tidb-eng…

e98f959

…ine-ext into raftstore-proxy-6.0-try

fix pd.rs of reporting to PD

fe64c1d

Signed-off-by: CalvinNeo <[email protected]>

fix

5bd9c73

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo merged commit 713b918 into pingcap:raftstore-proxy May 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge TiKV 6.0 into Proxy #55

Merge TiKV 6.0 into Proxy #55

CalvinNeo commented Apr 20, 2022 •

edited

Loading

CLAassistant commented Apr 20, 2022 •

edited

Loading

Merge TiKV 6.0 into Proxy #55

Merge TiKV 6.0 into Proxy #55

Conversation

CalvinNeo commented Apr 20, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

CLAassistant commented Apr 20, 2022 • edited Loading

CalvinNeo commented Apr 20, 2022 •

edited

Loading

CLAassistant commented Apr 20, 2022 •

edited

Loading