Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor TiFlash Proxy #5170

Closed
CalvinNeo opened this issue Jun 20, 2022 · 0 comments
Closed

Refactor TiFlash Proxy #5170

CalvinNeo opened this issue Jun 20, 2022 · 0 comments
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@CalvinNeo
Copy link
Member

CalvinNeo commented Jun 20, 2022 •
edited
Loading

Enhancement

We already have a demo. However, we need to let TiKV aware some engine traits other than RocksEngine, which TiFlash can benefit from.

TiKV side

Support abstraction of new KvEngine on TiKV side.

  1. Support TiKV using different KvEngines tikv/tikv#12849

Proxy side

Implement proxy based on new TiKV(not including cherry-picked TiKV's bugfixes).

  1. Introduce proxy_server and new-mock-engine-store tidb-engine-ext#120
    Introduce engine_tiflash(with compat_new_proxy to disable some features), proxy_server and new-mock-engine-store
    We perform a prelim test in a special branch Introduce proxy_server and new-mock-engine-store #5319
  2. Support on_empty_cmd observer in Proxy tidb-engine-ext#121
  3. Handle destroy peer by observer tidb-engine-ext#132
  4. Support pd task observer to compute store stats for TiFlash tidb-engine-ext#136
  5. Re-enable debug service tidb-engine-ext#131
  6. Avoid set addr/engine-addr/version/git-hash and HackedLockManager in src/node.rs tidb-engine-ext#139
  7. Migrate pre exec tidb-engine-ext#123
  8. Implement Proxy's own status_server tidb-engine-ext#141
  9. Support post_exec on Raftstore Proxy tidb-engine-ext#140
  10. Decouple apply snapshot by introducing pre(post)_apply_snapshot tidb-engine-ext#146
  11. Remove engine store server helper in raftstore tidb-engine-ext#148
  12. Reclaim TiKV's component server and status server tidb-engine-ext#152
  13. Reclaim TiKV's tikv_util crate tidb-engine-ext#154
  14. Add readme for proxy tidb-engine-ext#145
  15. Default enable raft-engine tidb-engine-ext#150
  16. Reuse engine_rocks in engine_tiflash tidb-engine-ext#143
  17. Decouple engine_store_ffi and raftstore tidb-engine-ext#157
  18. Don't write to rockdb when meet empty write batch tidb-engine-ext#159
  19. Use as less engine_rocks in engine_tiflash as we can tidb-engine-ext#156
    In this PR, we explains how to keep trace of engine_rocks from engine_tiflash.
  20. Workaround ci tidb-engine-ext#162
    Fix small bugs of ci tidb-engine-ext#163
  21. Reject when we recv multi file snapshot tidb-engine-ext#165
  22. Support can_apply_snapshot tidb-engine-ext#166
  23. cherry pick tikv#13234 for support customized addr/status_addr tidb-engine-ext#170
  24. Use pre_persist to control persistence tidb-engine-ext#171

After all, we can depend TiKV as a component

  1. Depend TiKV as a component tidb-engine-ext#177

TiFlash side

WE LEAVE OUT TIFLASH's COMMITS SINCE IT REPEATS PROXY's COMMITS

Make TiFlash own the new TiFlash Proxy

  1. Support fn_can_flush_data KVStore: decouple flush region and CompactLog with a new FFI fn_try_flush_data #5283 Introduce new FFIs tidb-engine-ext#119
  2. Make CompactLog do no persistence
  3. Enrich tests test(KVStore): Support Proxy's apply pattern on normal write and compact log #5623
@CalvinNeo CalvinNeo added the type/enhancement The issue or PR belongs to an enhancement. label Jun 20, 2022
@CalvinNeo CalvinNeo mentioned this issue Jul 4, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jul 7, 2022
@CalvinNeo
KVStore: decouple flush region and CompactLog with a new FFI fn_try_f…
0aec04a 
…lush_data (#5283)

ref #5170
This was referenced Jul 11, 2022
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Jul 12, 2022
@CalvinNeo
Introduce proxy_server and new-mock-engine-store (#5319)
0df17c6 
ref #5170
Lloyd-Pottiger pushed a commit to Lloyd-Pottiger/tiflash that referenced this issue Jul 12, 2022
@hongyunyan @Lloyd-Pottiger
Enhancement: Add how to run integration tests and microbenchmark test…
4e4f3b7 
…s in README (pingcap#5182)

close pingcap#5172, ref pingcap#5178

Enhancement: add a integrated test on DDL module (pingcap#5130)

ref pingcap#5129

Revert "Revise default background threads size" (pingcap#5176)

close pingcap#5177

chore: remove extra dyn cast (pingcap#5186)

close pingcap#5185

Add MPPReceiverSet, which includes ExchangeReceiver and CoprocessorReader (pingcap#5175)

ref pingcap#5095

DDL: Use Column Name Instead of Offset to Find the common handle cluster index (pingcap#5166)

close pingcap#5154

Add random failpoint in critical paths (pingcap#4876)

close pingcap#4807

Segment test framework (pingcap#5150)

close pingcap#5151

optimize ps v3 restore (pingcap#5163)

ref pingcap#4914

Fix build failed (pingcap#5196)

close pingcap#5195

feat: delta tree dispatching (pingcap#5199)

close pingcap#5200

feat: introduce specialized API to write fixed length data rapidly (pingcap#5181)

close pingcap#5183

Add gtest for Limit, TopN, Projection (pingcap#5187) (pingcap#5188)

close pingcap#5187

add `MPPTask::handleError()` (pingcap#5202)

ref pingcap#5095

Check result of starting grpc server (pingcap#5257)

close pingcap#5255

feat: add optimized routines for aarch64 (pingcap#5231)

close pingcap#5240

fix: aarch64-quick-fix (pingcap#5259)

close pingcap#5260

Update client-c to support ipv6 (pingcap#5270)

close pingcap#5247

upgrade prometheus-cpp to v1.0.1 (pingcap#5279)

ref pingcap#2103, close pingcap#5278

Fix README type error (pingcap#5273)

ref pingcap#5178

fix(cmake): make sure libc++ is utilized by tiflash-proxy (pingcap#5281)

close pingcap#5282

fix the wrong order of execution summary for list based executors (pingcap#5242)

close pingcap#5241

Schema: allow loading empty schema diff when the version grows up. (pingcap#5245)

close pingcap#5244

Optimize apply speed under heavy write pressure (pingcap#4883)

ref pingcap#4728

update proxy to raftstore-proxy-6.2 (pingcap#5287)

ref pingcap#4982

Flush segment cache when doing the compaction (pingcap#5284)

close pingcap#5179

metrics: Fix incorrect metrics for delta_merge tasks (pingcap#5061)

close pingcap#5055

dep: upgrade jemalloc (pingcap#5197)

close pingcap#5258

*: TiFlash pagectl/dttool use only-decryption mode (pingcap#5271)

close pingcap#5122

suppresion false positive report from tsan (pingcap#5303)

close pingcap#5088

Refine test framework code and tests (pingcap#5261)

close pingcap#5262

feat: add logical cpu cores and memory into grafana (pingcap#5124)

close pingcap#3821

Implement TimeToSec function push down (pingcap#5235)

close pingcap#5116

feat: implement shiftRight function push down (pingcap#5156)

close pingcap#5100

schema : make update to partition tables when 'set tiflash replica' (pingcap#5267)

close pingcap#5266

Replace initializer_list with vector for planner test framework (pingcap#5307)

close pingcap#5295

KVStore: decouple flush region and CompactLog with a new FFI fn_try_flush_data (pingcap#5283)

ref pingcap#5170

refine error message in mpptask (pingcap#5304)

ref pingcap#5095

Implement ReverseUTF8/Reverse function push down (pingcap#5233)

close pingcap#5111

Optimize comparision for collation `UTF8_BIN` and `UTF8MB4_BIN` (pingcap#5299)

ref pingcap#5294

feat : support set tiflash mode ddl action (pingcap#5256)

ref pingcap#5252

Add non-blocking functions for MPMCQueue (pingcap#5311)

close pingcap#5310

add random segment test for CI weekly (pingcap#5300)

close pingcap#5301

*: tidy FunctionString.cpp (pingcap#5312)

close pingcap#5313

ci: fix check-license github action (pingcap#5318)

close pingcap#5317

update proxy to raftstore-proxy-6.2 (pingcap#5316)

ref pingcap#4982

Change one `additional_input_at_end` to many streams in `ParallelInputsProcessor`  (pingcap#5274)

close pingcap#4856, close pingcap#5263

support fine grained shuffle for window function (pingcap#5048)

close pingcap#5142

feat: pushdown get_format into TiFlash (pingcap#5269)

close pingcap#5115

fix: format throw data truncated error (pingcap#5272)

close pingcap#4891

Print content of columns for gtest (pingcap#5243)

close pingcap#5203

*: also enable O3 for aarch64 (pingcap#5338)

close pingcap#5342

Add debug image build target for CentOS7 (pingcap#5344)

close pingcap#5343

*: mini refactor (pingcap#5326)

close pingcap#4739

Refactor initialize of background pool (pingcap#5190)

close pingcap#5189

delete copy/move ctor of MPMCQueue explicitly (pingcap#5328)

close pingcap#5329

Introduce proxy_server and new-mock-engine-store (pingcap#5319)

ref pingcap#5170

fix: incorrect uptime in grafana panel

Signed-off-by: Lloyd-Pottiger <[email protected]>
This was referenced Jul 14, 2022
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Jul 15, 2022
@CalvinNeo
Support on_empty_cmd in Proxy (#5372)
04ecd6e 
ref #5170
@CalvinNeo CalvinNeo mentioned this issue Jul 15, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jul 19, 2022
@CalvinNeo
Support pre_exec in Proxy (#5386)
e737618 
ref #5170
Lloyd-Pottiger pushed a commit to Lloyd-Pottiger/tiflash that referenced this issue Jul 19, 2022
@CalvinNeo @Lloyd-Pottiger
KVStore: decouple flush region and CompactLog with a new FFI fn_try_f…
c191d0a 
…lush_data (pingcap#5283)

ref pingcap#5170
Lloyd-Pottiger pushed a commit to Lloyd-Pottiger/tiflash that referenced this issue Jul 19, 2022
@CalvinNeo @Lloyd-Pottiger
Introduce proxy_server and new-mock-engine-store (pingcap#5319)
e17af98 
ref pingcap#5170
This was referenced Aug 15, 2022
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Aug 15, 2022
@CalvinNeo
Avoid hacks in Proxy's src/server.rs (#5622)
13cecf8 
ref #5170
@CalvinNeo CalvinNeo mentioned this issue Aug 17, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Aug 18, 2022
@CalvinNeo
proxy: Support post_exec (#5643)
4f5e5cd 
ref #5170
@JaySon-Huang JaySon-Huang mentioned this issue Aug 23, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Aug 23, 2022
@JaySon-Huang
Refactor KVStore/RegionPersister test (#5679)
4ec074b 
ref #5170
This was referenced Aug 24, 2022
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Aug 29, 2022
@CalvinNeo
proxy: Decouple prehandle snapshot from RegionWorker (#5720)
a46ca87 
ref #5170
This was referenced Sep 1, 2022
Merged
Merged
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Sep 2, 2022
@CalvinNeo
proxy: Reclaim server and remove engine_store_server_helper (#5767)
ac7d070 
ref #5170
This was referenced Sep 5, 2022
Merged
Merged
Merged
Merged
Merged
Merged
This was referenced Sep 13, 2022
Merged
Merged
Merged
Merged
Merged
This was referenced Sep 21, 2022
Merged
Closed
ti-chi-bot pushed a commit that referenced this issue Sep 26, 2022
@CalvinNeo
test(KVStore): Support Proxy's apply pattern on normal write and comp…
3bdb49e 
…act log (#5623)

ref #5170
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Depend TiKV as a component CalvinNeo/tidb-engine-ext
1 participant
@CalvinNeo