Improve slot migration speed and resource consumption using raw key values

[caipengbo]

Motivation

Currently, kvrocks performs slot migration by converting the data into RESP and then sending it to the destination(#430).
We made a little optimizations in #904 to speed up migration.

However, there are still some problems with command-based slot migration:

• we will determine the expire time during the data migration, which makes the data inconsistent (mentioned in the comments of Avoid using Get interface to obtain metadata when iterating data during migration #906)
• for data with an expiration time, if the expiration occurs during the migration process, then we cannot migrate the data and the entire migration process will fail.

Solution

We can use rawkv-based data migration, where we directly send rocksdb's key values data to the target.

With rawkv-based migration, neither the source nor the destination need to judge expire time, which completely solves the data inconsistency problem. It is equivalent to translating the raw key values data from one instance to another.

It can save a lot of CPU by eliminating the need to convert Key-Value to the RESP.

In my tests, it can:

• save up to 2x CPU on the target side
• for small values(100byte), the rawkv-based migration is 2.75 times faster than the command-based migration

Plan

I decomposed this issues into those tasks:

• Add the ApplyBatch command #1982
• Implement the migrate iterator for iterating key values #1989
• Implement WAL iterator for migrating by WAL logs #2007
• Implement the new migration type by applying raw batch #2008
• Make the raw key values batch migration parameters configurable #2009

[git-hulk]

@caipengbo Do you mean that we can send the write batch directly instead of the RESP format？

[caipengbo]

Do you mean that we can send the write batch directly instead of the RESP format？

@git-hulk Yup, we send the write batch directly, and the target calls the rocksdb::Write() interface directly

[git-hulk]

Do you mean that we can send the write batch directly instead of the RESP format？

@git-hulk Yup, we send the write batch directly, and the target calls the rocksdb::Write() interface directly

Got it, thanks a lot

[ZENOTME]

I'm interested in this issue, but I'm confused about

With rawkv-based migration, neither the source nor the destination need to judge expire time

Do you mean that we can send the write batch directly instead of the RESP format？

Does it means that: for a key-value pair with expire (k,v,expire), just send the (k,v) and ignore the expire?

[caipengbo]

Does it means that: for a key-value pair with expire (k,v,expire), just send the (k,v) and ignore the expire?

Yes, it ignore the expire. Let the target determine the expire.

[ZENOTME]

Let the target determine the expire.

Do you means the case like:

1. migrate (k,v) to target, and now (k,v) don't have expire
2. And then target get a command like 'EXPIRE k 10', now (k,v) have a new expire.

[caipengbo]

Do you means the case like:

Yes

[git-hulk]

@caipengbo Can you submit an issue to track this?

[caipengbo]

@caipengbo Can you submit an issue to track this?

Yeah, I plan to solve this issue after slot batch PR merged @git-hulk

[git-hulk]

@caipengbo Thanks a lot

[git-hulk]

@caipengbo PR #1534 has been pending for a while. This feature is valuable since it can help the cluster migration avoid depending on the log data of the write batch except for the performance benefit. I'm very eager to see this feature done, but #1534 has too many conflicts and it's a bit hard to review for now.

So I'm not sure if you're willing to continue working on this feature. If yes, I think we can
break down this feature into a few PRs and add more tests like:

• Add the ApplyBatch command
• Implement the merge iterator for iterating SST keys
• Implement WAL iterator for migrating by WAL logs
• Implement the migration by applying raw batch
• Make the raw batch migration parameters configurable

[caipengbo]

So I'm not sure if you're willing to continue working on this feature. If yes, I think we can break down this feature into a few PRs and add more tests.

Indeed, the previous code was too much for CR, and I was happy to go ahead and split it up into multiple PRs. @git-hulk

[git-hulk]

Indeed, the previous code was too much for CR, and I was happy to go ahead and split it up into multiple PRs. @git-hulk

@caipengbo Thank you!

[caipengbo]

I'm going to restart this work. I tracked some tasks in this issue, and I may change the task names in the future PRs.

[git-hulk]

I'm going to restart this work. I tracked some tasks in this issue, and I may change the task names in the future PRs.

cool, can also convert those sub-tasks into tracking issues.