-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: move streams implementation to storage layer. #2242
Conversation
1. remove logic of tree_id generate, the tree will be implemented when supporting XGROUP commands. 2. move all the helper functions to redis_stream.h.
当Pika内有stream数据时,使用 key * 命令应当可以扫描出stream类型的key,但目前key * 不会返回stream keys,请做一下支持。 |
|
* KEYS * Storage::DoCompactRange() * Storage::PKHScanRange() * Storage::PKHRSranRange()
@@ -25,7 +28,7 @@ using streamID = struct streamID { | |||
// We must store the streamID in memory in big-endian format. This way, our comparison of the serialized streamID byte | |||
// code will be equivalent to the comparison of the uint64_t numbers. | |||
inline void EncodeUint64InBigEndian(char* buf, uint64_t value) const { | |||
if (storage::kLittleEndian) { | |||
if (kLittleEndian) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这也算是通用函数了, 放到 coding.h 里统一起来会不会更合适点
Line 40 in 73687f3
inline void EncodeFixed64(char* buf, uint64_t value) { |
就像这里, 下面的 inline uint64_t DecodeUint64OfBigEndian 同理
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rocksdb 里面 key 应该是默认字典序?stream id 是 data key 的结尾,以大端编码能正常 range scan。pstd::EncodeFixed64 正好是小端编码,所以直接用会报错。
我看 List 的 data key 格式也是类似的,需要对 index 做 range scan,但它是实现了一个单独的 Comparator ,比较的时候还得解析出 index,感觉代码复杂度和效率都会低一些。
@Mixficsol own Review |
a885758
into
OpenAtomFoundation:unstable
…on#2242) * feat: move stream implememtation to storage with some bugs. * fix: fix all problem of stream commands * refactor: adjust stream code structure. 1. remove logic of tree_id generate, the tree will be implemented when supporting XGROUP commands. 2. move all the helper functions to redis_stream.h. * feat: support some basic function of storage layer. * fix: Add licence. * fix: compile problem in macos. * fix: length of stream now using int32_t. * fix: Support some commands and API of blackwidow. * KEYS * Storage::DoCompactRange() * Storage::PKHScanRange() * Storage::PKHRSranRange() * fix: build problem on ubuntu. * fix: remote key lock.
…on#2242) * feat: move stream implememtation to storage with some bugs. * fix: fix all problem of stream commands * refactor: adjust stream code structure. 1. remove logic of tree_id generate, the tree will be implemented when supporting XGROUP commands. 2. move all the helper functions to redis_stream.h. * feat: support some basic function of storage layer. * fix: Add licence. * fix: compile problem in macos. * fix: length of stream now using int32_t. * fix: Support some commands and API of blackwidow. * KEYS * Storage::DoCompactRange() * Storage::PKHScanRange() * Storage::PKHRSranRange() * fix: build problem on ubuntu. * fix: remote key lock.
Issue:#2241
Impl doc:#1717
Main modifications
Stream 元数据格式
同其它五类元数据不一样的是,Stream 没有超时机制,不需要时间戳。所以 Stream 的元数据并没有继承于
BaseMetaValue
,而是独立设计的格式。Stream 的元数据格式如下,其中包括了所有 stream 必要的元数据信息。
元数据在一个单独的 cf 中存储,存储格式为
key - StreamMetaValue
Stream 数据格式
Stream 的数据类似于 key - value 形式,其中 key 是每条消息的 message-id,而 value 是 message 本身的内容。
其中 key 的格式可以复用 BaseDataKey 的格式,包括了 key, version ,data 三个字段,data 存储的是序列化后的 message-id。
而 value 字段无需额外的编码,只需要将 message 序列化即可,具体细节可以看 #1717。
Tree 抽象
在实现 stream base 指令时,无需存储 cgroup 相关的元数据,所以也无需 rax-tree 的抽象了,故删除此部分,未来加入 xgroup 命令支持时,建议用第三个 cf 来存储相关元数据的 pel 数据。
此处删除的类为:
class TreeIDGenerator
结构变更
pika_stream_base.h 和 pika_stream_base.cc 中的定义和实现移动到了 redis_stream.h 和 redis_stream.cc 中。
不支持的 Redis 父类接口
由于 stream 不应该支持设置超时,故其不支持下列 Redis 父类中定义的这些接口:
这些接口如果被调用,会返回错误,它们都以下列形式实现: