Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make level_compaction_dynamic_level_bytes default true #2207

Merged
merged 1 commit into from
Dec 14, 2023

Conversation

wanghenshui
Copy link
Collaborator

@wanghenshui wanghenshui commented Dec 12, 2023

适用场景:
更多的适用于频繁去写同一个key,给其赋值不同的value值的场景。

总体原理:
改变compact策略

level_compaction_dynamic_level_bytes

  1. 原理
    如果打开level_compaction_dynamic_level_bytes,则base_level会从默认的Level 1 变成最高层 Level 6,即最开始Level 0会直接compact到Level 6,如果某次compact后,Level 6大小超过256M(target_file_size_base),假设300M,则base_level向上调整,此时base_level变成Level 5,而Level 5的大小上限是300M/10 = 30M,之后Level 0会直接compact到Level 5,如果Level 5超过30M,假设50M,则需要与Level 6进行compact,compact后,Level 5恢复到30M以下,Level 6稍微变大,假设320M,则基于320M继续调整base_level,即Level 5的大小上限,调整为320M/10 = 32M,随着写入持续进行,最终Level 5会超过256M(target_file_size_base),此时base_level需要继续上调,到Level 4,取Level 5和Level 6当前大小较大者,记为MaxSize,则Level 4的大小上限为MaxSize/100,Level 5的大小上限为Level 4大小上限乘以10,依次类推。

  2. 实现
    在rocksdb中:每次base_level及其大小上限(base_bytes)的调整发生在LogAndApply之后,根据当前每一层的现状来进行调整,实现逻辑在VersionStorageInfo::CalculateBaseBytes()中,大致过程如下:

从first_non_empty_level到最后一层,统计每一层的大小,找出最大者,记为max_level_size,最大者不一定是最后一层,因为有可能在某次compaction后,其他层的大小会大于最后一层
从倒数第二层往上到first_non_empty_level,假设有n层,则cur_level_size = max_level_size/(10^n),cur_level_size是当前first_non_empty_level的新的大小上限
如果cur_level_size > max_bytes_for_level_base(256M),则对cur_level_size除以10继续向上调整first_non_empty_level,直到调整到某一层,cur_level_size <= max_bytes_for_level_base(256M),此时该层为新的base_level,即新的first_non_empty_level,base_size为cur_level_size
然后从base_level开始,往下,每一层对base_size乘以10,当做该层新的大小限制

@wanghenshui wanghenshui changed the title enhance: make level_compaction_dynamic_level_bytes default true feat: make level_compaction_dynamic_level_bytes default true Dec 13, 2023
@AlexStocks AlexStocks merged commit 1488d53 into unstable Dec 14, 2023
15 of 17 checks passed
@wanghenshui wanghenshui deleted the level_compaction_dynamic_level_bytes branch December 15, 2023 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants