feat: make level_compaction_dynamic_level_bytes default true #2207
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
适用场景:
更多的适用于频繁去写同一个key,给其赋值不同的value值的场景。
总体原理:
改变compact策略
level_compaction_dynamic_level_bytes
原理
如果打开level_compaction_dynamic_level_bytes,则base_level会从默认的Level 1 变成最高层 Level 6,即最开始Level 0会直接compact到Level 6,如果某次compact后,Level 6大小超过256M(target_file_size_base),假设300M,则base_level向上调整,此时base_level变成Level 5,而Level 5的大小上限是300M/10 = 30M,之后Level 0会直接compact到Level 5,如果Level 5超过30M,假设50M,则需要与Level 6进行compact,compact后,Level 5恢复到30M以下,Level 6稍微变大,假设320M,则基于320M继续调整base_level,即Level 5的大小上限,调整为320M/10 = 32M,随着写入持续进行,最终Level 5会超过256M(target_file_size_base),此时base_level需要继续上调,到Level 4,取Level 5和Level 6当前大小较大者,记为MaxSize,则Level 4的大小上限为MaxSize/100,Level 5的大小上限为Level 4大小上限乘以10,依次类推。
实现
在rocksdb中:每次base_level及其大小上限(base_bytes)的调整发生在LogAndApply之后,根据当前每一层的现状来进行调整,实现逻辑在VersionStorageInfo::CalculateBaseBytes()中,大致过程如下:
从first_non_empty_level到最后一层,统计每一层的大小,找出最大者,记为max_level_size,最大者不一定是最后一层,因为有可能在某次compaction后,其他层的大小会大于最后一层
从倒数第二层往上到first_non_empty_level,假设有n层,则cur_level_size = max_level_size/(10^n),cur_level_size是当前first_non_empty_level的新的大小上限
如果cur_level_size > max_bytes_for_level_base(256M),则对cur_level_size除以10继续向上调整first_non_empty_level,直到调整到某一层,cur_level_size <= max_bytes_for_level_base(256M),此时该层为新的base_level,即新的first_non_empty_level,base_size为cur_level_size
然后从base_level开始,往下,每一层对base_size乘以10,当做该层新的大小限制