Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add some params for rocksdb write #2538

Merged
merged 13 commits into from
Mar 20, 2024

Conversation

MalikHou
Copy link
Contributor

Add some new params for rocksdb

Background

Frequent write stop/stall in large-volume random write tests

Change

  • add level0_file_num_compaction_trigger
  • add level0_stop_writes_trigger
  • add level0_slowdown_writes_trigger
  • add min_write_buffer_number_to_merge
  • add max_bytes_for_level_base
  • add max_subcompactions

For the sake of compatibility, the parameters in MR still maintain the default parameters of Rocksdb. If necessary, you can test and adjust by yourself.

Result

In the case of a large number of random writes, the write/stall situation can be greatly alleviated through specific parameter combinations.

@github-actions github-actions bot added the ✏️ Feature New feature or request label Mar 20, 2024
@MalikHou
Copy link
Contributor Author

MalikHou commented Mar 20, 2024

!!! This configuration is only reasonable for specific scenarios, please do not imitate it.!!!
eg:

db-instance-num:6
write-buffer-size : 512M
max-write-buffer-num : 3
max-write-buffer-size : 15G
min-write-buffer-number-to-merge : 1
level0-stop-writes-trigger : 10000
level0-slowdown-writes-trigger : 10000
level0-file-num-compaction-trigger : 4 
max_subcompactions :4
max-background-flushes : 6
max-background-compactions : 8
max-background-jobs : 12
target-file-size-base : 512M

before

WechatIMG129


after

WechatIMG693

@wangshao1
Copy link
Collaborator

我理解这些参数本身rocksdb已经支持了动态调节,您是否可以一块修改下pika代码,支持通过config set动态调节这些参数?

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I understand that rocksdb already supports dynamic adjustment of these parameters. Can you modify the pika code together to support dynamic adjustment of these parameters through config set?

include/pika_conf.h Outdated Show resolved Hide resolved
@MalikHou
Copy link
Contributor Author

我理解这些参数本身rocksdb已经支持了动态调节,您是否可以一块修改下pika代码,支持通过config set动态调节这些参数?

好的

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I understand that rocksdb already supports dynamic adjustment of these parameters. Can you modify the pika code together to support dynamic adjustment of these parameters through config set?

OK

@MalikHou
Copy link
Contributor Author

我理解这些参数本身rocksdb已经支持了动态调节,您是否可以一块修改下pika代码,支持通过config set动态调节这些参数?

pls view

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I understand that rocksdb already supports dynamic adjustment of these parameters. Can you modify the pika code together to support dynamic adjustment of these parameters through config set?

pls view

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


Uploading screenshot 2024-03-20 16.58.15.png…

@MalikHou
Copy link
Contributor Author

截屏2024-03-20 16 58 15

@AlexStocks AlexStocks merged commit e6f9a53 into OpenAtomFoundation:unstable Mar 20, 2024
14 checks passed
@AlexStocks AlexStocks added 3.5.5 and removed 3.5.4 labels May 15, 2024
bigdaronlee163 pushed a commit to bigdaronlee163/pika that referenced this pull request Jun 8, 2024
* feat:ad rocksdb write params

* add new param reload

---------

Co-authored-by: shengxin.hou <[email protected]>
cheniujh pushed a commit to cheniujh/pika that referenced this pull request Sep 24, 2024
* feat:ad rocksdb write params

* add new param reload

---------

Co-authored-by: shengxin.hou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.5.5 4.0.0 ✏️ Feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants