Feature: apply and remove all shared logs #1593
Labels
type/enhancement
Indicates new feature requests
Comments
empiredan
added a commit
that referenced
this issue
Sep 21, 2023
…1594) #1593 In XiaoMi/rdsn#1019 we've written private logs as WAL instead of shared logs, which also means shared log files would never be appended with new mutations. However, obsolete shared logs that had been applied to rocksdb were not removed since then. There is at least 1 shared log file which is never removed. We should change this policy of garbage collection, to delete all obsolete shared log files. To facilitate the garbage collection of shared log files, we also trigger checkpoints which would flush rocksdb data to disk for each replica that has prevented shared log files from being removed for garbage collection. It is necessary to limit the number of submitted replicas that would be flushed at a time to avoid too much consumption of I/O resources which might affect the processing of read/write requests from clients.
empiredan
added a commit
to empiredan/pegasus
that referenced
this issue
Oct 16, 2023
…pache#1594) apache#1593 In XiaoMi/rdsn#1019 we've written private logs as WAL instead of shared logs, which also means shared log files would never be appended with new mutations. However, obsolete shared logs that had been applied to rocksdb were not removed since then. There is at least 1 shared log file which is never removed. We should change this policy of garbage collection, to delete all obsolete shared log files. To facilitate the garbage collection of shared log files, we also trigger checkpoints which would flush rocksdb data to disk for each replica that has prevented shared log files from being removed for garbage collection. It is necessary to limit the number of submitted replicas that would be flushed at a time to avoid too much consumption of I/O resources which might affect the processing of read/write requests from clients.
empiredan
added a commit
to empiredan/pegasus
that referenced
this issue
Oct 19, 2023
…pache#1594) apache#1593 In XiaoMi/rdsn#1019 we've written private logs as WAL instead of shared logs, which also means shared log files would never be appended with new mutations. However, obsolete shared logs that had been applied to rocksdb were not removed since then. There is at least 1 shared log file which is never removed. We should change this policy of garbage collection, to delete all obsolete shared log files. To facilitate the garbage collection of shared log files, we also trigger checkpoints which would flush rocksdb data to disk for each replica that has prevented shared log files from being removed for garbage collection. It is necessary to limit the number of submitted replicas that would be flushed at a time to avoid too much consumption of I/O resources which might affect the processing of read/write requests from clients.
acelyc111
pushed a commit
that referenced
this issue
Oct 19, 2023
…1594) (#1638) #1593 In XiaoMi/rdsn#1019 we've written private logs as WAL instead of shared logs, which also means shared log files would never be appended with new mutations. However, obsolete shared logs that had been applied to rocksdb were not removed since then. There is at least 1 shared log file which is never removed. We should change this policy of garbage collection, to delete all obsolete shared log files. To facilitate the garbage collection of shared log files, we also trigger checkpoints which would flush rocksdb data to disk for each replica that has prevented shared log files from being removed for garbage collection. It is necessary to limit the number of submitted replicas that would be flushed at a time to avoid too much consumption of I/O resources which might affect the processing of read/write requests from clients. This PR is to cherry-pick #1594 into v2.5 to solve issue #1593.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In XiaoMi/rdsn#1019 we've written private logs as WAL instead of shared logs.
However, obsolete shared logs that had been applied to rocksdb were not removed since then. Since we've kept at least 1 shared log file which would never be removed, we should change policy of garbage collection to delete all shared logs.
The text was updated successfully, but these errors were encountered: