Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support triggering compaction of all files under a given directory #4303

Closed
SandyXSD opened this issue Dec 29, 2023 · 1 comment · Fixed by #4337
Closed

Support triggering compaction of all files under a given directory #4303

SandyXSD opened this issue Dec 29, 2023 · 1 comment · Fixed by #4337
Assignees
Labels
kind/feature New feature or request

Comments

@SandyXSD
Copy link
Contributor

What would you like to be added:
Support triggering compaction of all files under a given directory

Why is this needed:
An alternative light way of gc --compact for large clusters.

@jiefenghuang
Copy link
Contributor

jiefenghuang commented Jan 11, 2024

requirements

  • A new user compact command
  • Support triggering compaction of all files under a given directory

design

  • what we have

    • meta.engine.compact(inode Ino, index uint32, force bool)
  • what we should do

    • Iterate entries under specific path, get all set (inode, type, length)
    • For a file, calculate the chunk count based on its length. The chunk index ranges from 0 to (chunk count-1). Perform 'compact' operation on all chunks(maybe not exists).
      • chunkCount = (fileLength + chunkSize - 1) / chunkSize
      • Ensure that the 'compact' function behaves correctly for non-existent chunks.
      • For files with many chunk holes, performance is poor, but this scenario is infrequent. Currently, it is not being considered.
  • alternative

    • get all inodes under specific path
    • use meta.scan to get all chunks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
2 participants