Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support runtime chunk deduplication #1507

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Support runtime chunk deduplication #1507

wants to merge 5 commits into from

Commits on Dec 15, 2023

  1. storage: add helper copy_file_range() 

    Add helper copy_file_range() which:
- avoid copy data into userspace
- may support reflink on xfs etc

Signed-off-by: Jiang Liu <[email protected]>
    @jiangliu
    jiangliu committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    f07f16c View commit details
    Browse the repository at this point in the history

  2. storage: implement CasManager to support chunk dedup at runtime 

    Implement CasManager to support chunk dedup at runtime.
The manager provides to major interfaces:
- add chunk data to the CAS database
- check whether a chunk exists in CAS database and copy it to blob file
  by copy_file_range() if the chunk exists.

Signed-off-by: Jiang Liu <[email protected]>
    @jiangliu
    jiangliu committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    6507cd0 View commit details
    Browse the repository at this point in the history

  3. storage: enable chunk deduplication for file cache 

    Enable chunk deduplication for file cache. It works in this way:
- When a chunk is not in blob cache file yet, inquery CAS database
  whether other blob data files have the required chunk. If there's
  duplicated data chunk in other data files, copy the chunk data
  into current blob cache file by using copy_file_range().
- After downloading a data chunk from remote, save file/offset/chunk-id
  into CAS database, so it can be reused later.

Signed-off-by: Jiang Liu <[email protected]>
    @jiangliu
    jiangliu committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    3632389 View commit details
    Browse the repository at this point in the history

  4. storage: add smoking test for chunk dedup 

    Add smoking test case for chunk dedup.

Signed-off-by: Jiang Liu <[email protected]>
    @jiangliu
    jiangliu committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    ef47e51 View commit details
    Browse the repository at this point in the history

  5. storage: add documentation for cas 

    Add documentation for cas.

Signed-off-by: Jiang Liu <[email protected]>
    @jiangliu
    jiangliu committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    7d287c9 View commit details
    Browse the repository at this point in the history