Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

files cache improvements #8389

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Commits on Sep 19, 2024

  1. files cache improvements, fixes borgbackup#8385, fixes borgbackup#5658

    - changes to locally stored files cache:
    
      - store as files.<H(archive_name)>
      - user can manually control suffix via env var
      - if local files cache is not found, build from previous archive.
    - enable rebuilding the files cache via loading the previous
      archive's metadata from the repo (better than starting with
      empty files cache and needing to read/chunk/hash all files).
      previous archive == same archive name, latest timestamp in repo.
    - remove AdHocCache (not needed any more, slow)
    - remove BORG_CACHE_IMPL, we only have one
    - remove cache lock (this was blocking parallel backups to same
      repo from same machine/user).
    
    Cache entries now have ctime AND mtime.
    
    Note: TTL and age still needed for discarding removed files.
          But due to the separate files caches per series, the TTL
          was lowered to 2 (from 20).
    ThomasWaldmann committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    a891559 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c100e7b View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. fix race condition with data loss potential, fixes borgbackup#3536

    if we detect the conditions for this (rare) race,
    abort reading the file and retry.
    
    The caller (_process_any) will do up to MAX_RETRIES
    before giving up. If it gives up, a warning is logged
    and the file is not written to the archive and won't
    be memorized in the files cache either.
    
    Thus, the file will be read/chunked/hashed again at
    the next borg create run.
    ThomasWaldmann committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    381c33a View commit details
    Browse the repository at this point in the history
  2. fix race condition with data loss potential, fixes borgbackup#3536

    we discard all files cache entries referring to files
    with timestamps AFTER we started the backup.
    
    so, even in case we would back up an inconsistent file
    that has been changed while we backed it up, we would
    not have a files cache entry for it and would fully
    read/chunk/hash it again in next backup.
    ThomasWaldmann committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    341f02d View commit details
    Browse the repository at this point in the history