Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor backup restore to handle serialization errors and conflicts #316

Merged
merged 13 commits into from
Jan 2, 2024
Merged

refactor backup restore to handle serialization errors and conflicts #316

merged 13 commits into from
Jan 2, 2024
+821 −165

Commits on Jan 2, 2024

  1. refactor backup import to handle serialization errors and conflicts 

    This commit refactors the logic on the backup command into
a restorer type. The fundamental differences with the previous logic are:
- batch conflicts are detected, and provides to handling strategies: fail, skip or touch
  when using touch, they will also be retried if needed
- serialization errors are detected, and optionally can be retried using
  WithRelationships with TOUCH semantics
- retry strategy uses a backoff

any error on bulk import will cause the stream to close, so attempts to recover
from that by retrying means retrying with another bulk import stream or
using WriteRelationships. The latter was used because it normalizes retries
as BulkImport does not support TOUCH semantics.
    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    298abbd View commit details
    Browse the repository at this point in the history

  2. introduce restorer tests and adjust implementation accordingly

    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    e517327 View commit details
    Browse the repository at this point in the history

  3. better handling of context cancellation

    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    91ac26d View commit details
    Browse the repository at this point in the history

  4. restore: update fallback error codes 

    to include MySQL and Postgres
    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    e7129b3 View commit details
    Browse the repository at this point in the history

  5. restore: add a deadline on fallback WriteRelationships 

    since MySQL sometimes can get locked for a long time
on serialization errors
    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    065515f View commit details
    Browse the repository at this point in the history

  6. restore: better cancelation handling UX

    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    2ef5ab5 View commit details
    Browse the repository at this point in the history

  7. restore: also handle schema writing 

    so the restorer becomes fully self-contained
and facilitate exporting it
    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    b50830a View commit details
    Browse the repository at this point in the history

  8. PR feedback: better type for batchesPerTransaction

    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    fe5e962 View commit details
    Browse the repository at this point in the history

  9. fix panic on deprecated zed restore command 

    the flags were not registered in the deprecated
`zed restore` command when it was added
for back-compat after the command became
`zed backup restore`
    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    e8f8bde View commit details
    Browse the repository at this point in the history

  10. restore: refactor conflict handling flags 

    turned into a string that can be parsed into
an enum
    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    e31deab View commit details
    Browse the repository at this point in the history

  11. restore: improve progress report on retries

    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    ca68115 View commit details
    Browse the repository at this point in the history

  12. only print help on parsing errors

    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    f94a5fe View commit details
    Browse the repository at this point in the history

  13. PR feedback: reuse MustGetString and do not panic on unexpected flag … 

    …value
    @vroldanbet @josephschorr
    vroldanbet authored and josephschorr committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    461353e View commit details
    Browse the repository at this point in the history