Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dbnode] Background cold flush process #2508

Merged
merged 29 commits into from
Aug 6, 2020
Merged

[dbnode] Background cold flush process #2508

merged 29 commits into from
Aug 6, 2020

Commits on Aug 6, 2020

  1. Background cold flush process.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    688d956 View commit details
    Browse the repository at this point in the history

  2. Add a TODO.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    85dbbd1 View commit details
    Browse the repository at this point in the history

  3. Separate out cold flush into its own process.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    d01a98d View commit details
    Browse the repository at this point in the history

  4. Determine if snapshots are warm/cold. Update mocks.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    d7a5e0e View commit details
    Browse the repository at this point in the history

  5. Cold flush needs its own persist manager.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    efa8ce6 View commit details
    Browse the repository at this point in the history

  6. Formatting.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    5aa6bd0 View commit details
    Browse the repository at this point in the history

  7. Remove cleanup mgr from fs mgr.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    b7829f9 View commit details
    Browse the repository at this point in the history

  8. Separate out cleanup logic.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    467c7ef View commit details
    Browse the repository at this point in the history

  9. Update disk snapshot integration test and mocks.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    81545e0 View commit details
    Browse the repository at this point in the history

  10. Fix unit test.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    9dd1713 View commit details
    Browse the repository at this point in the history

  11. Terminate early and cache empty fileset on disk state.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    bba3beb View commit details
    Browse the repository at this point in the history

  12. Check individual block starts for buffer data.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    8b5c75e View commit details
    Browse the repository at this point in the history

  13. Fix unit test.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    7038739 View commit details
    Browse the repository at this point in the history

  14. Add timers for granular view of phases of flush duration.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    c5e0f6b View commit details
    Browse the repository at this point in the history

  15. Fix tests.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    699874f View commit details
    Browse the repository at this point in the history

  16. Cleanup coldflush manager and add a test.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    cdfcfc1 View commit details
    Browse the repository at this point in the history

  17. Update cold flush tests.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    79a2f92 View commit details
    Browse the repository at this point in the history

  18. Address PR comments.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    73b6733 View commit details
    Browse the repository at this point in the history

  19. Address PR comments.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    c3eedc5 View commit details
    Browse the repository at this point in the history

  20. Add test for bootstrapping from snapshots for all block starts.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    031de16 View commit details
    Browse the repository at this point in the history

  21. Verify that the data is consistent after cold flushing.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    43e7132 View commit details
    Browse the repository at this point in the history

  22. Update mocks

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    aec4856 View commit details
    Browse the repository at this point in the history

  23. Refactor bootstrapper initializing logic in integration tests.

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    8e14316 View commit details
    Browse the repository at this point in the history

  24. Make commit log bootstrapper bootstrap the entire original target ran… 

    …ge per shard.
    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    6f69086 View commit details
    Browse the repository at this point in the history

  25. Do not shortcircuit bootstrapping in base bootstrapper. Short circuit… 

    … directly in peers bootstrapper.:
    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    66d8c76 View commit details
    Browse the repository at this point in the history

  26. Pass only target shard ranges to all required. Use copies of shard ra… 

    …nges everywhere.
    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    a68e4c2 View commit details
    Browse the repository at this point in the history

  27. Next bootstrapper gets called in all cases now even in the empty rang… 

    …es and no remamining unfulfilled ranges case.
    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    e96b5ca View commit details
    Browse the repository at this point in the history

  28. Wait on a more deterministic condition (checking snapshot filesets on… 

    … disk).
    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    b26631c View commit details
    Browse the repository at this point in the history

  29. Add comment

    @notbdu
    notbdu committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    5f21c6c View commit details
    Browse the repository at this point in the history