Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

feat(bulk-load): bulk load succeed part1 - replica handle bulk load succeed #502

Merged
merged 3 commits into from
Jun 18, 2020

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Jun 16, 2020

The whole bulk load succeed process is like:

  1. meta server set app and all partitions bulk load status is succeed feat(bulk-load): bulk load ingestion part6 - meta handle bulk_load_response during ingestion #500
  2. meta server send bulk_load_request to primary feat(bulk-load): meta server send bulk load request #457
  3. primary handle bulk_load_request
  4. primary broadcast request to secondary feat(bulk-load): group bulk load request in replica group #460
  5. secondary handle bulk_load_request
  6. secondary report cleanup flag to primary
  7. primary report group cleanup flag to meta server
  8. meta handle bulk load succeed

This pull request is about replica handle bulk load succeed, step 3,5,6,7 above.

  • handle bulk_load_request in function do_bulk_load
    • when request status is succeed, local status is ingesting, will call handle_bulk_load_succeed
    • when request status is succeed, local status is succeed, will call handle_bulk_load_finish
    • both primary and secondary will execute this function
  • handle_bulk_load_succeed
    • create checkpoint here, this newest checkpoint will include data ingested from files
    • reset ingestion status and turn replica status to succeed
  • handle_bulk_load_finish
    • remove local bulk load directory who stores downloaded files (function remove_local_bulk_load_dir)
    • reset other local bulk load states and context (function clear_bulk_load_states and another function in further pull request)
    • this function will also be called when bulk load failed and canceling bulk load, so I call it handle_bulk_load_finish
  • report states
    • secondary will report whether itself bulk load context cleaned (function report_bulk_load_states_to_primary)
    • primary will report group flag to meta (function report_group_cleanup_flag)

@hycdong hycdong marked this pull request as ready for review June 16, 2020 08:55
// ThreadPool: THREAD_POOL_REPLICATION
error_code replica_bulk_loader::remove_local_bulk_load_dir(const std::string &bulk_load_dir)
{
if (!utils::filesystem::directory_exists(bulk_load_dir) ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any problem if we return ERR_OK when directory is not exist?

Copy link
Contributor Author

@hycdong hycdong Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is okay, would not cause other problem.

src/dist/replication/lib/bulk_load/replica_bulk_loader.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/bulk_load/replica_bulk_loader.cpp Outdated Show resolved Hide resolved
@hycdong hycdong merged commit d731a58 into XiaoMi:master Jun 18, 2020
@hycdong hycdong deleted the replica_handle_succeed branch June 18, 2020 05:45
hycdong added a commit to hycdong/rdsn that referenced this pull request Jun 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants