Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Meta] NoOp replication for remote translogs with NRT segment replication #4507

Closed
9 of 10 tasks
Tracked by #5671
ashking94 opened this issue Sep 14, 2022 · 0 comments
Closed
9 of 10 tasks
Tracked by #5671
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework v2.5.0 'Issues and PRs related to version v2.5.0'

Comments

@ashking94
Copy link
Member

ashking94 commented Sep 14, 2022

We have discussed NoOp replication on high level in #3706 and the proposal can be found here and the approach selected has been discussed here.

As part of the exercise, in brief, we want to achieve the following -

  • Use replication call for primary term validation, but the call itself is no op.
  • Since the replicas don't need the translog, recovery needs to be refactored for replica. We do not need to replay Lucene operations anymore. Similarly, there would be refactoring in replica-primary promotion and replica-replica recovery.
  • PRRL helps to keep a certain history of operations so that if a recovery was to happen, then the recovery can make use of replaying Lucene operations,
  • Global and local checkpoints are used in context of indexing & translog. Since the translog will be stored remotely, we are guaranteed to have translogs always in case of node failure and hence we have to decouple checkpoints interaction between primary and replica. Global / local checkpoint should hold no more relevance with remote translog.
  • Redefine insync shards now can be any replica since we have segments and translogs available remotely.

This issue is a meta-issue for tracking the progress of implementing no-op replication. We will proceed with implementing no-op replication along the following rough plan:

@ashking94 ashking94 added enhancement Enhancement or improvement to existing feature or request untriaged labels Sep 14, 2022
@ashking94 ashking94 added Storage:Durability Issues and PRs related to the durability framework v2.5.0 'Issues and PRs related to version v2.5.0' labels Nov 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework v2.5.0 'Issues and PRs related to version v2.5.0'
Projects
None yet
Development

No branches or pull requests

3 participants
@ashking94 @tlfeng and others