Request metadata updates from peers #1700
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tillrohrmann
force-pushed
the
issues/1697
branch
from
3852b8e
to
47d9364
Compare
crates/core/src/metadata/mod.rs
Outdated
Comment on lines
114
to
127
| pub fn version(&self, metadata_kind: MetadataKind) -> Version { | ||
| match metadata_kind { | ||
| MetadataKind::NodesConfiguration => self.nodes_config_version(), | ||
| MetadataKind::Schema => self.schema_version(), | ||
| MetadataKind::PartitionTable => self.partition_table_version(), | ||
| MetadataKind::Logs => self.logs_version(), | ||
| } | ||
| } |
tillrohrmann
force-pushed
the
issues/1697
branch
from
47d9364
to
61f21d5
Compare
tillrohrmann
force-pushed
the
issues/1697
branch
from
61f21d5
to
1cf0814
Compare
There was a problem hiding this comment.
Great work @tillrohrmann as usual. Only a couple of small comments and we should be good to merge.
tillrohrmann
force-pushed
the
issues/1697
branch
from
63eae33
to
7737147
Compare
|
Thanks for the review @AhmedSoliman. I've addressed your comments and rebased onto the latest |
tillrohrmann
force-pushed
the
issues/1697
branch
from
7737147
to
53657e3
Compare
This commit updates the network reactor to check the header of the remote node for newer metadata versions. If the remote node should have newer metadata, the reactor informs the MetadataFetcher which tries to fetch the metadata updates from the remote node. The MetadataFetcher makes sure that requests are regularly sent in case that messages are being lost. Moreover, it ensures that only one node that announced a given metadata version will be asked for the update. This fixes restatedev#1697.
Before, the MetadataMessageHandler panicked when being asked to send the Schema metadata. Now, it will send the metadata.
The underlying assumption is that metadata won't change often. Therefore, we assume that reading from Live<..> to obtain the metadata versions is more efficient in the ConnectionSender.
This commit makes the connection sender stateful by remembering which metadata version it has sent to a peer. Only new metadata versions, except for the NodesConfiguration version, will be sent in order to reduce a few bytes that need to be sent.
Urgency allows to specify how quickly the MetadataManager should react to observed metadata versions. Currently, we support Normal and High. Normal will pick up the observed metadata version on the next regular update interval. High will immediately ask the MetadataManager to sync the metadata from the metadata store.
tillrohrmann
force-pushed
the
issues/1697
branch
from
53657e3
to
54c47d0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit updates the network reactor to check the header of
the remote peer for newer metadata versions. If the remote peer
should have newer metadata, the reactor asks for it.
This fixes #1697.
This PR is based on #1699.