External metadata store

[tillrohrmann]

No description provided.

[muhamadazmy]

Discussion

Research

I was reviewing the code that exists right now regarding the metadata and the metastore and I think i can summaries the implementation as follows:

• When a node starts it creates a Local Metadata Store Service instance if (and only if) it has Metadata store role
  • This instance is used as a handler to metastore grpc service
  • The service is then available to be used by other node with the grpc client
• A metastore client is always created (on all node) and this works as the access handler to the local store (over grpc)
• The client is then passed to:
• Metadata manager
  • MetadataManager provides a writer/notifier interface on top of the underlying client
• Admin role
  • This takes metadata object, metadata writer, and metadata client instance
• Worker role
  • This takes metadata object, metadata writer, and metadata client instance

I realized that many metadata related objects are passed around to many components, which makes it hard to maintain and follow. For example Metadata, MetadataWriter, MetadataStoreClient, etc...

Suggestion.

The above design can be confusing when trying to follow up the code because of the many different parts. I think it can be simplified and reduce the amount of objects we have to pass around.

• MetadataManager, is responsible for maintaining the "view" of the node of the metadata. All operation on metadata must go over the Metadata manager.
  • The manager will maintain it's current view/copy of Metadata object.
  • The manager main function is .metadata() which returns a copy of Metadata object.
    • The metadata object is the only way the other components can read/update the metadata. The Metadata object is just a thin layer that access the supporting MetadataManager object.
  • The Metadata object exposes all functionality it has now + some of the low level client functionality (this includes the MetadataWriter + needed low level functionality)
• MetadataManager maintains the only instance of the MetadataStore the store gives the low level functionality that is leveraged by the manager to do it's job. The MetadataStore can actually be renamed to "VersionedStore" because all what it does is maintaining a versioned key/value store. - The VersionedStore (MetadataStore) can then has multiple implementations:
• InMemory
• Local (RocksDB) no HA
• gRPC (just a connection to another (multiple?) nodes that has MetaStore Role)
• External (for example etcd)
• Raft/HA (future implementation)

This way there will be NO need to pass metadata, metadata_writer, and metadata_client around. All components (that needs metadata) can just use Metadata object.

If a node has MetaStore role, the metadata manager itself can become a Handler to the grpc service. This can be done by simply passing the grpc requests to the underlying MetadataStore.

[tillrohrmann]

Thanks for summarizing the metadata store-related components. This looks good to me.

A few remarks regarding the original ideas:

The MetadataManager is indeed the component that is responsible for managing and updating the Metadata view on a Node. It can either request/receive updates from other nodes or directly from the MetadataStore. It has been implemented as a single actor. That's why the MetadataWriter exists that represents the sending end of the command channel on which the MetadataManager reacts (among other things). The MetadataWriter can be used by components that want to update the metadata (write access). Since most components shouldn't have write access to the metadata, it has not been combined with the Metadata struct which holds the read view of the available metadata. Technically, the Metadata struct holds internally the same sender as the MetadataWriter to send commands to the MetadataManager.

The MetadataStoreClient is the client used to interact with the MetadataStore. Right now, it is always used in combination with the MetadataWriter because once we update the metadata in the MetadataStore, we also want to update the Node's view on the metadata which we do by telling the MetadataManager via the MetadataWriter. Maybe here it makes sense to add a convenience wrapper that combines these two structs and ensures that every update to the MetadataStore will be reflected in a local update to the MetadataManager and reduing the number of structs being passed around.

The reason why I wouldn't make the MetadataStoreClient dependent on the MetadataManager is that we are also storing metadata in the MetadataStore which is not maintained by the MetadataManger (e.g. epoch numbers, scheduling plan and potentially other information).

Letting the MetadataManager act as the grpc handler for the existing MetadataStore implementation might be possible. Since we want to be able to run the MetadataStore on a different node we need the variant with the client in any case and that's why we are also using it when running in the same process.

[muhamadazmy]

Thank you for clarifying the reason behind it. This definitely improved my view of the design.

You explained that Metadata and MetadataWriter holds internally the same writer to talk to the metadata manager. That's exactly the reason why I think this should be a single "Metadata" object which is Read/Write, which can then be used to get a ReadOnly instance of itself which has subset of the functionality. This way you won't need to send Metadata and MetadataWriter almost everywhere when both read/write access is needed.

My take on the client is that it exposes the low level store to other components where I think should be guarded by the MetadataManager. The manager should be the only object that can access the store, this will also mean that changes to metadata via the Metadata object will be only handled by the manager, which means it can both update the local view and also propogate the change to other nodes.

The Manager can also react to updates from the underlying store (or notification from other nodes that version has been updated) to fetch the updated version from the store and update the local view. And maybe also notify waiters on changes.

My idea actually involving dropping the MetadataStoreClient, not make it dependent on the manager. The manager instead uses the store implementation internally directly.

Other functionality that is not related to metadata directly (like epoch numbers, ...) can still be exposed over the Manager itself. which will internally use the store to implement.