Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCP pubsub sink should set the message_id depending on settings #710

Closed
pocesar opened this issue Oct 2, 2023 · 2 comments
Closed

GCP pubsub sink should set the message_id depending on settings #710

pocesar opened this issue Oct 2, 2023 · 2 comments

Comments

@pocesar
Copy link

pocesar commented Oct 2, 2023

When emitting the events to pubsub for example, would be nice to have the message_id to be set to arbitrary values defined in the TOML.
For example, I'd want the block_hash to be set as the message_id when receiving block events, transaction_hash when receiving transaction events, etc

And also would be nice to be able to set ordering_key as well, to be able to order blocks, transactions either using a static ordering_key or getter from the field (like in the message_id case)

@falcucci
Copy link

falcucci commented Oct 22, 2023

@pocesar checking about the possibility of this feature,

seems like the publisher has no such control to send a custom message_id based on such rules, once this attr is reserved to be already setted by the server only.

/// ID of this message, assigned by the server when the message is published.
/// Guaranteed to be unique within the topic. This value may be read by a
/// subscriber that receives a `PubsubMessage` via a `Pull` call or a push
/// delivery. It must not be populated by the publisher in a `Publish` call.
#[prost(string, tag = "3")]
pub message_id: ::prost::alloc::string::String,

As an alternative we could maybe proceed with this, adding a custom attribute, meaning that we could send always as possible the block_hash and transaction_hash respectively as field/value to the message.

In that case we should decide about:

  • have both unique custom attributes;
  • unique attr with dynamic value;

which option would fit your use case better?

cc @scarmuega @paulobressan

@pocesar
Copy link
Author

pocesar commented Oct 30, 2023

@falcucci makes sense, but the missing ordering key is actually giving issues when publishing, we need to make sure that the order of blocks are linear

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants