-
Notifications
You must be signed in to change notification settings - Fork 492
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow ChangeFeedProcessor to work with streams #865
Comments
@ealsur what do you think? |
What about a delegate with Technically, we are sending the response to the serializer FromStream. If you don't customize the serializer, it would call the base FromStream azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/Serializer/CosmosJsonDotNetSerializer.cs Line 75 in 0843cae
You could optionally, have your own |
If you look at 871 you'll see that there are a number of internal Cosmos types that are inaccessible to a custom serializer making it unlikely in the near term. Trying the Stream delegate resulted in:
looks like you're unpacking batch. |
I would like the stream from the entire batch. |
Yeah, I guess it would try to unpack and send it as individual items, when it's not possible. We initially did not think about Stream support as this was trying to match the CFP V2 API, but we'll discuss it and see what comes up. |
I have a running version of Stream support for the ChangeFeedProcessor. |
@fuocor Sure, it's here https://github.com/Azure/azure-cosmos-dotnet-v3#contributing when you open a PR, you will be prompted for any requirement |
@ealsur Just a heads up, I gave the azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/Serializer/CosmosJsonSerializerWrapper.cs Line 24 in 0843cae
We have a very similar use case for the change feed processor, except we route into other actors for aggregation. For us the lack of server side filtering means we spend a lot of resources deserializing documents we don't end up needing to aggregate on when a new processor gets kicked off. Our containers are a few TB's, with documents of multiple types, and about 70% of them are used in aggregation. We are migrating to v3 now and our hoping to see some perf wins here. |
I'm a bit confused as to what the end result of this thread ended up being. I have a situation where I have multiple types of records stored in the same container. Is the solution to register my own custom serializer for dependency injection or is the solution to create a delegate of type |
Is your feature request related to a problem?
I have a messaging application exclusively using streams for Cosmos document management. There are numerous instances where none or only partial (de)serialization is required, With the ChangeFeedProcessor you require a typed delegate which flows through the serializer, something I do not want because all I am doing with document is a block blob copy to cold storage.
I am using Orleans with a dedicated service grain responsible for the change feed, providing a convenient means to offload the work from the principal process flow that saved the document.
I will only do partial deserialization in order to determine how the stream will be routed. The current approach of the CFP requires both deserialization and serialization to move a message forward.
Describe the solution you'd like
A delegate providing the raw stream.
Describe alternatives you've considered
Adding an inline method to save to cold storage,
The text was updated successfully, but these errors were encountered: