Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow user-specified functional transforms to be applied during deserialization. #398

Open
j-tyler opened this issue Oct 7, 2022 · 0 comments

Comments

@j-tyler
Copy link

j-tyler commented Oct 7, 2022

As an extension of issue 397, allow users to specify functional transforms that, ideally, will be applied during deserialization. This allows users to specify what is most-optimal for their use case. Why is this valuable?

  1. Records may be retrieved from data storage full of low cardinality duplicate strings. Interning these straight into the record can save memory usage.
  2. Records that end up cached should ideally use immutable collections. Allowing user to do this on deserialization can save memory allocation overhead.
  3. As specified in issue 397, utilization of specific libraries like fastutil could be a simple deserialization transform that the user configures.
    ...

I'm sure there are other reasons users would have. Currently my project takes Avro records and re-processes them after deserialization leading to wasted memory allocation overhead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant