-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement a file tree serialiser/de-serialiser #768
Comments
Thinking about this there's 3 options on how to approach this.
I'm thinking for prototyping I'll start with option 1 just to get things working. then if needed we can convert to option 2. |
|
Hey this was desynced from linear. What's the status and in relation to my question I actually preferred 2. |
Oh that's weird, I still need to find out what causes that. This is closed now. I ended up implementing it as option 2. But we don't so much as downgrade to a raw stream. It's just a raw stream that alternates between JSON messages for the tree structure and JSON + raw binary for the file contents. It also handles sending multiple trees consecutively like this. I did not implement any compression to the binary data. It should be fairly easy to apply to it though if we feel the need. |
Did you benchmark it? |
No, I didn't see a need. It's runs fast enough in jest tests. Nothing in the algorithm wouldn't scale very well. |
Specification
With the more complex unix commands being implemented in MatrixAI/Polykey-CLI#32 we need a utility to take a file tree and serealise it over a binary stream. This needs to follow the generator/parser pattern we've used in the past.
A file tree will be structured something like this
The serialiser needs to take this structure and generate a binary webstream ReadableStream. sending all of this information over. We need to determine a structure for the raw binary stream. It needs some way of efficiently encoding the data. Most of the data can be basic JSON. But the file contents need to be raw binary to reduce the amount of binary sent.
The de-serialiser the parsing method we've used before. It must throw errors if we run into badly formatted data.
I'll need to do some research and prototyping into how we want to format the binary stream. It should be possible to start with a JSON message stream and downgrade to a much simplified raw binary stream for sending all of the file contents.
Additional context
Related MatrixAI/Polykey-CLI#32
Tasks
The text was updated successfully, but these errors were encountered: