Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adjust JSON signature format to better support streaming and progressive I/O? #1507

Closed
ctb opened this issue May 8, 2021 · 3 comments
Closed

Comments

@ctb
Copy link
Contributor

ctb commented May 8, 2021

maybe we should adjust JSON signature format to support a collection of individual records, rather than a list?

Right now, the JSON output by sourmash is a list of signature records. There's no reason we couldn't also support multiple individual records, is there?

from #1440 (comment),

reference to explore streaming JSON output - https://medium.com/galvanize/streaming-structured-json-18da4edd4f20

Note: incorporating suggestion from #1093,

right now to combine signatures from multiple files into one, you need to use sourmash sig cat - it'd be nice to permit just cat.

this issue would address that, too, I think.

@ctb
Copy link
Contributor Author

ctb commented Apr 20, 2022

per @luizirber on slack -

good future project: turns out it is not so hard to make a streaming
parser out of serde, so we can load partial data (like specific ksize, or
downsampling while reading the data): https://serde.rs/stream-array.html

@ctb
Copy link
Contributor Author

ctb commented Aug 3, 2022

note from #1097 (comment) -

it seems also like zip files are a better answer to this.

@ctb
Copy link
Contributor Author

ctb commented Aug 3, 2022

basically it's not clear this is worthwhile, given our other signature save formats (.zip etc). We should just be using direct random-access signature formats, as suggested in #1262. Closing.

@ctb ctb closed this as completed Aug 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant