Potential modifications to signature format for major releases

[ctb]

Meta-issue to group together all of the issues that propose to modify the signature format for a sourmash 2.0 release.

• add accession/taxonomy info per Metadata - Add taxonomic breakdown to sourmash output  #195
• add actual k-mer strings per optionally (by default?) store the actual k-mers in the signatures #211
• add number of bp & number of sequences that went into signature add number of bp & sequences => signature file #246
• require a license in each signature file explicitly include license in signatures generated by sourmash #242
• provide at least some minimal provenance tracking e.g. provide a mechanism for tracking the source of genomes in SBTs #230 (source of data) and maybe command used to generate if e.g. k-mer trimming or preprocessing was done (Provide a standard format / provenance info for commands used in data preprocessing #269)
• fix default signature type (currently 'rnaseq') signature of type "mrnaseq" ? #58

[luizirber]

Relevant? https://github.com/marbl/Mash/blob/38d7abd3e545eafd067e627ef55fa171f1fc674c/src/mash/schema-1.0.0.json (for specifying what we want in the signatures?)

…

On Sat, Jun 3, 2017 at 10:40 AM, C. Titus Brown ***@***.***> wrote: Meta-issue to group together all of the issues that propose to modify the signature format for a sourmash 2.0 release. add accession/taxonomy info per #195 add actual k-mers strings per #211 add number of bp & number of sequences that went into signature #246 require a license in each signature file #242 provide at least some minimal provenance tracking e.g. #230 (source of data) and maybe command used to generate if e.g. k-mer trimming or preprocessing was done — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ctb]

Other ideas:

• support tagging and simple Folksonomy (although this might be better done via hypothesis and/or some kind of IPLD magic - we probably don't want to allow/require modifications of the signature itself!)

[ctb]

Also: provide some sort of verification/signature tester so that before making a bunch of
signatures public (e.g on IPFS) we can get suggestions on what kind of metadata we should add in.

[luizirber]

a great discussion on hierarchical vs tag-based file systems, which is very relevant for how to store metadata too: https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies

[ctb]

also, remove email (done in #335).

[ctb]

also add an md5sum for the input sample.

[ctb]

@luizirber here's a proposal - any thoughts?

https://hackmd.io/KYdgjATARiCcwFoBmIBsAGBAWYssIENhUBWBEAZhOAgqiQA4pYGg

current text:

Sourmash signatures - metadata thoughts

tl;dr? Keep the core signature format lean and mean, with a few required fields; put other stuff in the metadata attribute, which will be a list of dictionaries. This metadata attribute would be passed along with the signature internally, and reading/writing routines would need to leave it unchanged, but it would not affect md5sums of equality of signatures.

Required fields in a signature:

• class
• license
• hash function
• signatures
• version

Reserved block names.

We should identify some reserved block names that have special meaning. Obvious ones include:

• sample (for information about the sample that was computed upon) - number of bases, name, filename, maybe a download URL, etc.
• provenance for (maybe free form?) provenance info - what command was run on what system, etc.
• ipfs for IPFS data file retrieval information.
• ncbi or ncbi_taxonomy for accessions and/or taxonomy IDs and other such information.

Content of these should be more completely described and then encoded in software & a software validator.

Reserved metadata block proposal: tags

tl; dr? Can we build a useful tagging system in, to support a folksonomy?

See Better file organization around tags not hierarchies via @luizirber.

One thought: here it would be nice to have something that didn't change the actual signature file content, so that the hash didn't change (for e.g. IPFS distribution). Can this be done via some IPFS mechanism (IPNS?) or hypothesis, e.g. specify a unique URI for each signature that could then be annotated in hypothesis. (It looks like the term we want here is "external metadata")

Conundrum: how do we do forwarding?

Another use for external metadata would be forwarding between signatures (e.g. "signature 5e665d is from a sample that has been updated; new signature is 48d23d".) Again, we want to avoid updating the signature content with this information because that would change the hash.

Luiz comments

• IPLD allows traversing IPFS objects.
  I worked a bit on making the SBT JSON valid for IPLD too,
  they have examples using git https://github.com/ipfs/js-ipfs/tree/master/examples/traverse-ipld-graphs

• The annoying thing with metadata in IPFS objects is that any change will generate another hash =/
  But not sure how to best represent it outside, either (if it is another IPFS object, we still need to update the signature to point to it, which defeats the purpose).

  • oh, we could save the minhash in one object, and let different signatures point to the same data.
    this way people can make their own metadata or extras on the signatures,
    but everyone has the same values for the minhash.

• I really like the idea of making tags or extra metadata using hypothesis,
  how easy is to access their data outside the browser?

• Add a 'previous_version' field, pointing to the previous IPFS object.
  This way we can keep some simple versioning info

[ctb]

I am thinking this should be punted to 3.0, given all the other stuff we have in 2.0 already.

[ctb]

punted to... well, 4.0? 5.0? whatever :)

[ctb]

I'm starting to think that maybe a good goal would be to add a flexible selector framework (ref #1072) that would let us do keyword searches to select subsets of signatures based on tags, taxonomy, etc. This could integrate well with a folksonomy-style tagging approach.