[MRG] add `sourmash sig grep` #1864

ctb · 2022-03-05T15:03:48Z

Implements sourmash sig grep as in #1075.

sig grep does substring matching on name, filename, and md5. It supports regular expressions, as well as the -v (exclude instead of include) and -i (case insensitive) options. It also supports CSV output in manifest format (for use as picklists), and the -c/--count option to just count the number of signatures in each file.

A key feature of grep is that it (by default) requires manifests. That way users won't get caught up accidentally in searches of very large databases w/o manifests, which has been happening to me a lot recently.

Fixes #1075.

Examples

Extract by acccession:

% sourmash sig grep GCF_000019185. /group/ctbrowngrp/gtdb/databases/ctb/gtdb-rs202.genomic.k31.zip -o out.zip

...

loaded 258406 total that matched ksize & molecule type
extracted 1 signatures from 1 file(s)

Fail when no manifest present:

% sourmash sig grep GCF_000019185. /group/ctbrowngrp/gtdb/databases/gtdb-rs202.genomic.k31.zip -o out.zip

...

ERROR on filename '/group/ctbrowngrp/gtdb/databases/gtdb-rs202.genomic.k31.zip'.
sig grep requires a manifest by default, but no manifest present.

Count some things and output a manifest:

% sourmash sig grep -ci shewanella /group/ctbrowngrp/gtdb/databases/ctb/gtdb-rs202.genomic.k31.zip --csv matches.csv

== This is sourmash version 4.2.4.dev27+g0a1b713f. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

(no signatures will be output because of --silent/--count).
252 matches: /group/ctbrowngrp/gtdb/databases/ctb/gtdb-rs202.genomic.k31.zip
wrote manifest containing matches to CSV matches 'matches.csv'

Notes

eventually we should probably add the default manifest requirement to several more sig subcommands, like extract, but this will require a major version bump.

TODO

upgrade PR description with example command
tests needed!
Visit add a sig grep command? #1075 and extract non-grep stuff to new issue
make issue to require manifests in various sig subcomands (sig extract, for example)
write docs

codecov · 2022-03-05T15:10:26Z

Codecov Report

Merging #1864 (76b2b02) into latest (cccd06c) will increase coverage by 8.07%.
The diff coverage is 98.29%.

@@            Coverage Diff             @@
##           latest    #1864      +/-   ##
==========================================
+ Coverage   82.38%   90.46%   +8.07%     
==========================================
  Files         119       91      -28     
  Lines       12944     8859    -4085     
  Branches     1729     1751      +22     
==========================================
- Hits        10664     8014    -2650     
+ Misses       2016      579    -1437     
- Partials      264      266       +2

Flag	Coverage Δ
python	`90.46% <98.29%> (+0.10%)`	⬆️
rust	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/sourmash/manifest.py	`90.83% <93.33%> (+0.32%)`	⬆️
src/sourmash/sig/grep.py	`98.70% <98.70%> (ø)`
src/sourmash/cli/sig/__init__.py	`100.00% <100.00%> (ø)`
src/sourmash/cli/sig/grep.py	`100.00% <100.00%> (ø)`
src/sourmash/sig/__init__.py	`100.00% <100.00%> (ø)`
src/core/tests/storage.rs
src/core/src/ffi/hyperloglog.rs
src/core/src/sketch/hyperloglog/mod.rs
src/core/src/ffi/nodegraph.rs
src/core/tests/minhash.rs
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cccd06c...76b2b02. Read the comment docs.

…o add/sig_grep

ctb · 2022-03-05T17:40:02Z

@taylorreiter @bluegenes @luizirber any thoughts on additional functionality beyond regexps, -v, and -i? this is already an immediately useful command 😆

src/sourmash/sig/grep.py

bluegenes · 2022-03-05T17:52:36Z

@taylorreiter @bluegenes @luizirber any thoughts on additional functionality beyond regexps, -v, and -i? this is already an immediately useful command 😆

Maybe also -c/--count for just counting results you would get (suppressing normal matched output)?

ctb · 2022-03-05T17:58:35Z

Maybe also -c/--count for just counting results you would get (suppressing normal matched output)?

yeah, I like this!

what about other outputs, like -l for showing what files things are in? I can see it maybe being useful.

and another brainstorming thought - I'm guessing it might be handy to support switching the output to a manifest-style CSV so you can use this to create picklists.

…o add/sig_grep

ctb · 2022-03-06T16:55:37Z

Added -c/--count and --csv for outputting manifests of matches.

I think (modulo tests and some refactoring) this PR might be feature complete, at least until we get some experience in using the functionality and have more ideas on what to add. I'll push forward on the tests as I have time, but it's pretty usable already, so let me know if you have any thoughts or comments!

ctb · 2022-03-07T05:00:02Z

Welp, got the tests and docs done sooner than expected. Ready for review & merge @sourmash-bio/devs!

bluegenes

It looks like sig grep works with picklist input as well - picklist is applied first, then pattern match. If yes, I think we need a test for this case (pattern + picklist input). Happy to review again with add'l test, or leave to your discretion.

src/sourmash/cli/sig/grep.py

bluegenes · 2022-03-07T22:29:11Z

src/sourmash/sig/grep.py

+    if picklist:
+        sourmash_args.report_picklist(args, picklist)


does this need a test for grep + picklist? I see one for when this will fail (LCA)...

ooh, nice catch! test_sig_grep_7_picklist_md5_lca_fail tests the failure, but there's no test for the successful use of --picklist. Will add.

added in 76b2b02

looks good!

Co-authored-by: Tessa Pierce Ward <[email protected]>

ctb · 2022-03-07T23:47:54Z

🎉

ctb added 9 commits March 5, 2022 05:31

upgrade 'manifest' documentation, cli help

30bf6b9

alias fileinfo to summarize

f891e11

flakes cleanup

4fb5f99

rescue shadowed tests

7eab2f6

rescue shadowed tests

7feaad7

rescue shadowed tests

31d5586

add 'sig grep' command

c7b63eb

add some basic tests

44979e5

fix get manifest stuff

ebe2334

fail on no manifest

5a311c1

ctb mentioned this pull request Mar 5, 2022

provide simple/standard command-line pattern matching for search and gather. #1865

Closed

ctb added 3 commits March 5, 2022 08:38

check manifest req't

9bbc3f6

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

591c352

…o add/sig_grep

test various combinations of zip, -v, -i

5f6ad7f

bluegenes reviewed Mar 5, 2022

View reviewed changes

src/sourmash/sig/grep.py Outdated Show resolved Hide resolved

Base automatically changed from cleanup/misc to latest March 5, 2022 17:55

ctb added 4 commits March 5, 2022 09:59

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

5a2cce5

…o add/sig_grep

update with CSV output/manifest

c19f31e

added -c/--count

4ff79cf

adjust output

3baa0e2

ctb added 5 commits March 6, 2022 15:17

test fail extract

d2d600e

comment tests better

8b0a815

add test for count

66232fc

update docs

0f248e5

remove warnings

9a00d53

ctb added 3 commits March 6, 2022 18:56

cleanup; create CollectionManifest.filter_rows

00c3afb

create CollectionManifest.filter_on_columns

1072608

minor cleanup

56a8992

ctb changed the title ~~[WIP] add sourmash sig grep~~ [MRG] add sourmash sig grep Mar 7, 2022

Merge branch 'latest' into add/sig_grep

4d460c1

ctb mentioned this pull request Mar 7, 2022

[MRG] add --include-db-pattern and --exclude-db-pattern to search/gather #1871

Merged

5 tasks

Merge branch 'latest' into add/sig_grep

ef4f33f

bluegenes approved these changes Mar 7, 2022

View reviewed changes

ctb and others added 3 commits March 7, 2022 14:38

Merge branch 'latest' into add/sig_grep

351c65e

Update src/sourmash/cli/sig/grep.py

13fcfbd

Co-authored-by: Tessa Pierce Ward <[email protected]>

Add a straight up picklist test

76b2b02

ctb mentioned this pull request Mar 7, 2022

switch to requiring manifests by default for some sig commands #1872

Open

ctb merged commit ba38d14 into latest Mar 7, 2022

ctb deleted the add/sig_grep branch March 7, 2022 23:47

This was referenced Mar 8, 2022

Draft release notes for sourmash v4.3.0 #1873

Closed

deprecate or alias sourmash sig extract #1875

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG] add `sourmash sig grep` #1864

[MRG] add `sourmash sig grep` #1864

ctb commented Mar 5, 2022 •

edited

Loading

codecov bot commented Mar 5, 2022 •

edited

Loading

ctb commented Mar 5, 2022

bluegenes commented Mar 5, 2022 •

edited

Loading

ctb commented Mar 5, 2022

ctb commented Mar 6, 2022

ctb commented Mar 7, 2022

bluegenes left a comment •

edited

Loading

bluegenes Mar 7, 2022

ctb Mar 7, 2022

ctb Mar 7, 2022

bluegenes Mar 7, 2022

ctb commented Mar 7, 2022

[MRG] add sourmash sig grep #1864

[MRG] add sourmash sig grep #1864

Conversation

ctb commented Mar 5, 2022 • edited Loading

Examples

Notes

TODO

codecov bot commented Mar 5, 2022 • edited Loading

Codecov Report

ctb commented Mar 5, 2022

bluegenes commented Mar 5, 2022 • edited Loading

ctb commented Mar 5, 2022

ctb commented Mar 6, 2022

ctb commented Mar 7, 2022

bluegenes left a comment • edited Loading

Choose a reason for hiding this comment

bluegenes Mar 7, 2022

Choose a reason for hiding this comment

ctb Mar 7, 2022

Choose a reason for hiding this comment

ctb Mar 7, 2022

Choose a reason for hiding this comment

bluegenes Mar 7, 2022

Choose a reason for hiding this comment

ctb commented Mar 7, 2022

[MRG] add `sourmash sig grep` #1864

[MRG] add `sourmash sig grep` #1864

ctb commented Mar 5, 2022 •

edited

Loading

codecov bot commented Mar 5, 2022 •

edited

Loading

bluegenes commented Mar 5, 2022 •

edited

Loading

bluegenes left a comment •

edited

Loading