lca classify: Argument list too long

[nick-youngblut]

I'm getting an "Argument list too long" error when running sourmash lca classify, and the reason is that query files must be entered as --query QUERY [QUERY ...], and I have many 100's of queries (making the list too long). It would be helpful to add a feature in which the queries can be provided from a file (one query file per line).

Conda env:

# Name                    Version                   Build  Channel
atk                       2.25.90           hb9dd440_1002    conda-forge
bz2file                   0.98                       py_0    conda-forge
ca-certificates           2019.3.9             hecc5488_0    conda-forge
cairo                     1.16.0            ha4e643d_1000    conda-forge
certifi                   2019.3.9                 py36_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
dbus                      1.13.0            h4e0c4b3_1000    conda-forge
expat                     2.2.5             hf484d3e_1002    conda-forge
fontconfig                2.13.1            he4413a7_1000    conda-forge
freetype                  2.10.0               he983fc9_0    conda-forge
gdk-pixbuf                2.32.2                        1    bioconda
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
glib                      2.58.3            hf63aee3_1001    conda-forge
gobject-introspection     1.58.2          py36h2da5eee_1000    conda-forge
graphite2                 1.3.13            hf484d3e_1000    conda-forge
gstreamer                 1.14.4            h66beb1c_1001    conda-forge
gtk2                      2.24.31           hb68c50a_1001    conda-forge
harfbuzz                  2.3.1                h6824563_0    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
ijson                     2.3                        py_1    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
khmer                     3.0.0a2          py36hf484d3e_1    bioconda
kiwisolver                1.0.1           py36h6bb024c_1002    conda-forge
libblas                   3.8.0                4_openblas    conda-forge
libcblas                  3.8.0                4_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran-ng            7.3.0                hdf63c60_0
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0                4_openblas    conda-forge
libpng                    1.6.36            h84994c4_1000    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1
libtiff                   4.0.10            h648cc4a_1001    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.8             h143f9aa_1005    conda-forge
matplotlib                3.0.3                    py36_0    conda-forge
matplotlib-base           3.0.3            py36h167e16e_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.16.2           py36h8b7e671_1    conda-forge
openblas                  0.3.5             h9ac9557_1001    conda-forge
openssl                   1.0.2r               h14c3975_0    conda-forge
pango                     1.40.14           h4ea9474_1004    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
pip                       19.0.3                   py36_0    conda-forge
pixman                    0.34.0            h14c3975_1003    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
pyparsing                 2.3.1                      py_0    conda-forge
pyqt                      5.6.0           py36h13b7fb3_1008    conda-forge
python                    3.6.7             hd21baee_1002    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
qt                        5.6.2             hce4f676_1013    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
scipy                     1.2.1            py36h09a28d5_1    conda-forge
screed                    1.0                        py_1    bioconda
setuptools                40.8.0                   py36_0    conda-forge
sip                       4.18.1          py36hf484d3e_1000    conda-forge
six                       1.12.0                py36_1000    conda-forge
sourmash                  2.0.0a9          py36hfc679d8_0    bioconda
sqlite                    3.26.0            h67949de_1001    conda-forge
tk                        8.6.9             h84994c4_1001    conda-forge
tornado                   6.0.2            py36h516909a_0    conda-forge
wheel                     0.33.1                   py36_0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.9             h516909a_1004    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.7             h14c3975_1000    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-libxt                1.1.5             h14c3975_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge

[ctb]

Hi @nick-youngblut thank you for the issue! I wanted to explore some alternatives to providing lists of signatures in files -- for example,

• what if we provided a --traverse-directory argument that would just do all signatures under a current directory? sourmash search and sourmash index (for example) already support this.
• and/or what about using xargs to run sourmash lca classify with lots of signatures? (see the rm example from the xargs wikipedia page)

[nick-youngblut]

Thanks @ctb for the potential solutions! Both could solutions could definitely work, but each has a potential downside:

• The first has the limitation of not being able to select certain files as input, which can result in the accidental inclusion of extra input files.
• I'm worried that the second option will be substantially less computationally efficient. Is this worry true?

[ctb]

#1044 is heading in the right direction for some of this. A unified loading function per #875 or #978 could include solutions for this, too.

[ctb]

Note to self: in implementing --from-file stuff, be sure to do it in situations where order wrt other arguments doesn't matter.

I'm thinking index, lca index, lca classify, lca summarize, and maybe multigather.

[ctb]

--query-from-file for sourmash lca classify and sourmash lca summarize is implemented in #1059.