Draft release notes for sourmash v4.4.0

[ctb]

lists below based on:

git log --oneline v4.3.0..latest

copy/pasted from https://hackmd.io/u4XjptG8SuCmthxM-NtgXw, pls update top comment markdown over there.

---

release notes sourmash v4.4.0

This release contains many new features! Of particular note:

• sourmash now estimates and outputs average nucleotide identity (ANI) based on k-mer measures;
• sourmash sketch translate is no longer unusably slow;
• we provide Mac OS 'arm64' wheels for the new M1 Macs;
• we've added a number of support features for managing large collections of signatures and building very large databases;
• and we've added support for SQLite databases that can be used for storing and searching signatures and doing Kraken-style LCA analysis of genomes and metagenomes.

In addition, we have built updated Genbank genome databases (with contents from March 2022) as well as GTDB R07-RS207 databases; see the prepared databases page. We've also made some benchmarks available for these databases, so you can get some idea of the necessary computational resources for your searches.

Last but by no means least, we have begun providing a number of examples and recipes for using sourmash - see the new sourmash examples Web site!

---

Major new features:

• add ANI output to search, prefetch, and gather ([MRG] distance utility functions to support ANI estimation #1934, [MRG] add some search/gather/prefetch columns to enable ANI estimation #1952, [MRG] standardize and simplify search, prefetch, gather results by using dataclasses #1955, [MRG] Add ani utils to signature, sketchcomparison #1966, [MRG] output estimated ANI from sourmash compare, search, prefetch, and gather #1967, [MRG] do not report untrusted jaccard ANI #2011, 1798: implementation of function to test if cardinality estimate is accurate wrt scale size #2031, [MRG] prevent ANI estimation when sketch size estimate may be inaccurate #2032)
• new GTDB and Genbank database releases ([MRG] update the databases page with GTDB R07-RS207 and Genbank 2022.03 #2013, [MRG] update database download commands to use curl -JLO #2038)
• provide macos arm64 wheels (build macos arm64 wheels #1935)
• support for SQLite databases ([MRG] add sqlite3 implementations for Index, CollectionManifest, and LCA_Database #1808)
• implement sourmash sketch fromfile ([WIP] add support for sourmash sketch fromfile functionality. #1884, [MRG] implement sourmash sketch fromfile #1885, [MRG] create ComputeParameters.from_manifest_row in support of sourmash sketch fromfile #1886, [MRG] provide minimum sourmash version information about sketch fromfile #2009)
• add sourmash sig check for comparing picklists and databases ([MRG] add sourmash sig check for comparing picklists and databases #1907, [MRG] fix sig check manifest rebuilding #1915, [MRG] fix sig check --fail-if-missing #1917)
• add sig collect command ([MRG] add sig collect command #2036) for building standalone manifests from many databases
• Add direct loading of manifest CSVs as sourmash indices ([MRG] Add StandaloneManifestIndex class for direct loading of manifest CSVs #1891)
• add -A/--abundance-from to sig subtract & add sig inflate ([MRG] add -A/--abundance-from to sig subtract & add sig inflate #1889)
• advanced database format documentation ([MRG] add advanced database docs #2025)

Minor new features:

• add -d/--debug to sourmash sig describe; upgrade output errors. ([MRG] add -d/--debug to sourmash sig describe; upgrade output errors. #1782)
• add sum_hashes to sourmash sig describe output. ([MRG] add sum_hashes to sourmash sig describe output. #1882)

Bug fixes:

• catch TypeError in search w/abund vs flat at the command line ([MRG] catch TypeError in search w/abund vs flat at the command line #1928)
• speed up SeqToHashes translate ([MRG] Speed-up SeqToHashes() #1938, [MRG] speed up SeqToHashes translate #1946)

Cleanup and documentation fixes:

• better handle some pickfile errors ([MRG] better handle some pickfile errors #1924)
• remove unnecessary downsampling warnings ([MRG] remove unnecessary downsampling warnings #1971)
• use same wording for dayhoff/hp as for dna/protein ([MRG] cli help text consistency: use same wording for dayhoff/hp as for dna/protein #1929)
• rename covered_bp property to better reflect function ([MRG] rename covered_bp property to better reflect function #2050)

Developer updates:

• provide "protocol" tests for Index, CollectionManifest, and LCA_Database classes ([MRG] provide "protocol" tests for Index, CollectionManifest, and LCA_Database classes #1936)
• remove khmer CI tests ([MRG] Remove khmer from actions #1950)
• Benchmarks for seq_to_hashes in protein mode (Benchmarks for seq_to_hashes in protein mode #1944)
• add some tests for Jaccard output ordering ([MRG] add some tests for Jaccard output ordering #1926)
• Oxidize ZipStorage ([MRG] Oxidize ZipStorage #1909)
• cleanup and commenting of test_index.py tests. ([MRG] cleanup and commenting of test_index.py tests. #1898, [MRG] minor improvements to Index tests #1900)
• rationalize _signatures_with_internal ([MRG] refactor _signatures_with_internal #1896)
• Convert nix to flakes ([MRG] Convert nix to flakes #1904)
• fix docs build ([MRG] fix docs build #1897)
• Fix build/CI and unused imports papercuts (Fix build/CI and unused imports papercuts #1974)
• fix hypothesis CI (Fix hypothesis CI #2028)
• dependabot version updates (Bump r-lib/actions from 1 to 2 #1977, Bump supercharge/redis-github-action from 4687e0c1d27b144de1dadc32967b7ce7e1227c7f to 1.4.0 #1978, Update pytest requirement from ~=6.2.4 to >=6.2.4,<7.2.0 #1979, Bump actions/download-artifact from 2 to 3 #1980, Bump conda-incubator/setup-miniconda from e23d871804685e8c52189e5bd45e9145019f10af to 2.1.1 #1981, Bump actions/cache from 2 to 3 #1982, Bump jinja2 from 3.0.3 to 3.1.1 #1983, Update docutils requirement from <0.18.0,>=0.17.1 to >=0.17.1,<0.19.0 #1984, Update pytest-cov requirement from ~=2.12 to >=2.12,<4.0 #1985, Bump myst-parser from 0.17.0 to 0.17.2 #1986, Update cachetools requirement from <5,>=4 to >=4,<6 #1987, Bump counter from 0.5.3 to 0.5.4 #1988, Bump rayon from 1.5.1 to 1.5.2 #1989, Bump getrandom from 0.2.5 to 0.2.6 #1991, Bump wasm-bindgen-test from 0.3.29 to 0.3.30 #1993, Bump web-sys from 0.3.56 to 0.3.57 #1994, Bump codecov/codecov-action from 1 to 3 #1995, Bump actions/setup-python from 1 to 3 #1996, Bump pypa/cibuildwheel from 2.2.2 to 2.4.0 #1997, Bump actions/upload-artifact from 2 to 3 #1998, Bump pypa/cibuildwheel from 2.4.0 to 2.5.0 #2017, Bump jinja2 from 3.1.1 to 3.1.2 #2019, Bump thiserror from 1.0.30 to 1.0.31 #2020, Bump serde from 1.0.136 to 1.0.137 #2021, Bump num-iter from 0.1.42 to 0.1.43 #2022, Bump serde_json from 1.0.79 to 1.0.80 #2023, Bump docker/setup-qemu-action from 1 to 2 #2042)

[ctb]

https://github.com/sourmash-bio/sourmash/releases/tag/v4.4.0 🎉