diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index fd84cdc3ab..bdc2c28c17 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -136,15 +136,24 @@ jobs:
lints:
name: Lints
runs-on: ubuntu-latest
+ strategy:
+ fail-fast: false
+ matrix:
+ build: [beta, stable]
+ include:
+ - build: beta
+ rust: beta
+ - build: stable
+ rust: stable
steps:
- name: Checkout sources
uses: actions/checkout@v1
- - name: Install stable toolchain
+ - name: Install Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
- toolchain: stable
+ toolchain: ${{ matrix.rust }}
override: true
components: rustfmt, clippy
@@ -171,10 +180,10 @@ jobs:
toolchain: stable
target: wasm32-unknown-unknown
- - uses: actions-rs/install@v0.1
+ - uses: actions-rs/cargo@v1
with:
- crate: wasm-pack
- version: latest
+ command: install
+ args: --force wasm-pack --version 0.10.0
- name: Prepare node for running tests
uses: actions/setup-node@v1
@@ -188,6 +197,14 @@ jobs:
- name: run wasm-pack build
run: wasm-pack build src/core -d ../../pkg
+ - name: Prepare package for NPM publishing
+ working-directory: pkg
+ run: npm pack
+
+ - uses: actions/upload-artifact@v2
+ with:
+ path: 'pkg/sourmash*.tgz'
+
wasm32-wasi:
name: Run tests under wasm32-wasi
runs-on: ubuntu-latest
diff --git a/.github/workflows/rust_publish.yml b/.github/workflows/rust_publish.yml
index 30ab988bbc..2543fb81f7 100644
--- a/.github/workflows/rust_publish.yml
+++ b/.github/workflows/rust_publish.yml
@@ -18,9 +18,10 @@ jobs:
- uses: actions-rs/cargo@v1
with:
command: install
- args: --force wasm-pack --version 0.8.1
+ args: --force wasm-pack --version 0.10.0
- name: run wasm-pack
- run: wasm-pack build src/core -d ../../pkg
+ run: |
+ wasm-pack build src/core -d ../../pkg
- name: Prepare node for NPM publishing
uses: actions/setup-node@v1
@@ -29,7 +30,8 @@ jobs:
registry-url: https://registry.npmjs.org/
- name: Publish to NPM
- run: '(cd pkg && npm publish)'
+ working-directory: pkg
+ run: npm publish
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
diff --git a/README.md b/README.md
index 1df1b800ce..08b7004549 100644
--- a/README.md
+++ b/README.md
@@ -3,14 +3,15 @@
Quickly search, compare, and analyze genomic and metagenomic data sets.
[](http://sourmash.readthedocs.io/en/latest/)
-[](/~https://github.com/dib-lab/sourmash/actions/)
+[](https://gitter.im/sourmash-bio/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+[](/~https://github.com/sourmash-bio/sourmash/actions/)
[](https://anaconda.org/bioconda/sourmash)
-[](https://codecov.io/gh/dib-lab/sourmash)
+[](https://codecov.io/gh/sourmash-bio/sourmash)
[](http://joss.theoj.org/papers/10.21105/joss.00027)
-
+
-
+
Usage:
@@ -80,17 +81,16 @@ $ sourmash --help
```
which will install
-[the latest released version](/~https://github.com/dib-lab/sourmash/releases).
+[the latest released version](/~https://github.com/sourmash-bio/sourmash/releases).
## Support
-Please ask questions and files issues
-[on Github](/~https://github.com/dib-lab/sourmash/issues).
+For questions, please open an issue [on Github](/~https://github.com/sourmash-bio/sourmash/issues), or ask in our [chat](https://gitter.im/sourmash-bio/community?utm_source=share-link&utm_medium=link&utm_campaign=share-link).
## Development
Development happens on github at
-[dib-lab/sourmash](/~https://github.com/dib-lab/sourmash).
+[sourmash-bio/sourmash](/~https://github.com/sourmash-bio/sourmash).
sourmash is developed in Python and Rust, and you will need a Rust
environment to build it; see [the developer notes](doc/developer.md)
@@ -110,6 +110,18 @@ Tests require py.test and can be run with `make test`.
Please see [the developer notes](doc/developer.md) for more information
on getting set up with a development environment.
+## Research notice
+
+Please note that this repository is participating in a study into sustainability
+ of open source projects. Data will be gathered about this repository for
+ approximately the next 12 months, starting from 2021-06-11.
+
+Data collected will include number of contributors, number of PRs, time taken to
+ close/merge these PRs, and issues closed.
+
+For more information, please visit
+[our informational page](https://sustainable-open-science-and-software.github.io/) or download our [participant information sheet](https://sustainable-open-science-and-software.github.io/assets/PIS_sustainable_software.pdf).
+
----
CTB
diff --git a/doc/api-example.md b/doc/api-example.md
index ba82054437..e3c4c72e8e 100644
--- a/doc/api-example.md
+++ b/doc/api-example.md
@@ -395,7 +395,7 @@ True
(Beware, these are confusing techniques for working with hashes that
are easy to get wrong! We suggest
-[posting questions in the issue tracker](/~https://github.com/dib-lab/sourmash/issues)
+[posting questions in the issue tracker](/~https://github.com/sourmash-bio/sourmash/issues)
as you go, if you are interested in exploring this area!)
The hashing function used is identical between num and scaled signatures,
@@ -596,8 +596,8 @@ Now, let's load in all of the signatures from the test directory:
... hashes_inserted = db.insert(sig)
... print(f"Inserted {hashes_inserted} hashes into db.")
Inserted 493 hashes into db.
-Inserted 525 hashes into db.
Inserted 490 hashes into db.
+Inserted 525 hashes into db.
```
diff --git a/doc/classifying-signatures.md b/doc/classifying-signatures.md
index 787c2e2d32..8a15000867 100644
--- a/doc/classifying-signatures.md
+++ b/doc/classifying-signatures.md
@@ -139,7 +139,7 @@ Please see Appendix B, below, for some actual numbers and output.
**Buyer beware:** There are substantial challenges in doing this kind
of analysis on real metagenomic samples, relating to genome representation
-and strain overlap; see [this issue](/~https://github.com/dib-lab/sourmash/issues/461) for a discussion.
+and strain overlap; see [this issue](/~https://github.com/sourmash-bio/sourmash/issues/461) for a discussion.
### Computing signature similarity with angular similarity.
@@ -173,7 +173,7 @@ We suggest the following approach:
* explore the available databases;
-* then ask questions [via the issue tracker](/~https://github.com/dib-lab/sourmash/issues) and we will do our best to help you out!
+* then ask questions [via the issue tracker](/~https://github.com/sourmash-bio/sourmash/issues) and we will do our best to help you out!
This helps us figure out what people are actually interested in doing, and
any help we provide via the issue tracker will eventually be added into the
diff --git a/doc/command-line.md b/doc/command-line.md
index 6489b3167f..8619fb8b8b 100644
--- a/doc/command-line.md
+++ b/doc/command-line.md
@@ -70,9 +70,16 @@ There are seven main subcommands: `sketch`, `compare`, `plot`,
* `prefetch` selects signatures of interest from a very large collection of signatures, for later processing.
There are also a number of commands that work with taxonomic
-information; these are grouped under the `sourmash lca`
-subcommand. See [the LCA tutorial](tutorials-lca.md) for a
-walkthrough of these commands.
+information; these are grouped under the `sourmash tax` and
+`sourmash lca` subcommands.
+
+`sourmash tax` commands:
+
+* `tax metagenome` - summarize metagenome gather results at each taxonomic rank.
+* `tax genome` - summarize single-genome gather results and report most likely classification.
+* `tax annotate` - annotate gather results with lineage information (no summarization or classification).
+
+`sourmash lca` commands:
* `lca classify` classifies many signatures against an LCA database.
* `lca summarize` summarizes the content of metagenomes using an LCA database.
@@ -80,6 +87,9 @@ walkthrough of these commands.
* `lca rankinfo` summarizes the content of a database.
* `lca compare_csv` compares lineage spreadsheets, e.g. those output by `lca classify`.
+> See [the LCA tutorial](tutorials-lca.md) for a
+walkthrough of some of these commands.
+
Finally, there are a number of utility and information commands:
* `info` shows version and software information.
@@ -177,15 +187,14 @@ sourmash compare file1.sig [ file2.sig ... ]
```
Options:
-```
---output -- save the distance matrix to this file (as a numpy binary matrix)
---ksize -- do the comparisons at this k-mer size.
---containment -- calculate containment instead of similarity.
- C(i, j) = size(i intersection j) / size(i).
---from-file -- append the list of files in this text file to the input
+
+* `--output` -- save the distance matrix to this file (as a numpy binary matrix)
+* `--ksize` -- do the comparisons at this k-mer size.
+* `--containment` -- calculate containment instead of similarity; `C(i, j) = size(i intersection j) / size(i)`
+* `--from-file` -- append the list of files in this text file to the input
signatures.
---ignore-abundance -- ignore abundances in signatures.
-```
+* `--ignore-abundance` -- ignore abundances in signatures.
+* `--picklist` -- select a subset of signatures with [a picklist](#using-picklists-to-subset-large-collections-of-signatures)
**Note:** compare by default produces a symmetric similarity matrix that can be used as an input to clustering. With `--containment`, however, this matrix is no longer symmetric and cannot formally be used for clustering.
@@ -249,6 +258,9 @@ similarity match
...
```
+Note, as of sourmash 4.2.0, `search` supports `--picklist`, to
+[select a subset of signatures based on a CSV file](#using-picklists-to-subset-large-collections-of-signatures).
+
### `sourmash gather` - find metagenome members
The `gather` subcommand selects the best reference genomes to use for
@@ -289,6 +301,9 @@ which matches are no longer reported; by default, this is set to
50kb. see the Appendix in
[Classifying Signatures](classifying-signatures.md) for details.
+As of sourmash 4.2.0, `gather` supports `--picklist`, to
+[select a subset of signatures based on a CSV file](#using-picklists-to-subset-large-collections-of-signatures).
+
Note:
Use `sourmash gather` to classify a metagenome against a collection of
@@ -350,6 +365,9 @@ containing a list of file names to index; you can also provide individual
signature files, directories full of signatures, or other sourmash
databases.
+As of sourmash 4.2.0, `index` supports `--picklist`, to
+[select a subset of signatures based on a CSV file](#using-picklists-to-subset-large-collections-of-signatures).
+
### `sourmash prefetch` - select subsets of very large databases for more processing
The `prefetch` subcommand searches a collection of scaled signatures
@@ -375,6 +393,7 @@ Other options include:
* `--threshold-bp` to require a minimum estimated bp overlap for output;
* `--scaled` for downsampling;
* `--force` to continue past survivable errors;
+* `--picklist` select a subset of signatures with [a picklist](#using-picklists-to-subset-large-collections-of-signatures)
### Alternative search mode for low-memory (but slow) search: `--linear`
@@ -402,7 +421,301 @@ This combination of commands ensures that the more time- and
memory-intensive `gather` step is run only on a small set of relevant
signatures, rather than all the signatures in the database.
-## `sourmash lca` subcommands for taxonomic classification
+## `sourmash tax` subcommands for integrating taxonomic information into gather results
+
+The sourmash `tax` or `taxonomy` commands integrate taxonomic
+ information into the results of `sourmash gather`. All `tax` commands
+ require one or more properly formatted `taxonomy` files where the
+ identifiers correspond to those in the database(s) used for
+ `gather`. Note that if using multiple databases, the `gather` needs
+ to have been conducted against all desired databases within the same
+ `gather` command (we cannot combine separate `gather` runs for the
+ same query). For supported databases (e.g. GTDB, NCBI), we provide
+ taxonomy csv files, but they can also be generated for user-generated
+ databases. For more information, see [databases](databases.md).
+
+`tax` commands rely upon the fact that `gather` provides both the total
+ fraction of the query matched to each database matched, as well as a
+ non-overlapping `f_unique_to_query`, which is the fraction of the query
+ uniquely matched to each reference genome. The `f_unique_to_query` for
+ any reference match will always be between (0% of query matched) and 1
+ (100% of query matched), and for a query matched to multiple references,
+ the `f_unique_to_query` will sum to at most 1 (100% of query matched).
+ We use this property to aggregate gather matches at the desired
+ taxonomic rank. For example, if the gather results for a metagenome
+ include results for 30 different strains of a given species, we can sum
+ the fraction uniquely matched to each strain to obtain the fraction
+ uniquely matched to this species. Note that this summarization can
+ also take into account abundance weighting; see
+ [classifying signatures](classifying-signatures.md) for more
+ information.
+
+As with all reference-based analysis, results can be affected by the
+ completeness of the reference database. However, summarizing taxonomic
+ results from `gather` minimizes issues associated with increasing size
+ and redundancy of reference databases.
+
+For more details on how `gather` works and can be used to classify
+ signatures, see [classifying-signatures](classifying-signatures.md).
+
+
+### `sourmash tax metagenome` - summarize metagenome content from `gather` results
+
+`sourmash tax metagenome` summarizes gather results for each query metagenome by
+ taxonomic lineage.
+
+example command to summarize a single `gather csv`, where the query was gathered
+ against `gtdb-rs202` representative species database:
+
+```
+sourmash tax metagenome
+ --gather-csv HSMA33MX_gather_x_gtdbrs202_k31.csv \
+ --taxonomy gtdb-rs202.taxonomy.v2.csv
+```
+
+There are three possible output formats, `csv_summary`, `lineage_summary`, and
+ `krona`.
+
+#### `csv_summary` output format
+
+`csv_summary` is the default output format. This outputs a `csv` with lineage
+ summarization for each taxonomic rank. This output currently consists of six
+ columns, `query_name,rank,fraction,lineage,query_md5,query_filename`, where
+ `fraction` is the fraction of the query matched to the reported rank and
+ lineage.
+
+example `csv_summary` output from the command above:
+
+```
+query_name,rank,fraction,lineage
+HSMA33MX,superkingdom,0.131,d__Bacteria
+HSMA33MX,phylum,0.073,d__Bacteria;p__Bacteroidota
+HSMA33MX,phylum,0.058,d__Bacteria;p__Proteobacteria
+.
+.
+.
+HSMA33MX,species,0.058,d__Bacteria;p__Proteobacteria;c__Gammaproteobacteria;
+o__Enterobacterales;f__Enterobacteriaceae;g__Escherichia;s__Escherichia coli
+HSMA33MX,species,0.057,d__Bacteria;p__Bacteroidota;c__Bacteroidia;
+o__Bacteroidales;f__Bacteroidaceae;g__Prevotella;s__Prevotella copri
+HSMA33MX,species,0.016,d__Bacteria;p__Bacteroidota;c__Bacteroidia;
+o__Bacteroidales;f__Bacteroidaceae;g__Phocaeicola;s__Phocaeicola vulgatus
+```
+> The `query_md5` and `query_filename` columns are omitted here for brevity.
+
+#### `krona` output format
+
+`krona` format is a tab-separated list of these results at a specific rank.
+ The first column, `fraction` is the fraction of the query matched to the
+ reported rank and lineage. The remaining columns are `superkingdom`, `phylum`,
+ ... etc down to the rank used for summarization. This output can be used
+ directly for summary visualization.
+
+To generate `krona`, we add `--output-format krona` to the command above, and
+ need to specify a rank to summarize. Here's the command for reporting `krona`
+ summary at `species` level:
+
+```
+sourmash tax metagenome
+ --gather-csv HSMA33MX_gather_x_gtdbrs202_k31.csv \
+ --taxonomy gtdb-rs202.taxonomy.v2.csv \
+ --output-format krona --rank species
+```
+
+example krona output from this command:
+
+```
+fraction superkingdom phylum class order family genus species
+0.05815279361459521 Bacteria Proteobacteria Gammaproteobacteria Enterobacterales Enterobacteriaceae Escherichia Escherichia coli
+0.05701254275940707 Bacteria Bacteroidetes Bacteroidia Bacteroidales Prevotellaceae Prevotella Prevotella copri
+0.015637726014008795 Bacteria Bacteroidetes Bacteroidia Bacteroidales Bacteroidaceae Bacteroides Bacteroides vulgatus
+```
+
+#### `lineage_summary` output format
+
+The lineage summary format is most useful when comparing across metagenome queries.
+ Each row is a lineage at the desired reporting rank. The columns are each query
+ used for gather, with the fraction match reported for each lineage. This format
+ is commonly used as input for many external multi-sample visualization tools.
+
+To generate `lineage_summary`, we add `--output-format lineage_summary` to the summarize
+ command, and need to specify a rank to summarize. Here's the command for reporting
+ `lineage_summary` for two queries (HSMA33MX, PSM6XBW3) summary at `species` level.
+
+```
+sourmash tax metagenome
+ --gather-csv HSMA33MX_gather_x_gtdbrs202_k31.csv \
+ --gather-csv PSM6XBW3_gather_x_gtdbrs202_k31.csv \
+ --taxonomy gtdb-rs202.taxonomy.v2.csv \
+ --output-format krona --rank species
+```
+
+example `lineage_summary`:
+
+```
+lineage HSMA33MX PSM6XBW3
+d__Bacteria;p__Bacteroidota;c__Bacteroidia;o__Bacteroidales;f__Bacteroidaceae;g__Phocaeicola;s__Phocaeicola vulgatus 0.015637726014008795 0.015642822225843248
+d__Bacteria;p__Bacteroidota;c__Bacteroidia;o__Bacteroidales;f__Bacteroidaceae;g__Prevotella;s__Prevotella copri 0.05701254275940707 0.05703112269838684
+d__Bacteria;p__Proteobacteria;c__Gammaproteobacteria;o__Enterobacterales;f__Enterobacteriaceae;g__Escherichia;s__Escherichia coli 0.05815279361459521 0.05817174515235457
+```
+
+To produce multiple output types from the same command, add the types into the
+ `--output-format` argument, e.g. `--output-format summary krona lineage_summary`
+
+
+### `sourmash tax genome` - classify a genome using `gather` results
+
+`sourmash tax genome` reports likely classification for each query,
+ based on `gather` matches. By default, classification requires at least 10% of
+ the query to be matched. Thus, if 10% of the query was matched to a species, the
+ species-level classification can be reported. However, if 7% of the query was
+ matched to one species, and an additional 5% matched to a different species in
+ the same genus, the genus-level classification will be reported.
+
+Optionally, `genome` can instead report classifications at a desired `rank`,
+ regardless of match threshold (`--rank` argument, e.g. `--rank species`).
+
+Note that these thresholds and strategies are under active testing.
+
+To illustrate the utility of `genome`, let's consider a signature consisting
+ of two different Shewanella strains, `Shewanella baltica OS185 strain=OS185`
+ and `Shewanella baltica OS223 strain=OS223`. For simplicity, we gave this query
+ the name "Sb47+63".
+
+When we gather this signature against the `gtdb-rs202` representatives database,
+we see 66% matches to one strain, and 33% to the other:
+
+abbreviated gather_csv:
+
+```
+f_match,f_unique_to_query,name,query_name
+0.664,0.664,"GCF_000021665.1 Shewanella baltica OS223 strain=OS223, ASM2166v1",Sb47+63
+0.656,0.335,"GCF_000017325.1 Shewanella baltica OS185 strain=OS185, ASM1732v1",Sb47+63
+```
+
+> Here, `f_match` shows that independently, both strains match ~65% percent of
+ this mixed query. The `f_unique_to_query` column has the results of gather-style
+ decomposition. As the OS223 strain had a slightly higher `f_match` (66%), it
+ was the first match. The remaining 33% of the query matched to strain OS185.
+
+We can use `tax genome` on this gather csv to classify our "Sb47+63" mixed-strain query:
+
+```
+sourmash tax genome
+ --gather-csv 47+63_x_gtdb-rs202.gather.csv \
+ --taxonomy gtdb-rs202.taxonomy.v2.csv
+```
+> This command uses the default classification strategy, which uses a
+containment threshold of 0.1 (10%).
+
+There are two possible output formats, `csv_summary` and `krona`.
+
+#### `csv_summary` output format
+
+`csv_summary` is the default output format. This outputs a `csv` with taxonomic
+ classification for each query genome. This output currently consists of six
+ columns, `query_name,rank,fraction,lineage,query_md5,query_filename`, where
+ `fraction` is the fraction of the query matched to the reported rank and lineage.
+ The `status` column provides additional information on the classification:
+
+ - `match` - this query was classified
+ - `nomatch`- this query could not be classified
+ - `below_threshold` - this query was classified at the specified rank,
+ but the query fraction matched was below the containment threshold
+
+Here is the `csv_summary` output from classifying this mixed-strain Shewanella query to
+species level:
+
+```
+query_name,status,rank,fraction,lineage
+"Sb47+63",match,species,1.000,d__Bacteria;p__Proteobacteria;c__Gammaproteobacteria;o__Enterobacterales;f__Shewanellaceae;g__Shewanella;s__Shewanella baltica
+```
+>Here, we see that the match percentages to both strains have been aggregated,
+and we have 100% species-level `Shewanella baltica` annotation. We have omitted
+the `query_md5` and `query_filename` columns for brevity.
+
+#### `krona` output format
+
+`krona` format is a tab-separated list of these results at a specific rank.
+ The first column, `fraction` is the fraction of the query matched to the
+ reported rank and lineage. The remaining columns are `superkingdom`, `phylum`,
+ ... etc down to the rank used for summarization. This output can be used
+ directly for `krona` visualization.
+
+To generate `krona`, we must classify by `--rank` instead of using the
+ classification threshold. For the command, we add `--output-format krona`
+ and `--rank ` to the command above. Here's the command for producing
+ `krona` output for `species`-level classifications:
+
+```
+sourmash tax genome
+ --gather-csv Sb47+63_gather_x_gtdbrs202_k31.csv \
+ --taxonomy gtdb-rs202.taxonomy.v2.csv \
+ --output-format krona --rank species
+```
+> Note that specifying `--rank` forces classification by rank rather than
+by the containment threshold.
+
+Here is the `krona`-formatted output for this command:
+
+```
+fraction superkingdom phylum class order family genus species
+1.0 d__Bacteria p__Proteobacteria c__Gammaproteobacteria o__Enterobacterales f__Shewanellaceae g__Shewanella s__Shewanella baltica
+```
+
+To produce multiple output types from the same command, add the types into the
+ `--output-format` argument, e.g. `--output-format csv_summary krona`.
+ **Note that specifying the classification rank with `--rank`,
+ (e.g. `--rank species`), as needed for `krona` output, forces classification
+ by `rank` rather than by containment threshold.** If the query
+ classification at this rank does not meet the containment threshold
+ (default=0.1), the `status` column will contain `below_threshold`.
+
+
+### `sourmash tax annotate` - annotates gather output with taxonomy
+
+`sourmash tax annotate` adds a column with taxonomic lineage information
+ for each database match to gather output. Do not summarize or classify.
+ Note that this is not required for either `summarize` or `classify`.
+
+By default, `annotate` uses the name of each input gather csv to write an updated
+ version with lineages information. For example, annotating `sample1.gather.csv`
+ would produce `sample1.gather.with-lineages.csv`
+
+```
+sourmash tax annotate
+ --gather-csv Sb47+63_gather_x_gtdbrs202_k31.csv \
+ --taxonomy gtdb-rs202.taxonomy.v2.csv
+```
+> This will produce an annotated gather CSV, `Sb47+63_gather_x_gtdbrs202_k31.with-lineages.csv`
+
+### `sourmash tax prepare` - prepare and/or combine taxonomy files
+
+All `sourmash tax` commands must be given one or more taxonomy files as
+parameters to the `--taxonomy` argument. These files can be either CSV
+files or (as of sourmash 4.2.1) sqlite3 databases. sqlite3 databases
+are much faster for large taxonomies, while CSV files are easier to view
+and modify using spreadsheet software.
+
+`sourmash tax prepare` is a utility function that can ingest and validate
+multiple CSV files or sqlite3 databases, and output a CSV file or a sqlite3
+database. It can be used to combine multiple taxonomies into a single file,
+as well as change formats between CSV and sqlite3.
+
+The following command will take in two taxonomy files and combine them into
+a single taxonomy sqlite database.
+
+```
+sourmash tax prepare --taxonomy file1.csv file2.csv -o tax.db
+```
+
+Input databases formats can be mixed and matched, and the output format
+can be set to CSV like so:
+```
+sourmash tax prepare --taxonomy file1.csv file2.db -o tax.csv -F csv
+```
+
+## `sourmash lca` subcommands for in-memory taxonomy integration
These commands use LCA databases (created with `lca index`, below, or
prepared databases such as
@@ -589,6 +902,9 @@ see
You can use `--from-file` to pass `lca index` a text file containing a
list of file names to index.
+As of sourmash 4.2.0, `lca index` supports `--picklist`, to
+[select a subset of signatures based on a CSV file](#using-picklists-to-subset-large-collections-of-signatures).
+
### `sourmash lca rankinfo` - examine an LCA database
The `sourmash lca rankinfo` command displays k-mer specificity
@@ -818,6 +1134,12 @@ sourmash signature extract tests/test-data/*.fa.sig --name NC_009665
will extract the same signature, which has an accession number of
`NC_009665.1`.
+#### Using picklists with `sourmash sig extract`
+
+As of sourmash 4.2.0, `extract` also supports picklists, a feature by
+which you can select signatures based on values in a CSV file. See
+[Using picklists to subset large collections of signatures](#using-picklists-to-subset-large-collections-of-signatures), below.
+
### `sourmash signature flatten` - remove abundance information from signatures
Flatten the specified signature(s), removing abundances and setting
@@ -920,15 +1242,78 @@ Briefly,
signature files using `--query-from-file` (see below).
* `index` and `lca index` take a few fixed parameters (database name,
- taxonomy spreadsheet) and then an arbitrary number of other files
- that contain signatures, including files, directories, and indexed
- databases. These commands will also take `--from-file` (see below).
+ and for `lca index`, a taxonomy file) and then an arbitrary number of
+ other files that contain signatures, including files, directories,
+ and indexed databases. These commands will also take `--from-file`
+ (see below).
None of these commands currently support searching, comparing, or indexing
signatures with multiple ksizes or moltypes at the same time; you need
to pick the ksize and moltype to use for your search. Where possible,
scaled values will be made compatible.
+### Using picklists to subset large collections of signatures
+
+As of sourmash 4.2.0, many commands support *picklists*, a feature by
+which you can select or "pick out" signatures based on values in a CSV
+file.
+
+For example,
+```
+sourmash sig extract --picklist list.csv:md5:md5sum
+```
+will extract only the signatures that have md5sums matching the
+column `md5sum` in the CSV file `list.csv`. The command
+```
+sourmash sig extract --picklist list.csv::prefetch
+```
+will extract only the signatures found in the output
+of `sourmash prefetch ... -o list.csv`.
+
+The `--picklist` argument string must be of the format
+`pickfile:colname:coltype[:pickstyle]`, where `pickfile` is the path
+to a CSV file, `colname` is the name of the column to select from the
+CSV file (based on the headers in the first line of the CSV file), and
+`coltype` is the type of match. An optional pickstyle argument,
+`:include` or `:exclude`, can be added as a fourth parameter; if
+omitted, the default is `:include`.
+
+The following `coltype`s are currently supported by `sourmash sig extract`:
+
+* `name` - exact match to signature's name
+* `md5` - exact match to signature's md5sum
+* `md5prefix8` - match to 8-character prefix of signature's md5sum
+* `md5short` - same as `md5prefix8`
+* `ident` - exact match to signature's identifier
+* `identprefix` - match to signature's identifier, before '.'
+* `gather` - use the CSV output of `sourmash gather` as a picklist
+* `prefetch` - use the CSV output of `sourmash prefetch` as a picklist
+* `search` - use the CSV output of `sourmash prefetch` as a picklist
+* `manifest` - use the CSV output of `sourmash sig manifest` as a picklist
+
+Identifiers are constructed by using the first space delimited word in
+the signature name.
+
+One way to build a picklist is to use `sourmash sig describe --csv
+out.csv ` to construct an initial CSV file that you can
+then edit further.
+
+The picklist functionality also supports excluding (rather than
+including) signatures matching the picklist arguments. To specify a
+picklist for exclusion, add `:exclude` to the `--picklist` argument
+string, e.g. `pickfile:colname:coltype:exclude`.
+
+For example,
+```
+sourmash sig extract --picklist list.csv:md5:md5sum:exclude
+```
+will extract only the signatures that have md5sums that **do not** match
+entries in the column `md5sum` in the CSV file `list.csv`.
+
+In addition to `sig extract`, the following commands support
+`--picklist` selection: `index`, `search`, `gather`, `prefetch`,
+`compare`, `index`, and `lca index`.
+
### Storing (and searching) signatures
Backing up a little, there are many ways to store and search
@@ -1031,5 +1416,5 @@ signatures that were just created.
(This is a relatively new feature as of 3.4 and our testing may need
some work, so please
-[let us know](/~https://github.com/dib-lab/sourmash/issues) if there's
+[let us know](/~https://github.com/sourmash-bio/sourmash/issues) if there's
something that doesn't work and we will fix it :).
diff --git a/doc/databases.md b/doc/databases.md
index d0830635ed..d2b2308473 100644
--- a/doc/databases.md
+++ b/doc/databases.md
@@ -1,86 +1,37 @@
-# Prepared search databases
+# Prepared databases
-We provide several databases for download. Note that these databases can
-be used with both sourmash v3.5 and sourmash v4.0.
+## GTDB R06-rs202 - DNA databases
-## RefSeq microbial genomes - SBT
+All files below are available under https://osf.io/wxf9z/. The GTDB taxonomy spreadsheet (in a format suitable for `sourmash lca index`) is available [here](https://osf.io/p6z3w/).
-These database are formatted for use with `sourmash search` and
-`sourmash gather`. They are calculated with a scaled value of 2000.
+For each k-mer size, three databases are available.
-Approximately 91,000 microbial genomes (including viral and fungal)
-from NCBI RefSeq.
+* Zipfile collections can be used for a linear search. The signatures were calculated with a scaled of 1000, which robustly supports searches for ~10kb or larger matches.
+* SBT databases are indexed versions of the Zipfile collections that support faster search. They are also indexed with scaled=1000.
+* LCA databases are indexed versions of the Zipfile collections that also contain taxonomy information and can be used with regular search as well as with [the `lca` subcommands for taxonomic analysis](https://sourmash.readthedocs.io/en/latest/command-line.html#sourmash-lca-subcommands-for-taxonomic-classification). They are indexed with scaled=10,000, which robustly supports searches for 100kb or larger matches.
-* [RefSeq k=21, 2018.03.29][0] - 3.3 GB - [manifest](https://osf.io/wamfk/download)
-* [RefSeq k=31, 2018.03.29][1] - 3.3 GB - [manifest](https://osf.io/x3aut/download)
-* [RefSeq k=51, 2018.03.29][2] - 3.4 GB - [manifest](https://osf.io/zpkau/download)
+You can read more about the different database and index types [here](https://sourmash.readthedocs.io/en/latest/command-line.html#indexed-databases).
-## Genbank microbial genomes - SBT
+Legacy databases are available [here](legacy-databases.md)
-These database are formatted for use with `sourmash search` and
-`sourmash gather`.
+Note that the SBT and LCA databases can be used with sourmash v3.5 and later, while Zipfile collections can only be used with sourmash v4.1.0 and up.
-Approximately 98,000 microbial genomes (including viral and fungal)
-from NCBI Genbank.
+### GTDB genomic representatives (47.8k genomes)
-* [Genbank k=21, 2018.03.29][3] - 3.9 GB - [manifest](https://osf.io/vm5kb/download)
-* [Genbank k=31, 2018.03.29][4] - 3.9 GB - [manifest](https://osf.io/p87ec/download)
-* [Genbank k=51, 2018.03.29][5] - 3.9 GB - [manifest](https://osf.io/cbxg9/download)
+The GTDB genomic representatives are a low-redundancy subset of Genbank genomes.
-### Details
+| K-mer size | Zipfile collection | SBT | LCA |
+| -------- | -------- | -------- | ---- |
+| 21 | [download (1.3 GB)](https://osf.io/jp5zh/download) | [download (2.6 GB)](https://osf.io/py92w/download) | [download (114 MB)](https://osf.io/gk2za/download) |
+| 31 | [download (1.3 GB)](https://osf.io/nqmau/download) | [download (2.6 GB)](https://osf.io/w4bcm/download) | [download (131 MB)](https://osf.io/ypsjq/download) |
+| 51 | [download (1.3 GB)](https://osf.io/px6qd/download) | [download (2.6 GB)](https://osf.io/rv9zp/download) | [download (137 MB)](https://osf.io/297dp/download) |
-The individual signatures for the above SBTs were calculated as follows:
+### GTDB all genomes (258k genomes)
-```
-sourmash compute -k 4,5 \
- -n 2000 \
- --track-abundance \
- --name-from-first \
- -o {output} \
- {input}
+These databases contain the complete GTDB collection of 258,406 genomes.
-sourmash compute -k 21,31,51 \
- --scaled 2000 \
- --track-abundance \
- --name-from-first \
- -o {output} \
- {input}
-```
-
-See [github.com/dib-lab/sourmash_databases](/~https://github.com/dib-lab/sourmash_databases) for a Snakemake workflow
-to build the databases.
-
-[0]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/refseq-k21.sbt.zip
-[1]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/refseq-k31.sbt.zip
-[2]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/refseq-k51.sbt.zip
-
-[3]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/genbank-k21.sbt.zip
-[4]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/genbank-k31.sbt.zip
-[5]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/genbank-k51.sbt.zip
-
-## Genbank LCA Database
-
-These databases are formatted for use with `sourmash lca`; they are
-v2 LCA databases and will work with sourmash v2.0a11 and later.
-They are calculated with a scaled value of 10000 (1e5).
-
-Approximately 87,000 microbial genomes (including viral and fungal)
-from NCBI Genbank.
-
-* [Genbank k=21, 2017.11.07](https://osf.io/d7rv8/download), 109 MB
-* [Genbank k=31, 2017.11.07](https://osf.io/4f8n3/download), 120 MB
-* [Genbank k=51, 2017.11.07](https://osf.io/nemkw/download), 125 MB
-
-### Details
-
-The above LCA databases were calculated as follows:
-
-```
-sourmash lca index genbank-genomes-taxonomy.2017.05.29.csv \
- genbank-k21.lca.json.gz -k 21 --scaled=10000 \
- -f --traverse-directory .sbt.genbank-k21 --split-identifiers
-```
-
-See
-[github.com/dib-lab/2018-ncbi-lineages](/~https://github.com/dib-lab/2018-ncbi-lineages)
-for information on preparing the genbank-genomes-taxonomy file.
+| K-mer size | Zipfile collection | SBT | LCA |
+| -------- | -------- | -------- | ---- |
+| 21 | [download (7.8 GB)](https://osf.io/vgex4/download) | [download (15 GB)](https://osf.io/ar67j/download) | [download (266 MB)](https://osf.io/hm3c4/download) |
+| 31 | [download (7.8 GB)](https://osf.io/94mzh/download) | [download (15 GB)](https://osf.io/dmsz8/download) | [download (286 MB)](https://osf.io/9xdg2/download) |
+| 51 | [download (7.8 GB)](https://osf.io/x9cdp/download) | [download (15 GB)](https://osf.io/8fc3t/download) | [download (299 MB)](https://osf.io/3cdp6/download) |
diff --git a/doc/developer.md b/doc/developer.md
index 088d9ab2b6..c2929b43d6 100644
--- a/doc/developer.md
+++ b/doc/developer.md
@@ -4,7 +4,7 @@
You can get the latest development branch with:
```
-git clone /~https://github.com/dib-lab/sourmash.git
+git clone /~https://github.com/sourmash-bio/sourmash.git
```
sourmash runs under Python 3.7 and later.
@@ -102,8 +102,8 @@ We use [GitHub Actions][2] for continuous integration.
Code coverage can be viewed interactively at [codecov.io][1].
-[1]: https://codecov.io/gh/dib-lab/sourmash/
-[2]: /~https://github.com/dib-lab/sourmash/actions
+[1]: https://codecov.io/gh/sourmash-bio/sourmash/
+[2]: /~https://github.com/sourmash-bio/sourmash/actions
## Code organization
diff --git a/doc/index.md b/doc/index.md
index b4517a6412..75d83e2f55 100644
--- a/doc/index.md
+++ b/doc/index.md
@@ -24,7 +24,7 @@ Please also see the `mash` [software](http://mash.readthedocs.io/en/latest/) and
[paper (Ondov et al., 2016)](http://dx.doi.org/10.1186/s13059-016-0997-x) for
background information on how and why MinHash works.
-**Questions? Thoughts?** Ask us on the [sourmash issue tracker](/~https://github.com/dib-lab/sourmash/issues/)!
+**Questions? Thoughts?** Ask us on the [sourmash issue tracker](/~https://github.com/sourmash-bio/sourmash/issues/)!
**Want to migrate to sourmash v4?** sourmash v4 is now available, and
has a number of incompatibilites with v2 and v3. Please see
@@ -71,15 +71,15 @@ be stored, searched, explored, and taxonomically annotated.
* `sourmash` relies on an underlying Rust core for performance.
-* `sourmash` is developed [on GitHub](/~https://github.com/dib-lab/sourmash)
+* `sourmash` is developed [on GitHub](/~https://github.com/sourmash-bio/sourmash)
and is **freely and openly available** under the BSD 3-clause license.
- Please see [the README](/~https://github.com/dib-lab/sourmash/blob/latest/README.md)
+ Please see [the README](/~https://github.com/sourmash-bio/sourmash/blob/latest/README.md)
for more information on development, support, and contributing.
You can take a look at sourmash analyses on real data
-[in a saved Jupyter notebook](/~https://github.com/dib-lab/sourmash/blob/latest/doc/sourmash-examples.ipynb),
+[in a saved Jupyter notebook](/~https://github.com/sourmash-bio/sourmash/blob/latest/doc/sourmash-examples.ipynb),
and experiment with it yourself
-[interactively in a Jupyter Notebook](https://mybinder.org/v2/gh/dib-lab/sourmash/latest?filepath=doc%2Fsourmash-examples.ipynb)
+[interactively in a Jupyter Notebook](https://mybinder.org/v2/gh/sourmash-bio/sourmash/latest?filepath=doc%2Fsourmash-examples.ipynb)
at [mybinder.org](http://mybinder.org).
## Installing sourmash
@@ -94,7 +94,7 @@ or conda:
$ conda install -c conda-forge -c bioconda sourmash
```
-Please see [the README file in github.com/dib-lab/sourmash](/~https://github.com/dib-lab/sourmash/blob/latest/README.md)
+Please see [the README file in github.com/sourmash-bio/sourmash](/~https://github.com/sourmash-bio/sourmash/blob/latest/README.md)
for more information.
## Memory and speed
@@ -104,7 +104,7 @@ many other software programs used for genome search and taxonomic
classification.
`sourmash search` and `sourmash gather` can be used to search 100k
-genbank microbial genomes ([using our prepared databases](databases.md)
+genbank microbial genomes ([using our prepared databases](databases.md))
with about 20 GB of disk and in under 1 GB of RAM.
Typically a search for a single genome takes about 30 seconds on a laptop.
diff --git a/doc/legacy-databases.md b/doc/legacy-databases.md
new file mode 100644
index 0000000000..5b8312ae9d
--- /dev/null
+++ b/doc/legacy-databases.md
@@ -0,0 +1,112 @@
+# Legacy Databases
+
+Sourmash databases have evolved over time.
+We have changed how the database is stored (uncompressed `.zip`) and how we name each signature.
+All SBT databases below are in `.sbt.zip` format.
+Note that the SBT and LCA databases can be used with sourmash v3.5 and later, while Zipfile collections can only be used with sourmash v4.1.0 and up.
+We detail these changes below, and include links to legacy databases.
+See [github.com/sourmash-bio/databases](/~https://github.com/sourmash-bio/databases) for a Snakemake workflow that builds current and legacy databases.
+
+## Sourmash signature names
+
+Earlier versions of sourmash databases were built using individual signatures that were calculated as follows:
+
+```
+sourmash compute -k 4,5 \
+ -n 2000 \
+ --track-abundance \
+ --name-from-first \
+ -o {output} \
+ {input}
+
+sourmash compute -k 21,31,51 \
+ --scaled 2000 \
+ --track-abundance \
+ --name-from-first \
+ -o {output} \
+ {input}
+```
+
+We moved away from this strategy because `--name-from-first` named each signature from the name of the first sequence in the FASTA file.
+While the species name of the organism was present in this name, the accession number corresponded to the accession of the first sequence fragment in the file, not the genome assembly.
+As such, we revised our strategy so that signatures are named by genome assembly accession and species name.
+This requires the `assembly_summary.txt` file to be parsed.
+
+## Sourmash database compression
+
+## Legacy databases
+
+### RefSeq microbial genomes - SBT
+
+These database are formatted for use with `sourmash search` and
+`sourmash gather`. They are calculated with a scaled value of 2000.
+
+Approximately 91,000 microbial genomes (including viral and fungal)
+from NCBI RefSeq.
+
+* [RefSeq k=21, 2018.03.29][0] - 3.3 GB - [manifest](https://osf.io/wamfk/download)
+* [RefSeq k=31, 2018.03.29][1] - 3.3 GB - [manifest](https://osf.io/x3aut/download)
+* [RefSeq k=51, 2018.03.29][2] - 3.4 GB - [manifest](https://osf.io/zpkau/download)
+
+### Genbank microbial genomes - SBT
+
+These database are formatted for use with `sourmash search` and
+`sourmash gather`.
+
+Approximately 98,000 microbial genomes (including viral and fungal)
+from NCBI Genbank.
+
+* [Genbank k=21, 2018.03.29][3] - 3.9 GB - [manifest](https://osf.io/vm5kb/download)
+* [Genbank k=31, 2018.03.29][4] - 3.9 GB - [manifest](https://osf.io/p87ec/download)
+* [Genbank k=51, 2018.03.29][5] - 3.9 GB - [manifest](https://osf.io/cbxg9/download)
+
+
+[0]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/refseq-k21.sbt.zip
+[1]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/refseq-k31.sbt.zip
+[2]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/refseq-k51.sbt.zip
+
+[3]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/genbank-k21.sbt.zip
+[4]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/genbank-k31.sbt.zip
+[5]: https://sourmash-databases.s3-us-west-2.amazonaws.com/zip/genbank-k51.sbt.zip
+
+### Genbank microbial genomes - LCA
+
+These databases are formatted for use with `sourmash lca`; they are
+v2 LCA databases and will work with sourmash v2.0a11 and later.
+They are calculated with a scaled value of 10000 (1e5).
+
+Approximately 87,000 microbial genomes (including viral and fungal)
+from NCBI Genbank.
+
+* [Genbank k=21, 2017.11.07](https://osf.io/d7rv8/download), 109 MB
+* [Genbank k=31, 2017.11.07](https://osf.io/4f8n3/download), 120 MB
+* [Genbank k=51, 2017.11.07](https://osf.io/nemkw/download), 125 MB
+
+
+The above LCA databases were calculated as follows:
+
+```
+sourmash lca index genbank-genomes-taxonomy.2017.05.29.csv \
+ genbank-k21.lca.json.gz -k 21 --scaled=10000 \
+ -f --traverse-directory .sbt.genbank-k21 --split-identifiers
+```
+
+See
+[github.com/dib-lab/2018-ncbi-lineages](/~https://github.com/dib-lab/2018-ncbi-lineages)
+for information on preparing the genbank-genomes-taxonomy when signatures are generated using `--name-from-first`.
+
+### GTDB databases - SBT
+
+All files below are available [here](https://osf.io/wxf9z/).
+
+Release 89
+
+* [GTDB k=31, release 89](https://osf.io/5mb9k/download)
+
+Release 95
+
+* [GTDB k=21, scaled=1000](https://osf.io/4yhe2/download)
+* [GTDB k=31, scaled=1000](https://osf.io/4n3m5/download)
+* [GTDB k=51, scaled=1000](https://osf.io/c8wj7/download)
+
+
diff --git a/doc/more-info.md b/doc/more-info.md
index b98150d694..f370fd2130 100644
--- a/doc/more-info.md
+++ b/doc/more-info.md
@@ -112,7 +112,7 @@ tries to connect to X11 to use the Tkinter backend.
The solution is to force the use of the 'Agg' backend in matplotlib;
see [this stackoverflow answer](https://stackoverflow.com/a/34294056)
-or [this sourmash issue comment](/~https://github.com/dib-lab/sourmash/issues/254#issuecomment-304274590).
+or [this sourmash issue comment](/~https://github.com/sourmash-bio/sourmash/issues/254#issuecomment-304274590).
Newer versions of matplotlib do not seem to have this problem.
@@ -120,7 +120,7 @@ Newer versions of matplotlib do not seem to have this problem.
[1]:/~https://github.com/edawson/rkmh
[2]:/~https://github.com/lskatz/mashtree/blob/master/README.md
[3]:/~https://github.com/onecodex/finch-rs
-[4]:/~https://github.com/dib-lab/sourmash/blob/latest/utils/compute-dna-mh-another-way.py
+[4]:/~https://github.com/sourmash-bio/sourmash/blob/latest/utils/compute-dna-mh-another-way.py
[5]:http://ivory.idyll.org/blog/2016-sourmash.html
[6]:http://ivory.idyll.org/blog/2016-sourmash-signatures.html
[7]:http://ivory.idyll.org/blog/2016-sourmash-sbt.html
diff --git a/doc/release.md b/doc/release.md
index 56741fc1ca..de093f591f 100644
--- a/doc/release.md
+++ b/doc/release.md
@@ -29,7 +29,7 @@ and also check if the [rendered docs] are updated.
1\. The below should be done in a clean checkout:
```
cd $(mktemp -d)
-git clone git@github.com:dib-lab/sourmash.git
+git clone git@github.com:sourmash-bio/sourmash.git
cd sourmash
```
@@ -46,7 +46,7 @@ git tag -a v${new_version}${rc}
git push --tags origin
```
-[the releases page]: /~https://github.com/dib-lab/sourmash/releases
+[the releases page]: /~https://github.com/sourmash-bio/sourmash/releases
3\. Test the release candidate. Bonus: repeat on macOS:
```
@@ -62,7 +62,7 @@ python -m venv testenv4
cd testenv1
source bin/activate
-git clone --depth 1 --branch v${new_version}${rc} /~https://github.com/dib-lab/sourmash.git
+git clone --depth 1 --branch v${new_version}${rc} /~https://github.com/sourmash-bio/sourmash.git
cd sourmash
python -m pip install -U setuptools pip wheel setuptools_scm
python -m pip install -r requirements.txt
@@ -74,7 +74,7 @@ cd ../../testenv2
deactivate
source bin/activate
python -m pip install -U setuptools pip wheel setuptools_scm
-python -m pip install -e git+/~https://github.com/dib-lab/sourmash.git@v${new_version}${rc}#egg=sourmash[test]
+python -m pip install -e git+/~https://github.com/sourmash-bio/sourmash.git@v${new_version}${rc}#egg=sourmash[test]
cd src/sourmash
make test
make dist
@@ -145,7 +145,7 @@ git push --delete origin v${new_version}${rc}
3\. Upload wheels from GitHub Releases to PyPI
-[GitHub Actions will automatically build wheels and upload them to GitHub Releases](/~https://github.com/dib-lab/sourmash/actions?query=workflow%3Acibuildwheel).
+[GitHub Actions will automatically build wheels and upload them to GitHub Releases](/~https://github.com/sourmash-bio/sourmash/actions?query=workflow%3Acibuildwheel).
This will take about 45 minutes, or more. After they're built, they must be
copied over to PyPI manually.
diff --git a/doc/requirements.md b/doc/requirements.md
index 95f2b2dfbd..d6a5353be9 100644
--- a/doc/requirements.md
+++ b/doc/requirements.md
@@ -15,4 +15,4 @@ under Python 3.7 and later. Please see [the development repository README][0]
for
information on source code, tests, and continuous integration.
-[0]:/~https://github.com/dib-lab/sourmash/blob/latest/README.md
+[0]:/~https://github.com/sourmash-bio/sourmash/blob/latest/README.md
diff --git a/doc/sourmash-sketch.md b/doc/sourmash-sketch.md
index f136570d32..2c9852c74e 100644
--- a/doc/sourmash-sketch.md
+++ b/doc/sourmash-sketch.md
@@ -91,7 +91,7 @@ The output signature(s) will be saved in locations that depend on your input par
`sourmash sketch protein` and `sourmash sketch translate` output protein sketches by default, but can also use the `dayhoff` and `hp` encodings. The [Dayhoff encoding](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-367/tables/1) collapses multiple amino acids into a smaller alphabet so that amino acids that share biochemical properties map to the same character. The hp encoding divides amino acids into hydrophobic and polar (hydrophilic) amino acids, collapsing amino acids with hydrophobic side chains together and doing the same for polar amino acids.
-We are still in the process of benchmarking these encodings; ask [on the issue tracker](/~https://github.com/dib-lab/sourmash/issues) if you are interested in updates.
+We are still in the process of benchmarking these encodings; ask [on the issue tracker](/~https://github.com/sourmash-bio/sourmash/issues) if you are interested in updates.
### Parameter strings
@@ -121,7 +121,7 @@ These were chosen by a committee of PhDs as being good defaults for an initial a
More seriously, the DNA parameters were chosen based on the analyses done by Koslicki and Falush in [MetaPalette: a k-mer Painting Approach for Metagenomic Taxonomic Profiling and Quantification of Novel Strain Variation](https://msystems.asm.org/content/1/3/e00020-16).
-The protein, dayhoff, and hp parameters were selected based on unpublished research results and/or magic formulas. We are working on publishing the results! Please ask on the [issue tracker](/~https://github.com/dib-lab/sourmash/issues) if you are curious.
+The protein, dayhoff, and hp parameters were selected based on unpublished research results and/or magic formulas. We are working on publishing the results! Please ask on the [issue tracker](/~https://github.com/sourmash-bio/sourmash/issues) if you are curious.
### More complex parameter string examples
@@ -185,4 +185,4 @@ You can use `sourmash sig describe` to get detailed information about the conten
We try to provide good documentation and error messages, but may not succeed in answer all your questions! So we're happy to help out!
-Please post questions [on the sourmash issue tracker](/~https://github.com/dib-lab/sourmash/issues). If you find something confusing or buggy about the documentation or about sourmash, we'd love to fix it -- for you *and* for everyone else!
+Please post questions [on the sourmash issue tracker](/~https://github.com/sourmash-bio/sourmash/issues). If you find something confusing or buggy about the documentation or about sourmash, we'd love to fix it -- for you *and* for everyone else!
diff --git a/doc/support.md b/doc/support.md
index 406ed81e65..4c585b1181 100644
--- a/doc/support.md
+++ b/doc/support.md
@@ -11,11 +11,11 @@ bugs, and some of our best features have come from user
requests. Please help us improve sourmash for everyone by asking
questions as you have them!
-Please ask questions and file bug descriptions [on the GitHub issue tracker for sourmash, dib-lab/sourmash/issues][0].
+Please ask questions and file bug descriptions [on the GitHub issue tracker for sourmash, sourmash-bio/sourmash/issues][0].
You can also ask questions of Titus on Twitter at [@ctitusbrown][1].
-[0]:/~https://github.com/dib-lab/sourmash/issues
+[0]:/~https://github.com/sourmash-bio/sourmash/issues
[1]:https://twitter.com/ctitusbrown/
## Versioning and stability of features and APIs
@@ -116,7 +116,7 @@ If you depend on sourmash, we recommend using the following process:
* pin sourmash to the major version you developed against, e.g. `sourmash >=3,<4`.
* when ready to upgrade sourmash, upgrade to the latest minor release within that major version (e.g. sourmash 3.5.x).
-* scan for deprecations that affect you, check [the release notes](/~https://github.com/dib-lab/sourmash/releases),
+* scan for deprecations that affect you, check [the release notes](/~https://github.com/sourmash-bio/sourmash/releases),
and fix any major issues noted.
* upgrade to the next major version (e.g. sourmash 4.0) and run your integration tests or workflow.
* fix outstanding issues.
@@ -130,7 +130,7 @@ If you want to upgrade workflows and scripts from prior releases of
sourmash to sourmash v4.0, we suggest doing this in two stages.
First, upgrade to the latest version of sourmash 3.5.x (currently
-[v3.5.1](/~https://github.com/dib-lab/sourmash/releases/tag/v3.5.1)),
+[v3.5.1](/~https://github.com/sourmash-bio/sourmash/releases/tag/v3.5.1)),
which is compatible with all files and command lines used in previous
versions of sourmash (v2.x and v3.x). After upgrading to 3.5.x, scan
the sourmash output for deprecation warnings and fix those.
@@ -178,12 +178,12 @@ Second, the `MinHash` class API has changed significantly!
Third, `SourmashSignature` objects no longer have a `name()` method but instead a `name` property, which can be assigned to. This property is now `None` when no name has been assigned. Note that `str(sig)` should now be used to retrieve a display name, and should replace all previous uses of `sig.name()`.
Fourth, a few top-level functions have been deprecated: `load_signatures(...)`, `load_one_signature(...)`, `create_sbt_index(...)`, and `load_sbt_index(...)`.
-* `load_signatures(...)`, `load_one_signature(...)` should be replaced with `load_file_as_signatures(...)`. Note there is currently no top-level way to load signatures from strings. For now, if you need that functionality, you can use `sourmash.signature.load_signatures(...)` and `sourmash.signature.load_one_signature(...)`, but please be aware that these are not considered part of the public API that is under semantic versioning, so they may change in the next minor point release; this is tracked in /~https://github.com/dib-lab/sourmash/issues/1312.
+* `load_signatures(...)`, `load_one_signature(...)` should be replaced with `load_file_as_signatures(...)`. Note there is currently no top-level way to load signatures from strings. For now, if you need that functionality, you can use `sourmash.signature.load_signatures(...)` and `sourmash.signature.load_one_signature(...)`, but please be aware that these are not considered part of the public API that is under semantic versioning, so they may change in the next minor point release; this is tracked in /~https://github.com/sourmash-bio/sourmash/issues/1312.
* `load_sbt_index(...)` have been deprecated. Please use `load_file_as_index(...)` instead.
* `create_sbt_index(...)` has been deprecated. There is currently no replacement, although you can use it directly from `sourmash.sbtmh` if necessary.
Fifth, directory traversal now happens by default when loading signatures, so remove `traverse=True` arguments to several functions in `sourmash_args` - `load_dbs_and_sigs`, `load_file_as_index`, `and load_file_as_signatures`.
Please post questions and concerns to the
-[sourmash issue tracker](/~https://github.com/dib-lab/sourmash/issues)
+[sourmash issue tracker](/~https://github.com/sourmash-bio/sourmash/issues)
and we'll be happy to help!
diff --git a/doc/tutorial-basic.md b/doc/tutorial-basic.md
index 41311f9f96..d6b6f270a5 100644
--- a/doc/tutorial-basic.md
+++ b/doc/tutorial-basic.md
@@ -121,7 +121,7 @@ Let's grab a sample collection of 50 E. coli genomes and unpack it --
mkdir ecoli_many_sigs
cd ecoli_many_sigs
-curl -O -L /~https://github.com/dib-lab/sourmash/raw/latest/data/eschericia-sigs.tar.gz
+curl -O -L /~https://github.com/sourmash-bio/sourmash/raw/latest/data/eschericia-sigs.tar.gz
tar xzf eschericia-sigs.tar.gz
rm eschericia-sigs.tar.gz
@@ -246,7 +246,7 @@ from the
[Shakya et al. 2013 mock metagenome paper.][2]
```
-wget /~https://github.com/dib-lab/sourmash/raw/latest/doc/_static/shakya-unaligned-contigs.sig
+wget /~https://github.com/sourmash-bio/sourmash/raw/latest/doc/_static/shakya-unaligned-contigs.sig
sourmash gather -k 31 shakya-unaligned-contigs.sig genbank-k31.lca.json.gz
```
diff --git a/doc/using-sourmash-a-guide.md b/doc/using-sourmash-a-guide.md
index 757d3d7654..2b62be021b 100644
--- a/doc/using-sourmash-a-guide.md
+++ b/doc/using-sourmash-a-guide.md
@@ -8,7 +8,7 @@ So! You've installed sourmash, run a few of the tutorials and commands,
and now you actually want to *use* it. This guide is here to answer some
of your questions, and explain why we can't answer others.
-(If you have additional questions, please [file an issue!](/~https://github.com/dib-lab/sourmash/issues))
+(If you have additional questions, please [file an issue!](/~https://github.com/sourmash-bio/sourmash/issues))
## What k-mer size(s) should I use?
diff --git a/include/sourmash.h b/include/sourmash.h
index 7f88fcc203..a7bb4a9b71 100644
--- a/include/sourmash.h
+++ b/include/sourmash.h
@@ -225,6 +225,8 @@ SourmashKmerMinHash *kmerminhash_new(uint64_t scaled,
uint32_t kmerminhash_num(const SourmashKmerMinHash *ptr);
+void kmerminhash_remove_from(SourmashKmerMinHash *ptr, const SourmashKmerMinHash *other);
+
void kmerminhash_remove_hash(SourmashKmerMinHash *ptr, uint64_t h);
void kmerminhash_remove_many(SourmashKmerMinHash *ptr,
diff --git a/nix/sources.json b/nix/sources.json
index 7f9a68a779..fece90acd7 100644
--- a/nix/sources.json
+++ b/nix/sources.json
@@ -5,10 +5,10 @@
"homepage": "/~https://github.com/nmattia/niv",
"owner": "nmattia",
"repo": "niv",
- "rev": "af958e8057f345ee1aca714c1247ef3ba1c15f5e",
- "sha256": "1qjavxabbrsh73yck5dcq8jggvh3r2jkbr6b5nlz5d9yrqm9255n",
+ "rev": "1819632b5823e0527da28ad82fecd6be5136c1e9",
+ "sha256": "08jz17756qchq0zrqmapcm33nr4ms9f630mycc06i6zkfwl5yh5i",
"type": "tarball",
- "url": "/~https://github.com/nmattia/niv/archive/af958e8057f345ee1aca714c1247ef3ba1c15f5e.tar.gz",
+ "url": "/~https://github.com/nmattia/niv/archive/1819632b5823e0527da28ad82fecd6be5136c1e9.tar.gz",
"url_template": "/~https://github.com///archive/.tar.gz"
},
"nixpkgs": {
@@ -17,10 +17,10 @@
"homepage": "",
"owner": "NixOS",
"repo": "nixpkgs",
- "rev": "e62feb3bf4a603e26755238303bda0c24651e155",
- "sha256": "1gkamm044jrksjrisr7h9grg8p2y6rk01x6391asrx988hm2rh9s",
+ "rev": "84aa23742f6c72501f9cc209f29c438766f5352d",
+ "sha256": "0h7xl6q0yjrbl9vm3h6lkxw692nm8bg3wy65gm95a2mivhrdjpxp",
"type": "tarball",
- "url": "/~https://github.com/NixOS/nixpkgs/archive/e62feb3bf4a603e26755238303bda0c24651e155.tar.gz",
+ "url": "/~https://github.com/NixOS/nixpkgs/archive/84aa23742f6c72501f9cc209f29c438766f5352d.tar.gz",
"url_template": "/~https://github.com///archive/.tar.gz"
},
"rust-overlay": {
@@ -29,10 +29,10 @@
"homepage": null,
"owner": "oxalica",
"repo": "rust-overlay",
- "rev": "d8efe70dc561c4bea0b7bf440d36ce98c497e054",
- "sha256": "0hdj5d635dq6zwj8d4ady1kyl9mwmsxvy6vqyd3xq0p2w18ffi4r",
+ "rev": "7368784b67f963508b67064ee758537b7c8e40c8",
+ "sha256": "1sr0lsk5m6c0dqp3429c0fj0picrvsaw1hjn6nd83pzxm54g7fcr",
"type": "tarball",
- "url": "/~https://github.com/oxalica/rust-overlay/archive/d8efe70dc561c4bea0b7bf440d36ce98c497e054.tar.gz",
+ "url": "/~https://github.com/oxalica/rust-overlay/archive/7368784b67f963508b67064ee758537b7c8e40c8.tar.gz",
"url_template": "/~https://github.com///archive/.tar.gz"
}
}
diff --git a/setup.cfg b/setup.cfg
index 6dd80fde71..37e5444e52 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -55,8 +55,8 @@ where = src
# this syntax may change in the future
[options.extras_require]
test =
- pytest>=6
- pytest-cov<2.6
+ pytest~=6.2.4
+ pytest-cov~=2.12
recommonmark
hypothesis
demo =
@@ -94,9 +94,6 @@ norecursedirs =
.tox
.asv
.eggs
-python_files =
- src/sourmash/*.py
- tests/*.py
testpaths =
tests
doc
diff --git a/src/core/CHANGELOG.md b/src/core/CHANGELOG.md
new file mode 100644
index 0000000000..05eff4f83b
--- /dev/null
+++ b/src/core/CHANGELOG.md
@@ -0,0 +1,160 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.11.0] - 2021-07-07
+
+Added:
+
+- Add HyperLogLog implementation (#1223)
+
+Changed:
+
+- Update `MinHash.set_abundances` to remove hash if 0 abund; handle negative abundances. (#1575)
+- Improving `MinHash.remove_many(...)` performance (#1571)
+- Improved intersection and union calculations (#1475)
+- Bump MSRV to 1.42 (and other dep fixes) (#1461)
+- Rework the `find` functionality for `Index` classes (#1392)
+- Rationalize `SourmashSignature.name` and `str(sig)` (#1179)
+
+Fixed:
+
+- Fix needless borrows as suggested by clippy (#1636)
+- Fix Rust 1.59 lints (#1600)
+- Clean up clippy lints from 1.52 (#1505)
+- Fix clippy lints introduced in 1.51 (#1407)
+- CI/Rust: update and fix cbindgen config (#1473)
+- pin needletail version to keep MSRV at 1.37 (#1393)
+- Update proptest requirement from 0.9.6 to 1.0.0 (#1344)
+- Fix clippy lints introduced in 1.50 and update nix configs (#1332)
+- Update finch requirement from 0.3.0 to 0.4.1 (#1290)
+- update rand for test, and activate "js" feature for getrandom (#1275)
+- Fix new clippy warnings from Rust 1.49 (#1267)
+- CI: small build fixes (#1252)
+
+Removed:
+
+- Remove 10x support in compute (#1229)
+
+## [0.10.0] - 2020-10-08
+
+Added:
+
+- Add `clear` option to set_abundances(...) method (#1046)
+
+Changed:
+
+- Replace mx by scaled (#1139)
+
+Fixed:
+
+- Fix Rust panic error in signature creation (#1172)
+- Update typed-builder requirement from 0.6.0 to 0.7.0 (#1121)
+- update CI for latest branch name change (#1150)
+- Update typed-builder requirement from 0.6.0 to 0.7.0 (#1121)
+
+## [0.9.0] - 2020-07-13
+
+Added:
+
+- Cache md5sum calculation (#1058)
+- Expose more of the API for wasm (signature and ComputeParameters) (#1058)
+- Getters and setters for ComputeParameters (#1058)
+
+Changed:
+
+- Migrate from failure to thiserror (#1058)
+- Bump MSRV to 1.37 (#1058)
+
+Fixed:
+
+- Use the derive feature in serde instead of serde_derive (#1058)
+- Use nohash-hasher crate instead of previous NoHashHasher from finch.
+- Update typed-builder to 0.6.0 (#1058)
+- stricter niffler versions and add new gz feature to it (#1070)
+
+## [0.8.0] - 2020-06-26
+
+Added:
+
+- compute-optimized MinHash (for small scaled or large cardinalities) (#1045)
+
+## [0.7.0] - 2020-05-12
+
+Changed:
+
+- Hide internal representation in core (#986)
+
+Fixed:
+
+- update FFI and cbindgen (#986)
+
+## [0.6.0] - 2020-04-28
+
+Added:
+
+- Nodegraph implementation based on khmer.Nodegraph (#799)
+
+## [0.5.0] - 2020-02-08
+
+Added:
+
+- add_hash_with_abundance method in core library (#892)
+
+Changed:
+
+- More refactoring of MinHash comparison code (#882)
+- Replace mins_push and abunds_push with set_abundances (#887)
+
+Fixed:
+
+- add_hash with num doesn't set abundances properly (#891)
+
+## [0.4.0] - 2020-01-26
+
+Added:
+
+- Compute improvements: Parameter sets for defining signatures, add_protein implemented (#845)
+- add_many for faster insertion of multiple hashes (#826)
+
+Changed:
+
+- Compare/similarity now have a downsample argument (#856)
+
+Fixed:
+
+- Improve sketching performance with lookup tables for complement and DNA validation (#861) (#865)
+- Use tarpaulin instead of grcov (#862)
+- set up publishing workflow for NPM and crates.io (#824)
+
+## [0.3.0] - 2020-01-05
+
+Added:
+
+- Similarity with abundance method for MinHash (#808)
+- Experimental support for indices in Rust (#773)
+- Experimental SBT with MQF internal nodes in Rust (#772)
+
+Changed:
+
+- Make the sourmash crate library-only (#812)
+
+Fixed:
+
+- Use once_cell instead of lazy_static and lazy-init (#815)
+- Fix mem leak in get_mins (#807)
+- Fixes for WASI and WASM compilation (#771) (#723)
+
+[unreleased]: /~https://github.com/sourmash-bio/sourmash/compare/r0.11.0...HEAD
+[0.11.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.10.0...r0.11.0
+[0.10.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.9.0...r0.10.0
+[0.9.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.9.0...r0.10.0
+[0.8.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.8.0...r0.9.0
+[0.7.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.7.0...r0.8.0
+[0.6.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.6.0...r0.7.0
+[0.5.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.5.0...r0.6.0
+[0.4.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.4.0...r0.5.0
+[0.3.0]: /~https://github.com/sourmash-bio/sourmash/compare/r0.3.0...r0.4.0
diff --git a/src/core/Cargo.toml b/src/core/Cargo.toml
index 3de8c37734..daceb338d4 100644
--- a/src/core/Cargo.toml
+++ b/src/core/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "sourmash"
-version = "0.10.0"
+version = "0.11.0"
authors = ["Luiz Irber "]
description = "MinHash sketches for genomic data"
repository = "/~https://github.com/dib-lab/sourmash"
@@ -44,19 +44,6 @@ primal-check = "0.3.1"
thiserror = "1.0"
typed-builder = "0.9.0"
-[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dependencies.wasm-bindgen]
-version = "0.2.62"
-features = ["serde-serialize"]
-
-[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dev-dependencies]
-wasm-bindgen-test = "0.3.0"
-
-[target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies.assert_cmd]
-version = "1.0.1"
-
-[package.metadata.wasm-pack.profile.release]
-wasm-opt = false # /~https://github.com/rustwasm/wasm-pack/issues/886
-
[dev-dependencies]
assert_matches = "1.3.0"
criterion = "0.3.2"
@@ -82,3 +69,19 @@ harness = false
[[bench]]
name = "minhash"
harness = false
+
+## Wasm section. Crates only used for WASM, as well as specific configurations
+
+[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dependencies.wasm-bindgen]
+version = "0.2.62"
+features = ["serde-serialize"]
+
+[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dependencies.web-sys]
+version = "0.3.51"
+features = ["console", "File"]
+
+[target.'cfg(all(target_arch = "wasm32", target_vendor="unknown"))'.dev-dependencies]
+wasm-bindgen-test = "0.3.0"
+
+[target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies.assert_cmd]
+version = "1.0.1"
diff --git a/src/core/src/cmd.rs b/src/core/src/cmd.rs
index aef8481405..25c4d8ca1d 100644
--- a/src/core/src/cmd.rs
+++ b/src/core/src/cmd.rs
@@ -1,6 +1,3 @@
-#[cfg(all(target_arch = "wasm32", target_vendor = "unknown"))]
-use wasm_bindgen::prelude::*;
-
use getset::{CopyGetters, Getters, Setters};
use typed_builder::TypedBuilder;
@@ -24,7 +21,7 @@ pub fn prepare(index_path: &str) -> Result<(), Error> {
impl Signature {
pub fn from_params(params: &ComputeParameters) -> Signature {
- let template = build_template(¶ms);
+ let template = build_template(params);
Signature::builder()
.hash_function("0.murmur64")
@@ -36,7 +33,6 @@ impl Signature {
}
#[allow(dead_code)]
-#[cfg_attr(all(target_arch = "wasm32", target_vendor = "unknown"), wasm_bindgen)]
#[derive(TypedBuilder, CopyGetters, Getters, Setters)]
pub struct ComputeParameters {
#[getset(get = "pub", set = "pub")]
diff --git a/src/core/src/encodings.rs b/src/core/src/encodings.rs
index 0ceddc2cbb..fee92e4bb6 100644
--- a/src/core/src/encodings.rs
+++ b/src/core/src/encodings.rs
@@ -4,12 +4,9 @@ use std::iter::Iterator;
use std::str;
use once_cell::sync::Lazy;
-#[cfg(all(target_arch = "wasm32", target_vendor = "unknown"))]
-use wasm_bindgen::prelude::*;
use crate::Error;
-#[cfg_attr(all(target_arch = "wasm32", target_vendor = "unknown"), wasm_bindgen)]
#[allow(non_camel_case_types)]
#[derive(Debug, Clone, Copy, PartialEq)]
#[repr(u32)]
diff --git a/src/core/src/ffi/minhash.rs b/src/core/src/ffi/minhash.rs
index 1618b9de62..7c3c6358c5 100644
--- a/src/core/src/ffi/minhash.rs
+++ b/src/core/src/ffi/minhash.rs
@@ -366,6 +366,15 @@ unsafe fn kmerminhash_add_from(ptr: *mut SourmashKmerMinHash, other: *const Sour
}
}
+ffi_fn! {
+ unsafe fn kmerminhash_remove_from(ptr: *mut SourmashKmerMinHash, other: *const SourmashKmerMinHash)
+ -> Result<()> {
+ let mh = SourmashKmerMinHash::as_rust_mut(ptr);
+ let other_mh = SourmashKmerMinHash::as_rust(other);
+ mh.remove_from(other_mh)
+ }
+}
+
ffi_fn! {
unsafe fn kmerminhash_count_common(ptr: *const SourmashKmerMinHash, other: *const SourmashKmerMinHash, downsample: bool)
-> Result {
diff --git a/src/core/src/ffi/utils.rs b/src/core/src/ffi/utils.rs
index 69baac7b88..04652cbeef 100644
--- a/src/core/src/ffi/utils.rs
+++ b/src/core/src/ffi/utils.rs
@@ -18,7 +18,7 @@ thread_local! {
pub static LAST_ERROR: RefCell> = RefCell::new(None);
}
-#[allow(clippy::clippy::wrong_self_convention)]
+#[allow(clippy::wrong_self_convention)]
pub trait ForeignObject: Sized {
type RustObject;
diff --git a/src/core/src/index/mod.rs b/src/core/src/index/mod.rs
index 507020fe3c..b2ad8c0fc7 100644
--- a/src/core/src/index/mod.rs
+++ b/src/core/src/index/mod.rs
@@ -114,11 +114,11 @@ where
N: Comparable,
{
fn similarity(&self, other: &L) -> f64 {
- (*self).similarity(&other)
+ (*self).similarity(other)
}
fn containment(&self, other: &L) -> f64 {
- (*self).containment(&other)
+ (*self).containment(other)
}
}
@@ -195,7 +195,7 @@ impl SigStore {
// TODO: better matching here, what if it is not a mh?
if let Sketch::MinHash(mh) = &ng.signatures[0] {
if let Sketch::MinHash(omh) = &ong.signatures[0] {
- return mh.count_common(&omh, false).unwrap() as u64;
+ return mh.count_common(omh, false).unwrap() as u64;
}
}
unimplemented!();
@@ -252,7 +252,7 @@ impl Comparable> for SigStore {
// TODO: better matching here, what if it is not a mh?
if let Sketch::MinHash(mh) = &ng.signatures[0] {
if let Sketch::MinHash(omh) = &ong.signatures[0] {
- return mh.similarity(&omh, true, false).unwrap();
+ return mh.similarity(omh, true, false).unwrap();
}
}
@@ -275,7 +275,7 @@ impl Comparable> for SigStore {
// TODO: better matching here, what if it is not a mh?
if let Sketch::MinHash(mh) = &ng.signatures[0] {
if let Sketch::MinHash(omh) = &ong.signatures[0] {
- let common = mh.count_common(&omh, false).unwrap();
+ let common = mh.count_common(omh, false).unwrap();
let size = mh.size();
return common as f64 / size as f64;
}
@@ -290,7 +290,7 @@ impl Comparable for Signature {
// TODO: better matching here, what if it is not a mh?
if let Sketch::MinHash(mh) = &self.signatures[0] {
if let Sketch::MinHash(omh) = &other.signatures[0] {
- return mh.similarity(&omh, true, false).unwrap();
+ return mh.similarity(omh, true, false).unwrap();
}
}
@@ -310,7 +310,7 @@ impl Comparable for Signature {
// TODO: better matching here, what if it is not a mh?
if let Sketch::MinHash(mh) = &self.signatures[0] {
if let Sketch::MinHash(omh) = &other.signatures[0] {
- let common = mh.count_common(&omh, false).unwrap();
+ let common = mh.count_common(omh, false).unwrap();
let size = mh.size();
return common as f64 / size as f64;
}
diff --git a/src/core/src/index/sbt/mhbt.rs b/src/core/src/index/sbt/mhbt.rs
index b3fcbe6987..974318cf29 100644
--- a/src/core/src/index/sbt/mhbt.rs
+++ b/src/core/src/index/sbt/mhbt.rs
@@ -76,13 +76,13 @@ impl Comparable> for Node {
fn similarity(&self, other: &Node) -> f64 {
let ng: &Nodegraph = self.data().unwrap();
let ong: &Nodegraph = other.data().unwrap();
- ng.similarity(&ong)
+ ng.similarity(ong)
}
fn containment(&self, other: &Node) -> f64 {
let ng: &Nodegraph = self.data().unwrap();
let ong: &Nodegraph = other.data().unwrap();
- ng.containment(&ong)
+ ng.containment(ong)
}
}
diff --git a/src/core/src/index/sbt/mod.rs b/src/core/src/index/sbt/mod.rs
index 5d3a25cd05..6638615317 100644
--- a/src/core/src/index/sbt/mod.rs
+++ b/src/core/src/index/sbt/mod.rs
@@ -822,7 +822,7 @@ impl BinaryTree {
let mut similar_node_pos = 0;
let mut current_max = 0;
for (pos, cmpe) in current_round.iter().enumerate() {
- let common = BinaryTree::intersection_size(&next_node, &cmpe);
+ let common = BinaryTree::intersection_size(&next_node, cmpe);
if common > current_max {
current_max = common;
similar_node_pos = pos;
diff --git a/src/core/src/signature.rs b/src/core/src/signature.rs
index f223b32bc0..8da0600b0b 100644
--- a/src/core/src/signature.rs
+++ b/src/core/src/signature.rs
@@ -14,9 +14,6 @@ use rayon::prelude::*;
use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
-#[cfg(all(target_arch = "wasm32", target_vendor = "unknown"))]
-use wasm_bindgen::prelude::*;
-
use crate::encodings::{aa_to_dayhoff, aa_to_hp, revcomp, to_aa, HashFunctions, VALID};
use crate::index::storage::ToWriter;
use crate::sketch::Sketch;
@@ -134,7 +131,7 @@ pub trait SigsTrait {
if hash_function.protein() {
for aa_kmer in seq.windows(ksize) {
- let hash = crate::_hash_murmur(&aa_kmer, self.seed());
+ let hash = crate::_hash_murmur(aa_kmer, self.seed());
self.add_hash(hash);
}
return Ok(());
@@ -151,7 +148,7 @@ pub trait SigsTrait {
};
for aa_kmer in aa_seq.windows(ksize) {
- let hash = crate::_hash_murmur(&aa_kmer, self.seed());
+ let hash = crate::_hash_murmur(aa_kmer, self.seed());
self.add_hash(hash);
}
@@ -242,7 +239,6 @@ impl SigsTrait for Sketch {
}
}
-#[cfg_attr(all(target_arch = "wasm32", target_vendor = "unknown"), wasm_bindgen)]
#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
pub struct Signature {
#[serde(default = "default_class")]
@@ -478,7 +474,7 @@ impl Signature {
self.signatures
.iter_mut()
.for_each(|sketch| {
- sketch.add_sequence(&seq, force).unwrap(); }
+ sketch.add_sequence(seq, force).unwrap(); }
);
}
}
@@ -498,7 +494,7 @@ impl Signature {
self.signatures
.iter_mut()
.try_for_each(|sketch| {
- sketch.add_protein(&seq) }
+ sketch.add_protein(seq) }
)?;
}
}
diff --git a/src/core/src/sketch/hyperloglog/estimators.rs b/src/core/src/sketch/hyperloglog/estimators.rs
index 4c2fbe02cc..60eadc5201 100644
--- a/src/core/src/sketch/hyperloglog/estimators.rs
+++ b/src/core/src/sketch/hyperloglog/estimators.rs
@@ -120,7 +120,7 @@ pub fn joint_mle(
let mut ceq = vec![0; q + 2];
for (k1_, k2_) in k1.iter().zip(k2.iter()) {
- match k1_.cmp(&k2_) {
+ match k1_.cmp(k2_) {
cmp::Ordering::Less => {
c1[*k1_ as usize] += 1;
cg2[*k2_ as usize] += 1;
diff --git a/src/core/src/sketch/hyperloglog/mod.rs b/src/core/src/sketch/hyperloglog/mod.rs
index 1ee5b89c25..a1bc8eb9d6 100644
--- a/src/core/src/sketch/hyperloglog/mod.rs
+++ b/src/core/src/sketch/hyperloglog/mod.rs
@@ -120,7 +120,7 @@ impl HyperLogLog {
wtr.write_u8(self.p as u8)?; // number of bits used for indexing
wtr.write_u8(self.q as u8)?; // number of bits used for counting leading zeroes
wtr.write_u8(self.ksize as u8)?; // ksize
- wtr.write_all(&self.registers.as_slice())?;
+ wtr.write_all(self.registers.as_slice())?;
Ok(())
}
diff --git a/src/core/src/sketch/minhash.rs b/src/core/src/sketch/minhash.rs
index a9d8a9d778..472314d1a6 100644
--- a/src/core/src/sketch/minhash.rs
+++ b/src/core/src/sketch/minhash.rs
@@ -17,9 +17,6 @@ use crate::signature::SigsTrait;
use crate::sketch::hyperloglog::HyperLogLog;
use crate::Error;
-#[cfg(all(target_arch = "wasm32", target_vendor = "unknown"))]
-use wasm_bindgen::prelude::*;
-
pub fn max_hash_for_scaled(scaled: u64) -> u64 {
match scaled {
0 => 0,
@@ -35,7 +32,6 @@ pub fn scaled_for_max_hash(max_hash: u64) -> u64 {
}
}
-#[cfg_attr(all(target_arch = "wasm32", target_vendor = "unknown"), wasm_bindgen)]
#[derive(Debug, TypedBuilder)]
pub struct KmerMinHash {
num: u32,
@@ -327,7 +323,7 @@ impl KmerMinHash {
}
if abundance == 0 {
- // well, don't add it.
+ self.remove_hash(hash);
return;
}
@@ -415,6 +411,13 @@ impl KmerMinHash {
};
}
+ pub fn remove_from(&mut self, other: &KmerMinHash) -> Result<(), Error> {
+ for min in &other.mins {
+ self.remove_hash(*min);
+ }
+ Ok(())
+ }
+
pub fn remove_many(&mut self, hashes: &[u64]) -> Result<(), Error> {
for min in hashes {
self.remove_hash(*min);
@@ -590,8 +593,8 @@ impl KmerMinHash {
self.num,
);
- combined_mh.merge(&self)?;
- combined_mh.merge(&other)?;
+ combined_mh.merge(self)?;
+ combined_mh.merge(other)?;
let it1 = Intersection::new(self.mins.iter(), other.mins.iter());
@@ -623,8 +626,8 @@ impl KmerMinHash {
self.num,
);
- combined_mh.merge(&self)?;
- combined_mh.merge(&other)?;
+ combined_mh.merge(self)?;
+ combined_mh.merge(other)?;
let it1 = Intersection::new(self.mins.iter(), other.mins.iter());
@@ -714,9 +717,9 @@ impl KmerMinHash {
let downsampled_mh = second.downsample_max_hash(first.max_hash)?;
first.similarity(&downsampled_mh, ignore_abundance, false)
} else if ignore_abundance || self.abunds.is_none() || other.abunds.is_none() {
- self.jaccard(&other)
+ self.jaccard(other)
} else {
- self.angular_similarity(&other)
+ self.angular_similarity(other)
}
}
@@ -932,7 +935,6 @@ mod test {
//#############
// A MinHash implementation for low scaled or large cardinalities
-#[cfg_attr(all(target_arch = "wasm32", target_vendor = "unknown"), wasm_bindgen)]
#[derive(Debug, TypedBuilder)]
pub struct KmerMinHashBTree {
num: u32,
@@ -1304,7 +1306,7 @@ impl KmerMinHashBTree {
for hash in &self.mins {
*new_abunds.entry(*hash).or_insert(0) +=
- abunds.get(&hash).unwrap_or(&0) + oabunds.get(&hash).unwrap_or(&0);
+ abunds.get(hash).unwrap_or(&0) + oabunds.get(hash).unwrap_or(&0);
}
self.abunds = Some(new_abunds)
}
@@ -1371,8 +1373,8 @@ impl KmerMinHashBTree {
self.num,
);
- combined_mh.merge(&self)?;
- combined_mh.merge(&other)?;
+ combined_mh.merge(self)?;
+ combined_mh.merge(other)?;
let it1 = Intersection::new(self.mins.iter(), other.mins.iter());
@@ -1403,8 +1405,8 @@ impl KmerMinHashBTree {
self.num,
);
- combined_mh.merge(&self)?;
- combined_mh.merge(&other)?;
+ combined_mh.merge(self)?;
+ combined_mh.merge(other)?;
let it1 = Intersection::new(self.mins.iter(), other.mins.iter());
@@ -1448,7 +1450,7 @@ impl KmerMinHashBTree {
let b_sq: u64 = other_abunds.values().map(|a| (a * a)).sum();
for (hash, value) in abunds.iter() {
- if let Some(oa) = other_abunds.get(&hash) {
+ if let Some(oa) = other_abunds.get(hash) {
prod += value * oa
}
}
@@ -1479,9 +1481,9 @@ impl KmerMinHashBTree {
let downsampled_mh = second.downsample_max_hash(first.max_hash)?;
first.similarity(&downsampled_mh, ignore_abundance, false)
} else if ignore_abundance || self.abunds.is_none() || other.abunds.is_none() {
- self.jaccard(&other)
+ self.jaccard(other)
} else {
- self.angular_similarity(&other)
+ self.angular_similarity(other)
}
}
diff --git a/src/core/src/sketch/nodegraph.rs b/src/core/src/sketch/nodegraph.rs
index 37c89fb4cf..56ee7d5940 100644
--- a/src/core/src/sketch/nodegraph.rs
+++ b/src/core/src/sketch/nodegraph.rs
@@ -202,7 +202,7 @@ impl Nodegraph {
let len = size_of::() * slice.len();
slice::from_raw_parts(slice.as_ptr() as *const u8, len)
};
- wtr.write_all(&buf)?;
+ wtr.write_all(buf)?;
// Replace when byteorder PR is released
if rem != 0 {
@@ -241,22 +241,19 @@ impl Nodegraph {
let byte_size = tablesize / 8 + 1;
let rem = byte_size % 4;
- let blocks: Vec = if rem == 0 {
+ let blocks: Vec = {
let mut blocks = vec![0; byte_size / 4];
rdr.read_u32_into::(&mut blocks)?;
- blocks
- } else {
- let mut blocks = vec![0; byte_size / 4];
- rdr.read_u32_into::(&mut blocks)?;
-
- let mut values = [0u8; 4];
- for item in values.iter_mut().take(rem) {
- let byte = rdr.read_u8().expect("error reading bins");
- *item = byte;
+ if rem != 0 {
+ let mut values = [0u8; 4];
+ for item in values.iter_mut().take(rem) {
+ let byte = rdr.read_u8().expect("error reading bins");
+ *item = byte;
+ }
+ let mut block = vec![0u32; 1];
+ LittleEndian::read_u32_into(&values, &mut block);
+ blocks.push(block[0]);
}
- let mut block = vec![0u32; 1];
- LittleEndian::read_u32_into(&values, &mut block);
- blocks.push(block[0]);
blocks
};
diff --git a/src/core/src/wasm.rs b/src/core/src/wasm.rs
index 3ada5e7f86..ad656d9955 100644
--- a/src/core/src/wasm.rs
+++ b/src/core/src/wasm.rs
@@ -1,11 +1,25 @@
-use wasm_bindgen::prelude::*;
+// When the `wee_alloc` feature is enabled, use `wee_alloc` as the global
+// allocator.
+#[cfg(feature = "wee_alloc")]
+#[global_allocator]
+static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;
-use serde_json;
+use wasm_bindgen::prelude::*;
-use crate::cmd::ComputeParameters;
+use crate::cmd::ComputeParameters as _ComputeParameters;
use crate::encodings::HashFunctions;
-use crate::signature::{Signature, SigsTrait};
-use crate::sketch::minhash::KmerMinHash;
+use crate::signature::Signature as _Signature;
+use crate::signature::SigsTrait;
+use crate::sketch::minhash::KmerMinHash as _KmerMinHash;
+
+#[wasm_bindgen]
+pub struct KmerMinHash(_KmerMinHash);
+
+#[wasm_bindgen]
+pub struct Signature(_Signature);
+
+#[wasm_bindgen]
+pub struct ComputeParameters(_ComputeParameters);
#[wasm_bindgen]
impl KmerMinHash {
@@ -32,25 +46,26 @@ impl KmerMinHash {
HashFunctions::murmur64_DNA
};
- KmerMinHash::new(
+ KmerMinHash(_KmerMinHash::new(
scaled as u64,
ksize,
hash_function,
seed as u64,
track_abundance,
num,
- )
+ ))
}
#[wasm_bindgen]
pub fn add_sequence_js(&mut self, buf: &str) {
- self.add_sequence(buf.as_bytes(), true)
+ self.0
+ .add_sequence(buf.as_bytes(), true)
.expect("Error adding sequence");
}
#[wasm_bindgen]
pub fn to_json(&mut self) -> String {
- serde_json::to_string(self).unwrap()
+ serde_json::to_string(&self.0).unwrap()
}
}
@@ -58,8 +73,13 @@ impl KmerMinHash {
impl ComputeParameters {
#[wasm_bindgen(constructor)]
pub fn new_with_params() -> ComputeParameters {
- let params = ComputeParameters::default();
- params
+ let params = _ComputeParameters::default();
+ ComputeParameters(params)
+ }
+
+ #[wasm_bindgen]
+ pub fn set_ksizes(&mut self, ksizes: Vec) {
+ self.0.set_ksizes(ksizes);
}
}
@@ -69,18 +89,28 @@ impl Signature {
pub fn new_from_params(params: &ComputeParameters) -> Signature {
//let params = ComputeParameters::default();
- Signature::from_params(¶ms)
+ Signature(_Signature::from_params(¶ms.0))
}
#[wasm_bindgen]
pub fn add_sequence_js(&mut self, buf: &str) {
- self.add_sequence(buf.as_bytes(), true)
+ self.0
+ .add_sequence(buf.as_bytes(), true)
.expect("Error adding sequence");
}
+ #[wasm_bindgen]
+ pub fn add_from_file(&mut self, fp: web_sys::File) {
+ unimplemented!()
+ }
+
#[wasm_bindgen]
pub fn to_json(&mut self) -> String {
- serde_json::to_string(self).unwrap()
+ serde_json::to_string(&self.0).unwrap()
+ }
+
+ pub fn size(&self) -> usize {
+ self.0.size()
}
}
diff --git a/src/sourmash/__init__.py b/src/sourmash/__init__.py
index 463f718a7a..8735c06cea 100644
--- a/src/sourmash/__init__.py
+++ b/src/sourmash/__init__.py
@@ -111,6 +111,7 @@ def search_sbt_index(*args, **kwargs):
from .sbtmh import create_sbt_index
from . import lca
+from . import tax
from . import sbt
from . import sbtmh
from . import sbt_storage
diff --git a/src/sourmash/cli/__init__.py b/src/sourmash/cli/__init__.py
index c38c3e7afc..1b20c90def 100644
--- a/src/sourmash/cli/__init__.py
+++ b/src/sourmash/cli/__init__.py
@@ -37,6 +37,7 @@
from . import sig as signature
from . import sketch
from . import storage
+from . import tax
class SourmashParser(ArgumentParser):
@@ -92,6 +93,7 @@ def parse_args(self, args=None, namespace=None):
def get_parser():
module_descs = {
+ 'tax': 'Integrate taxonomy information based on "gather" results',
'lca': 'Taxonomic operations',
'sketch': 'Create signatures',
'sig': 'Manipulate signature files',
diff --git a/src/sourmash/cli/compare.py b/src/sourmash/cli/compare.py
index dcec015bd5..f7387f68b2 100644
--- a/src/sourmash/cli/compare.py
+++ b/src/sourmash/cli/compare.py
@@ -1,6 +1,7 @@
"""compare sequence signatures made by compute"""
-from sourmash.cli.utils import add_ksize_arg, add_moltype_args
+from sourmash.cli.utils import (add_ksize_arg, add_moltype_args,
+ add_picklist_args)
def subparser(subparsers):
@@ -47,6 +48,7 @@ def subparser(subparsers):
subparser.add_argument(
'-p', '--processes', metavar='N', type=int, default=None,
help='Number of processes to use to calculate similarity')
+ add_picklist_args(subparser)
def main(args):
diff --git a/src/sourmash/cli/gather.py b/src/sourmash/cli/gather.py
index 3d2e6d1a24..6e0addd427 100644
--- a/src/sourmash/cli/gather.py
+++ b/src/sourmash/cli/gather.py
@@ -1,6 +1,7 @@
"""search a metagenome signature against dbs"""
-from sourmash.cli.utils import add_ksize_arg, add_moltype_args
+from sourmash.cli.utils import (add_ksize_arg, add_moltype_args,
+ add_picklist_args)
def subparser(subparsers):
@@ -60,8 +61,6 @@ def subparser(subparsers):
'--cache-size', default=0, type=int, metavar='N',
help='number of internal SBT nodes to cache in memory (default: 0, cache all nodes)'
)
- add_ksize_arg(subparser, 31)
- add_moltype_args(subparser)
# advanced parameters
subparser.add_argument(
@@ -80,6 +79,10 @@ def subparser(subparsers):
help="use prefetch before gather; see documentation",
)
+ add_ksize_arg(subparser, 31)
+ add_moltype_args(subparser)
+ add_picklist_args(subparser)
+
def main(args):
import sourmash
diff --git a/src/sourmash/cli/index.py b/src/sourmash/cli/index.py
index 1be7f06690..334b394bfe 100644
--- a/src/sourmash/cli/index.py
+++ b/src/sourmash/cli/index.py
@@ -25,7 +25,8 @@
---
"""
-from sourmash.cli.utils import add_moltype_args, add_ksize_arg
+from sourmash.cli.utils import (add_ksize_arg, add_moltype_args,
+ add_picklist_args)
def subparser(subparsers):
@@ -44,7 +45,6 @@ def subparser(subparsers):
'-q', '--quiet', action='store_true',
help='suppress non-error output'
)
- add_ksize_arg(subparser, 31)
subparser.add_argument(
'-d', '--n_children', metavar='D', type=int, default=2,
help='number of children for internal nodes; default=2'
@@ -70,7 +70,9 @@ def subparser(subparsers):
'--scaled', metavar='FLOAT', type=float, default=0,
help='downsample signatures to the specified scaled factor'
)
+ add_ksize_arg(subparser, 31)
add_moltype_args(subparser)
+ add_picklist_args(subparser)
def main(args):
diff --git a/src/sourmash/cli/lca/index.py b/src/sourmash/cli/lca/index.py
index 581ff63dcd..fd205b6f9e 100644
--- a/src/sourmash/cli/lca/index.py
+++ b/src/sourmash/cli/lca/index.py
@@ -1,6 +1,7 @@
"""create LCA database"""
-from sourmash.cli.utils import add_ksize_arg, add_moltype_args
+from sourmash.cli.utils import (add_ksize_arg, add_moltype_args,
+ add_picklist_args)
def subparser(subparsers):
@@ -18,8 +19,6 @@ def subparser(subparsers):
subparser.add_argument(
'--scaled', metavar='S', default=10000, type=float
)
- add_ksize_arg(subparser, 31)
- add_moltype_args(subparser)
subparser.add_argument(
'-q', '--quiet', action='store_true',
help='suppress non-error output'
@@ -42,7 +41,11 @@ def subparser(subparsers):
)
subparser.add_argument(
'--split-identifiers', action='store_true',
- help='split names in signatures on whitspace and period'
+ help='split names in signatures on whitespace'
+ )
+ subparser.add_argument(
+ '--keep-identifier-versions', action='store_true',
+ help='do not remove accession versions'
)
subparser.add_argument('-f', '--force', action='store_true')
subparser.add_argument(
@@ -52,6 +55,14 @@ def subparser(subparsers):
'--require-taxonomy', action='store_true',
help='ignore signatures with no taxonomy entry'
)
+ subparser.add_argument(
+ '--fail-on-missing-taxonomy', action='store_true',
+ help='fail quickly if taxonomy is not available for an identifier',
+ )
+
+ add_ksize_arg(subparser, 31)
+ add_moltype_args(subparser)
+ add_picklist_args(subparser)
def main(args):
diff --git a/src/sourmash/cli/prefetch.py b/src/sourmash/cli/prefetch.py
index 27a254c68e..e04c537193 100644
--- a/src/sourmash/cli/prefetch.py
+++ b/src/sourmash/cli/prefetch.py
@@ -1,6 +1,7 @@
"""search a signature against dbs, find all overlaps"""
-from sourmash.cli.utils import add_ksize_arg, add_moltype_args
+from sourmash.cli.utils import (add_ksize_arg, add_moltype_args,
+ add_picklist_args)
def subparser(subparsers):
@@ -63,6 +64,7 @@ def subparser(subparsers):
)
add_ksize_arg(subparser, 31)
add_moltype_args(subparser)
+ add_picklist_args(subparser)
def main(args):
diff --git a/src/sourmash/cli/search.py b/src/sourmash/cli/search.py
index 9ff4ab9985..c4e1d41323 100644
--- a/src/sourmash/cli/search.py
+++ b/src/sourmash/cli/search.py
@@ -1,6 +1,7 @@
"""search a signature against other signatures"""
-from sourmash.cli.utils import add_ksize_arg, add_moltype_args
+from sourmash.cli.utils import (add_ksize_arg, add_moltype_args,
+ add_picklist_args)
def subparser(subparsers):
@@ -59,6 +60,7 @@ def subparser(subparsers):
)
add_ksize_arg(subparser, 31)
add_moltype_args(subparser)
+ add_picklist_args(subparser)
def main(args):
diff --git a/src/sourmash/cli/sig/__init__.py b/src/sourmash/cli/sig/__init__.py
index 36a224ef86..d9d71463d9 100644
--- a/src/sourmash/cli/sig/__init__.py
+++ b/src/sourmash/cli/sig/__init__.py
@@ -12,6 +12,7 @@
from . import filter
from . import flatten
from . import intersect
+from . import manifest
from . import merge
from . import rename
from . import subtract
diff --git a/src/sourmash/cli/sig/cat.py b/src/sourmash/cli/sig/cat.py
index 99d53090d7..6da03f886e 100644
--- a/src/sourmash/cli/sig/cat.py
+++ b/src/sourmash/cli/sig/cat.py
@@ -6,7 +6,11 @@
def subparser(subparsers):
subparser = subparsers.add_parser('cat')
- subparser.add_argument('signatures', nargs='+')
+ subparser.add_argument('signatures', nargs='*')
+ subparser.add_argument(
+ '--from-file',
+ help='a text file containing a list of files to load signatures from'
+ )
subparser.add_argument(
'-q', '--quiet', action='store_true',
help='suppress non-error output'
@@ -19,6 +23,10 @@ def subparser(subparsers):
'-u', '--unique', action='store_true',
help='keep only distinct signatures, removing duplicates (based on md5sum)'
)
+ subparser.add_argument(
+ '-f', '--force', action='store_true',
+ help='try to load all files as signatures'
+ )
def main(args):
diff --git a/src/sourmash/cli/sig/extract.py b/src/sourmash/cli/sig/extract.py
index d2066e8bcc..9ea71eb229 100644
--- a/src/sourmash/cli/sig/extract.py
+++ b/src/sourmash/cli/sig/extract.py
@@ -2,7 +2,8 @@
import sys
-from sourmash.cli.utils import add_moltype_args, add_ksize_arg
+from sourmash.cli.utils import (add_moltype_args, add_ksize_arg,
+ add_picklist_args)
def subparser(subparsers):
@@ -27,6 +28,7 @@ def subparser(subparsers):
)
add_ksize_arg(subparser, 31)
add_moltype_args(subparser)
+ add_picklist_args(subparser)
def main(args):
diff --git a/src/sourmash/cli/sig/manifest.py b/src/sourmash/cli/sig/manifest.py
new file mode 100644
index 0000000000..805c9d1a56
--- /dev/null
+++ b/src/sourmash/cli/sig/manifest.py
@@ -0,0 +1,27 @@
+"""create a manifest for a collection of signatures"""
+
+import sourmash
+from sourmash.logging import notify, print_results, error
+
+
+def subparser(subparsers):
+ subparser = subparsers.add_parser('manifest')
+ subparser.add_argument('location')
+ subparser.add_argument(
+ '-q', '--quiet', action='store_true',
+ help='suppress non-error output'
+ )
+ subparser.add_argument(
+ '-o', '--output', '--csv', metavar='FILE',
+ help='output information to a CSV file',
+ required=True,
+ )
+ subparser.add_argument(
+ '-f', '--force', action='store_true',
+ help='try to load all files as signatures'
+ )
+
+
+def main(args):
+ import sourmash
+ return sourmash.sig.__main__.manifest(args)
diff --git a/src/sourmash/cli/tax/__init__.py b/src/sourmash/cli/tax/__init__.py
new file mode 100644
index 0000000000..2a9b5b0302
--- /dev/null
+++ b/src/sourmash/cli/tax/__init__.py
@@ -0,0 +1,33 @@
+"""Define the command line interface for sourmash tax
+
+The top level CLI is defined in ../__init__.py. This module defines the CLI for
+`sourmash tax` operations.
+"""
+
+from . import metagenome
+from . import genome
+from . import annotate
+from . import prepare
+from ..utils import command_list
+from argparse import SUPPRESS, RawDescriptionHelpFormatter
+import os
+import sys
+
+
+def subparser(subparsers):
+ subparser = subparsers.add_parser('tax', formatter_class=RawDescriptionHelpFormatter, usage=SUPPRESS, aliases=['taxonomy'])
+ desc = 'Operations\n'
+ clidir = os.path.dirname(__file__)
+ ops = command_list(clidir)
+ for subcmd in ops:
+ docstring = getattr(sys.modules[__name__], subcmd).__doc__
+ helpstring = 'sourmash tax {op:s} --help'.format(op=subcmd)
+ desc += ' {hs:33s} {ds:s}\n'.format(hs=helpstring, ds=docstring)
+ s = subparser.add_subparsers(
+ title="Integrate taxonomy information based on 'gather' results", dest='subcmd', metavar='subcmd', help=SUPPRESS,
+ description=desc
+ )
+ for subcmd in ops:
+ getattr(sys.modules[__name__], subcmd).subparser(s)
+ subparser._action_groups.reverse()
+ subparser._optionals.title = 'Options'
diff --git a/src/sourmash/cli/tax/annotate.py b/src/sourmash/cli/tax/annotate.py
new file mode 100644
index 0000000000..f25a532554
--- /dev/null
+++ b/src/sourmash/cli/tax/annotate.py
@@ -0,0 +1,67 @@
+"""annotate gather results with taxonomy information"""
+
+usage="""
+
+ sourmash tax annotate --gather-csv [ ... ] --taxonomy-csv [ ... ]
+
+The 'tax annotate' command reads in gather results CSVs and annotates them
+ with taxonomic information.
+
+By default, `tax annotate` produces a gather CSV with an additional `lineage`
+ column containing the taxonomic information for each database match.
+
+Please see the 'tax annotate' documentation for more details:
+ https://sourmash.readthedocs.io/en/latest/command-line.html#sourmash-tax-annotate-annotates-gather-output-with-taxonomy
+"""
+
+import sourmash
+from sourmash.logging import notify, print_results, error
+
+
+def subparser(subparsers):
+ subparser = subparsers.add_parser('annotate',
+ aliases=['annotate'],
+ usage=usage)
+ subparser.add_argument(
+ '-g', '--gather-csv', nargs='*', default = [],
+ help='CSV output files from sourmash gather'
+ )
+ subparser.add_argument(
+ '--from-file', metavar='FILE', default=None,
+ help='input many gather results as a text file, with one gather CSV per line'
+ )
+ subparser.add_argument(
+ '-q', '--quiet', action='store_true',
+ help='suppress non-error output'
+ )
+ subparser.add_argument(
+ '-t', '--taxonomy-csv', '--taxonomy', metavar='FILE',
+ nargs="+", required=True,
+ help='database lineages CSV'
+ )
+ subparser.add_argument(
+ '-o', '--output-dir', default= "",
+ help='directory for output files'
+ )
+ subparser.add_argument(
+ '--keep-full-identifiers', action='store_true',
+ help='do not split identifiers on whitespace'
+ )
+ subparser.add_argument(
+ '--keep-identifier-versions', action='store_true',
+ help='after splitting identifiers, do not remove accession versions'
+ )
+ subparser.add_argument(
+ '--fail-on-missing-taxonomy', action='store_true',
+ help='fail quickly if taxonomy is not available for an identifier',
+ )
+ subparser.add_argument(
+ '-f', '--force', action = 'store_true',
+ help='continue past errors in file and taxonomy loading',
+ )
+
+def main(args):
+ import sourmash
+ if not args.gather_csv and not args.from_file:
+ raise ValueError(f"No gather CSVs found! Please input via `-g` or `--from-file`.")
+ return sourmash.tax.__main__.annotate(args)
diff --git a/src/sourmash/cli/tax/genome.py b/src/sourmash/cli/tax/genome.py
new file mode 100644
index 0000000000..f37fb35c8a
--- /dev/null
+++ b/src/sourmash/cli/tax/genome.py
@@ -0,0 +1,102 @@
+"""classify genomes from gather results"""
+
+usage="""
+
+ sourmash tax genome --gather-csv [ ... ] --taxonomy-csv [ ... ]
+
+The 'tax genome' command reads in genome gather result CSVs and reports likely
+classification for each query genome.
+
+By default, classification uses a containment threshold of 0.1, meaning at least
+10 percent of the query was covered by matches with the reported taxonomic rank and lineage.
+You can specify an alternate classification threshold or force classification by
+taxonomic rank instead, e.g. at species or genus-level.
+
+The default output format consists of five columns,
+ `query_name,status,rank,fraction,lineage`, where `fraction` is the fraction
+ of the query matched to the reported rank and lineage. The `status` column
+ provides additional information on the classification, and can be:
+ - `match` - this query was classified
+ - `nomatch`- this query could not be classified
+ - `below_threshold` - this query was classified at the specified rank,
+ but the query fraction matched was below the containment threshold
+
+Optionally, you can report classifications in `krona` format, but note
+that this forces classification by rank, rather than containment threshold.
+
+Please see the 'tax genome' documentation for more details:
+ https://sourmash.readthedocs.io/en/latest/command-line.html#sourmash-tax-genome-classify-a-genome-using-gather-results
+"""
+
+import argparse
+import sourmash
+from sourmash.logging import notify, print_results, error
+from sourmash.cli.utils import add_tax_threshold_arg
+
+def subparser(subparsers):
+ subparser = subparsers.add_parser('genome',
+ aliases=['classify'],
+ usage=usage)
+ subparser.add_argument(
+ '-g', '--gather-csv', nargs='*', default = [],
+ help='CSVs output by sourmash gather for this sample'
+ )
+ subparser.add_argument(
+ '--from-file', metavar='FILE', default=None,
+ help='input many gather results as a text file, with one gather CSV per line'
+ )
+ subparser.add_argument(
+ '-q', '--quiet', action='store_true',
+ help='suppress non-error output'
+ )
+ subparser.add_argument(
+ '-t', '--taxonomy-csv', '--taxonomy', metavar='FILE',
+ nargs='+', required=True,
+ help='database lineages CSV'
+ )
+ subparser.add_argument(
+ '-o', '--output-base', default='-',
+ help='base filepath for output file(s) (default stdout)'
+ )
+ subparser.add_argument(
+ '--output-dir', default= "",
+ help='directory for output files'
+ )
+ subparser.add_argument(
+ '-r', '--rank', choices=['strain', 'species', 'genus', 'family', 'order', 'class', 'phylum', 'superkingdom'],
+ help='Summarize genome taxonomy at this rank and above. Note that the taxonomy CSV must contain lineage information at this rank.'
+ )
+ subparser.add_argument(
+ '--keep-full-identifiers', action='store_true',
+ help='do not split identifiers on whitespace'
+ )
+ subparser.add_argument(
+ '--keep-identifier-versions', action='store_true',
+ help='after splitting identifiers, do not remove accession versions'
+ )
+ subparser.add_argument(
+ '--fail-on-missing-taxonomy', action='store_true',
+ help='fail quickly if taxonomy is not available for an identifier',
+ )
+ subparser.add_argument(
+ '--output-format', default=['csv_summary'], nargs='+', choices=["csv_summary", "krona"],
+ help='choose output format(s)',
+ )
+ subparser.add_argument(
+ '-f', '--force', action = 'store_true',
+ help='continue past survivable errors in loading taxonomy database or gather results',
+ )
+ add_tax_threshold_arg(subparser, 0.1)
+
+
+def main(args):
+ import sourmash
+ if not args.gather_csv and not args.from_file:
+ raise ValueError(f"No gather CSVs found! Please input via `-g` or `--from-file`.")
+ if len(args.output_format) > 1:
+ if args.output_base == "-":
+ raise TypeError(f"Writing to stdout is incompatible with multiple output formats {args.output_format}")
+ if not args.rank:
+ if any(x in ["krona"] for x in args.output_format):
+ raise ValueError(f"Rank (--rank) is required for krona output format.")
+ return sourmash.tax.__main__.genome(args)
diff --git a/src/sourmash/cli/tax/metagenome.py b/src/sourmash/cli/tax/metagenome.py
new file mode 100644
index 0000000000..4eb6b352db
--- /dev/null
+++ b/src/sourmash/cli/tax/metagenome.py
@@ -0,0 +1,90 @@
+"""summarize metagenome gather results"""
+
+usage="""
+
+ sourmash tax metagenome --gather-csv [ ... ] --taxonomy-csv [ ... ]
+
+The 'tax metagenome' command reads in metagenome gather result CSVs and
+summarizes by taxonomic lineage.
+
+The default output format consists of four columns,
+ `query_name,rank,fraction,lineage`, where `fraction` is the fraction
+ of the query matched to that reported rank and lineage. The summarization
+ is reported for each taxonomic rank.
+
+Alternatively, you can output results at a specific rank (e.g. species)
+in `krona` or `lineage_summary` formats.
+
+Please see the 'tax metagenome' documentation for more details:
+ https://sourmash.readthedocs.io/en/latest/command-line.html#sourmash-tax-metagenome-summarize-metagenome-content-from-gather-results
+"""
+
+import sourmash
+from sourmash.logging import notify, print_results, error
+
+
+def subparser(subparsers):
+ subparser = subparsers.add_parser('metagenome',
+ aliases=['summarize'],
+ usage=usage)
+ subparser.add_argument(
+ '-g', '--gather-csv', nargs='*', default = [],
+ help='CSVs from sourmash gather'
+ )
+ subparser.add_argument(
+ '--from-file', metavar='FILE', default = None,
+ help='input many gather results as a text file, with one gather CSV per line'
+ )
+ subparser.add_argument(
+ '-q', '--quiet', action='store_true',
+ help='suppress non-error output'
+ )
+ subparser.add_argument(
+ '-o', '--output-base', default='-',
+ help='base filepath for output file(s) (default stdout)'
+ )
+ subparser.add_argument(
+ '--output-dir', default= "",
+ help='directory for output files'
+ )
+ subparser.add_argument(
+ '-t', '--taxonomy-csv', '--taxonomy', metavar='FILE',
+ nargs='+', required=True,
+ help='database lineages CSV'
+ )
+ subparser.add_argument(
+ '--keep-full-identifiers', action='store_true',
+ help='do not split identifiers on whitespace'
+ )
+ subparser.add_argument(
+ '--keep-identifier-versions', action='store_true',
+ help='after splitting identifiers, do not remove accession versions'
+ )
+ subparser.add_argument(
+ '--fail-on-missing-taxonomy', action='store_true',
+ help='fail quickly if taxonomy is not available for an identifier',
+ )
+ subparser.add_argument(
+ '--output-format', default=['csv_summary'], nargs='+', choices=["csv_summary", "krona", "lineage_summary"],
+ help='choose output format(s)',
+ )
+ subparser.add_argument(
+ '-r', '--rank', choices=['strain','species', 'genus', 'family', 'order', 'class', 'phylum', 'superkingdom'],
+ help='For non-default output formats: Summarize genome taxonomy at this rank and above. Note that the taxonomy CSV must contain lineage information at this rank.'
+ )
+ subparser.add_argument(
+ '-f', '--force', action = 'store_true',
+ help='continue past errors in taxonomy database loading',
+ )
+
+def main(args):
+ import sourmash
+ if not args.gather_csv and not args.from_file:
+ raise ValueError(f"No gather CSVs found! Please input via `-g` or `--from-file`.")
+ if len(args.output_format) > 1:
+ if args.output_base == "-":
+ raise TypeError(f"Writing to stdout is incompatible with multiple output formats {args.output_format}")
+ if not args.rank:
+ if any(x in ["krona", "lineage_summary"] for x in args.output_format):
+ raise ValueError(f"Rank (--rank) is required for krona and lineage_summary output formats.")
+ return sourmash.tax.__main__.metagenome(args)
diff --git a/src/sourmash/cli/tax/prepare.py b/src/sourmash/cli/tax/prepare.py
new file mode 100644
index 0000000000..01d978c73e
--- /dev/null
+++ b/src/sourmash/cli/tax/prepare.py
@@ -0,0 +1,60 @@
+"""combine multiple taxonomy databases into one."""
+
+usage="""
+
+ sourmash tax prepare --taxonomy-csv [ ... ] -o