f_orig_query sums to greater than 1.0

[nmb85]

Hi, I've run sourmash gather on approx 1,600 different metagenomes with the following:
parallel -j 64 sourmash gather -k 21 {} -o {/.}.gt ~/sourmash_dbs/nih_smgc_k21.sbt.zip ~/sourmash_dbs/gtdb-rs207.genomic-reps.dna.k21.zip ~/sourmash_dbs/genbank-2022.03-fungi-k21.zip ::: ../*.sig

My immediate goal is to know the total proportion of the metegenome that is contained in my databases. When I sum up the f_orig_query column in each of +1,000 resultant csv files (for X in *.csv; do echo -ne $X"\t"; awk -F "," 'NR>1{sum=sum+$2} END{print(sum)}' $X; done | sort -nk 2,2), I get a wide distribution of proportions, and >99% are between 0 and 1.0, which is what I'd expect. However, the f_orig_query column in 7 of the csv files sums to more than 1.0, which I would not expect. Do I misunderstand the meaning of f_orig_query column, or is there something else that causes a sum greater than 1.0?

I've searched the issues, read this lovely guide to interpreting gather csvs, and haven't found an answer to my question, but I'm skeptical that it hasn't been asked already, so I apologize for any redundancy.

Here is an example gather csv from one of the metagenomes with a f_orig_query column sum greater than 1.0:
ju473iap108192021_S185_adj.csv

[ctb]

hi @nmb85, I think you're looking for f_unique_to_query - f_orig_query is the fraction of the original query that matches, and so if there are overlapping matches (e.g. a single E. coli k-mer in the metagenome will match to many E. coli genomes!) they will be counted multiple times.

[ctb]

(if f_unique_to_query sums to more than 1 for any data set, that is a real big problem for us so please let me know 😆 )

[nmb85]

Great, thanks so much! The sums for f_unique_to_query are indeed 100% between 0 and 1.0! Issue closed.

[ctb]

I will see if I can figure out how to improve the docs too!

[ctb]

hi @nmb85 thanks again for this - over in #2301 I'm updating the docs like so,

The sum of f_unique_to_query across all rows is what is reported in by gather
as the fraction of query k-mers hit by the recovered matches
(unweighted) and should never be greater than 1!

and for f_orig_query I wrote:

Where there are overlapping matches (e.g. to multiple
E. coli species in a gut metagenome) the overlaps will be represented
multiple times in this column.

lmk if you think there should be additions!

[nmb85]

Explicitly stating that f_unique_to_query should never be greater than one and that f_orig_query includes overlaps does the trick, I think! Thanks for clarifying! Sourmash is so useful and your clear documentation really makes that utility accessible.

[ctb]

😊