memory profiling of `sourmash gather` - FYI

[ctb]

mprof run sourmash gather SRR606249.k31.sig.gz /group/ctbrowngrp/sourmash-db/gtdb-rs207/gtdb-rs207.genomic-reps.dna.k31.zip --save-prefetch-csv p2.csv -o g2.csv
mprof plot -o ~/transfer/gather.png

shows three phases:

• initial load;
• prefetch;
• gather

[ctb]

it looks like the Rust ZipStorage implementation might be the cause of the steady increase in memory. I generated the below graphs with mprof by doing a prefetch() over ~2000 sketches in a ZipFileLinearIndex with something that had no matches. With the current default ZipStorage read-only implementation, I get the first graph. When I force the use of _RwZipStorage in sbt_storage.py, I get the second graph.

A quick perusal of the Rust code in src/core/src/storage.rs doesn't turn up any obvious caching or anything. @luizirber any ideas?

rs zip storage

py zip storage

[ctb]

I note the complicated discussion of memory mapping in the rust ZipStorage PR here. Is this behavior maybe related to that?

[ctb]

more - I did use guppy3 (on the Python side) and the total heap usage in Python, at least, does not change.

[ctb]

I dug a bit more this morning, and saw this in the mprof docs -

memory_profiler supports different memory tracking backends including: ‘psutil’, ‘psutil_pss’, ‘psutil_uss’, ‘posix’, ‘tracemalloc’. If no specific backend is specified the default is to use “psutil” which measures RSS aka “Resident Set Size”. In some cases (particularly when tracking child processes) RSS may overestimate memory usage

which linked to this section in the psutil docs:

uss (Linux, macOS, Windows): aka “Unique Set Size”, this is the memory which is unique to a process and which would be freed if the process was terminated right now.

I ran my benchmarking script like so:

mprof run --backend psutil_uss python myprefetch.py

and got the same chart:

here's the code, which only takes 20 seconds or so to run:

# run in ~ctbrown/scratch/2022-pymagsearch

import sourmash
from sourmash import sourmash_args

class empty:
    pass
obj = empty()
obj.picklist = 'mf.csv::manifest'
picklist = sourmash_args.load_picklist(obj)

sigfile = 'gtdb-rs207-k31/af7ac805.k=31.scaled=1000.DNA.dup=0.63.sig'
ss = list(sourmash.load_file_as_signatures(sigfile)).pop()
ss = ss.to_mutable()
ss.minhash = ss.minhash.flatten()

dbfile = '/group/ctbrowngrp/sourmash-db/gtdb-rs207/gtdb-rs207.genomic-reps.dna.k31.zip'

for i in range(2):
    db = sourmash.load_file_as_index(dbfile)
    db = sourmash_args.apply_picklist_and_pattern(db, picklist, None)
    print("iteration:", i)
    print("matches: ", list(db.prefetch(ss, 50000)))

[ctb]

this is a good discussion/approach: using a small container to verify small memory usage 😆 #1909 (comment)