Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leverage dask directly to improve compute_MVBS efficiency #845

Closed
leewujung opened this issue Oct 11, 2022 · 2 comments
Closed

Leverage dask directly to improve compute_MVBS efficiency #845

leewujung opened this issue Oct 11, 2022 · 2 comments
Assignees
Labels
enhancement This makes echopype better
Milestone

Comments

@leewujung
Copy link
Member

During investigation of #834, I found an efficiency issue in the current implementation of compute_MVBS if the data is delayed. For the 19 files in the OOI eclipse notebook, if the ds_Sv data are persisted in memory it takes 5 mins to compute, but if ds_Sv is lazy-loaded the operations never finished.

Further investigation by @b-reyes showed that the xarray groupby_bins operations are the major hiccups. The under-the-hood implementation can likely be improved dramatically by doing indexing directly using dask, and we can do this in the next release.

@leewujung leewujung added the enhancement This makes echopype better label Oct 11, 2022
@leewujung leewujung added this to the 0.6.4 milestone Oct 11, 2022
@leewujung leewujung moved this to Todo in Echopype Oct 11, 2022
@leewujung
Copy link
Member Author

leewujung commented Oct 13, 2022

Temporary fix? @lsetiawan will try to see if we can delay the function for now to allow this to run through.

the lines that use xarray functions to do mean:

sv_groupby_bins = (
sv.resample(ping_time=pbin, skipna=True)
.mean(skipna=True)
.groupby_bins("echo_range", bins=rint, right=False, include_lowest=True)
.mean(skipna=True)
)

@leewujung
Copy link
Member Author

This is now addressed in #878.

Repository owner moved this from Todo to Done in Echopype Nov 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This makes echopype better
Projects
Status: Done
Development

No branches or pull requests

2 participants