Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy rectilinear interpolator #6084

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b77af1d
lazy interpolation using map_complete_blocks
fnattino Jul 25, 2024
0021cb0
pre-commit fixes
fnattino Jul 25, 2024
18908ae
replace test on interpolation with lazy data
fnattino Aug 27, 2024
47a8599
Merge branch 'main' into lazy-rectilinearinterpolator-2
fnattino Sep 11, 2024
555f3c7
Update lib/iris/analysis/_interpolation.py
fnattino Sep 13, 2024
7a08108
Update lib/iris/analysis/_interpolation.py
fnattino Sep 13, 2024
c453e01
Merge branch 'lazy-rectilinearinterpolator-2' of github.com:fnattino/…
fnattino Sep 13, 2024
3814383
resume local import
fnattino Sep 17, 2024
0c5dc9a
add entry to latest.rst
fnattino Sep 20, 2024
3481e46
add author name to list
fnattino Sep 20, 2024
63714f0
drop duplicated method
fnattino Nov 28, 2024
7aaefd8
Merge branch 'main' into lazy-rectilinearinterpolator-2
fnattino Nov 28, 2024
98143f2
new signature of map_complete_blocks
fnattino Nov 28, 2024
948c75e
update docstrings on lazy data
fnattino Nov 28, 2024
09974f3
Merge branch 'main' into lazy-rectilinearinterpolator-2
fnattino Nov 28, 2024
18c6e7a
Merge branch 'main' into lazy-rectilinearinterpolator-2
fnattino Dec 6, 2024
d190a8b
update userguide with lazy interpolator
fnattino Dec 6, 2024
3b210e4
Merge branch 'main' into lazy-rectilinearinterpolator-2
trexfeathers Dec 18, 2024
47bce0e
the unstructured NN regridder does not support lazy data
fnattino Dec 22, 2024
609c75a
remove caching an interpolator
fnattino Dec 22, 2024
dd14caa
update what's new entry
fnattino Dec 22, 2024
48f47eb
Merge branch 'main' into lazy-rectilinearinterpolator-2
fnattino Dec 22, 2024
2fb8fc3
remove links to docs section about caching interpolators
fnattino Dec 22, 2024
a4774b8
Merge branch 'main' into lazy-rectilinearinterpolator-2
fnattino Jan 7, 2025
fb8cffd
Merge branch 'main' into lazy-rectilinearinterpolator-2
fnattino Jan 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 15 additions & 55 deletions docs/src/userguide/interpolation_and_regridding.rst
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wanted to double check, but I think we should also remove the subsections on "caching an interpolator" and "caching a regridder": as discussed in the thread below #6084 (comment), almost all the work is done when calling the interpolator (within RectilinearInterpolator._interpolate) so there seem to be no actual benefit in constructing and reusing the interpolator objects? And I think this is also true for the regridder, for which all the work seems to done within RectilinearRegridder._regrid?

Re-using regridders

Please do NOT remove the section about re-using a regridder. This receives much more attention from the team and we know that it provides important performance benefits - the preparation step is more expensive for regridding.

Re-using interpolators

Thanks for the heads-up: I was not aware that interpolator re-use was an advertised feature (we talk a lot more about regridder re-use).

Since it's an advertised feature, I raised this with @SciTools/peloton to make sure we are happy with losing this functionality:

  • We confirmed that re-use code does not break, which is essential. The only change for the user is that re-use no longer provides a benefit.
  • It is believable that the performance benefits of re-use are not as great as with regridding - we are not caching weights here, but a SciPy object.
  • The ideal situation would be parallelisation AND re-use, but if there is a choice then parallelisation wins.
  • There are future plans (perhaps @stephenworsley can provide cross-references here) to use modern matrix-multiplication for all regridding and interpolation, which makes us more comfortable with the possibility that performance still isn't the best for all cases - it will get even better in the long-term!

So we're still 👍 on removing the re-use functionality. Outstanding work for you:

  • Remove this documentation section as you suggest.
  • Share in the What's New entry that this is no longer a performance feature (although the suggested code will still work).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for checking the documentation updates and for sharing about the future plans for interpolation and regridding! I have added the two fixes, please let me know if there is anything left to do on this!

Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ The following are the regridding schemes that are currently available in Iris:
* point in cell regridding (:class:`iris.analysis.PointInCell`) and
* area-weighted regridding (:class:`iris.analysis.AreaWeighted`, first-order conservative).

The linear, nearest-neighbor, and area-weighted regridding schemes support
lazy regridding, i.e. if the source cube has lazy data, the resulting cube
will also have lazy data.
The linear and nearest-neighbour interpolation schemes, and the linear, nearest-neighbour,
and area-weighted regridding schemes support lazy regridding, i.e. if the source cube has lazy data,
the resulting cube will also have lazy data.
See :doc:`real_and_lazy_data` for an introduction to lazy data.
See :doc:`../further_topics/which_regridder_to_use` for a more in depth overview of the different regridders.

Expand Down Expand Up @@ -194,46 +194,6 @@ For example, to mask values that lie beyond the range of the original data:
[-- 494.44451904296875 588.888916015625 683.333251953125 777.77783203125
872.2222290039062 966.666748046875 1061.111083984375 1155.555419921875 --]


.. _caching_an_interpolator:

Caching an Interpolator
^^^^^^^^^^^^^^^^^^^^^^^

If you need to interpolate a cube on multiple sets of sample points you can
'cache' an interpolator to be used for each of these interpolations. This can
shorten the execution time of your code as the most computationally
intensive part of an interpolation is setting up the interpolator.

To cache an interpolator you must set up an interpolator scheme and call the
scheme's interpolator method. The interpolator method takes as arguments:

#. a cube to be interpolated, and
#. an iterable of coordinate names or coordinate instances of the coordinates that are to be interpolated over.

For example:

>>> air_temp = iris.load_cube(iris.sample_data_path('air_temp.pp'))
>>> interpolator = iris.analysis.Nearest().interpolator(air_temp, ['latitude', 'longitude'])

When this cached interpolator is called you must pass it an iterable of sample points
that have the same form as the iterable of coordinates passed to the constructor.
So, to use the cached interpolator defined above:

>>> latitudes = np.linspace(48, 60, 13)
>>> longitudes = np.linspace(-11, 2, 14)
>>> for lat, lon in zip(latitudes, longitudes):
... result = interpolator([lat, lon])

In each case ``result`` will be a cube interpolated from the ``air_temp`` cube we
passed to interpolator.

Note that you must specify the required extrapolation mode when setting up the cached interpolator.
For example::

>>> interpolator = iris.analysis.Nearest(extrapolation_mode='nan').interpolator(cube, coords)


.. _regridding:

Regridding
Expand Down Expand Up @@ -417,24 +377,24 @@ In each case ``result`` will be the input cube regridded to the grid defined by
the target grid cube (in this case ``rotated_psl``) that we used to define the
cached regridder.

Regridding Lazy Data
^^^^^^^^^^^^^^^^^^^^
Interpolating and Regridding Lazy Data
--------------------------------------

If you are working with large cubes, especially when you are regridding to a
high resolution target grid, you may run out of memory when trying to
regrid a cube. When this happens, make sure the input cube has lazy data
If you are working with large cubes, you may run out of memory when trying to
interpolate or regrid a cube. For instance, this might happen when regridding to a
high resolution target grid. When this happens, make sure the input cube has lazy data

>>> air_temp = iris.load_cube(iris.sample_data_path('A1B_north_america.nc'))
>>> air_temp
<iris 'Cube' of air_temperature / (K) (time: 240; latitude: 37; longitude: 49)>
>>> air_temp.has_lazy_data()
True

and the regridding scheme supports lazy data. All regridding schemes described
here support lazy data. If you still run out of memory even while using lazy
data, inspect the
`chunks <https://docs.dask.org/en/latest/array-chunks.html>`__
:
and the interpolation or regridding scheme supports lazy data. All interpolation and
regridding schemes described here with exception of :class:`iris.analysis.PointInCell`
(point-in-cell regridder) and :class:`iris.analysis.UnstructuredNearest` (nearest-neighbour
regridder) support lazy data. If you still run out of memory even while using lazy data,
inspect the `chunks <https://docs.dask.org/en/latest/array-chunks.html>`__ :

>>> air_temp.lazy_data().chunks
((240,), (37,), (49,))
Expand All @@ -455,6 +415,6 @@ dimension, to regrid it in 8 chunks of 30 timesteps at a time:
Assuming that Dask is configured such that it processes only a few chunks of
the data array at a time, this will further reduce memory use.

Note that chunking in the horizontal dimensions is not supported by the
regridding schemes. Chunks in these dimensions will automatically be combined
Note that chunking in the horizontal dimensions is not supported by the interpolation
and regridding schemes. Chunks in these dimensions will automatically be combined
before regridding.
9 changes: 8 additions & 1 deletion docs/src/whatsnew/latest.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,13 @@ This document explains the changes made to Iris for this release

#. N/A

#. `@fnattino`_ enabled lazy cube interpolation using the linear and
nearest-neighbour interpolators (:class:`iris.analysis.Linear` and
:class:`iris.analysis.Nearest`). Note that this implementation removes
performance benefits linked to caching an interpolator object. While this does
not break previously suggested code (instantiating and re-using an interpolator
object remains possible), this is no longer an advertised feature. (:pull:`6084`)


🔥 Deprecations
===============
Expand Down Expand Up @@ -98,7 +105,7 @@ This document explains the changes made to Iris for this release
Whatsnew author names (@github name) in alphabetical order. Note that,
core dev names are automatically included by the common_links.inc:


.. _@fnattino: httsps://github.com/fnattino


.. comment
Expand Down
8 changes: 2 additions & 6 deletions lib/iris/analysis/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2673,9 +2673,7 @@ def interpolator(self, cube, coords):
the given coordinates.

Typically you should use :meth:`iris.cube.Cube.interpolate` for
interpolating a cube. There are, however, some situations when
constructing your own interpolator is preferable. These are detailed
in the :ref:`user guide <caching_an_interpolator>`.
interpolating a cube.

Parameters
----------
Expand Down Expand Up @@ -2876,9 +2874,7 @@ def interpolator(self, cube, coords):
by the dimensions of the specified coordinates.

Typically you should use :meth:`iris.cube.Cube.interpolate` for
interpolating a cube. There are, however, some situations when
constructing your own interpolator is preferable. These are detailed
in the :ref:`user guide <caching_an_interpolator>`.
interpolating a cube.

Parameters
----------
Expand Down
Loading
Loading