Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

geotiff writer fails with dask distributed #2834

Closed
gerritholl opened this issue Jun 20, 2024 · 2 comments
Closed

geotiff writer fails with dask distributed #2834

gerritholl opened this issue Jun 20, 2024 · 2 comments
Labels
bug component:writers

Comments

@gerritholl
Copy link
Member

Describe the bug

The geotiff writer fails when using the dask distributed scheduler.

To Reproduce

from dask.distributed import Client, LocalCluster
from glob import glob
from satpy import Scene
from satpy.utils import debug_on; debug_on()

seviri_files = glob("/media/nas/x21308/scratch/SEVIRI/202103300900/H-000*")

def main():
    cluster = LocalCluster()
    client = Client(cluster)
    sc = Scene(filenames=seviri_files, reader=["seviri_l1b_hrit"])
    sc.load(["IR_108"])
    sc.save_datasets("geotiff")

if __name__ == "__main__":
    main()

Expected behavior

I expect an image to be produced without error messages.

Actual results

Computing writer results fails due to a rasterio exception: TypeError: self._hds cannot be converted to a Python object for pickling

[DEBUG: 2024-06-20 15:42:47 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/seviri_l1b_hrit.yaml',)
[DEBUG: 2024-06-20 15:42:48 : satpy.readers.yaml_reader] Assigning to seviri_l1b_hrit: ['/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000024___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000014___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000012___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000022___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000011___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000016___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000015___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000009___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000021___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000010___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000023___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000017___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000020___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000019___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000013___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000018___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-_________-PRO______-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-_________-EPI______-202103300900-__']
[INFO: 2024-06-20 15:42:48 : hrit_msg] No IMPF configuration field found in prologue.
[DEBUG: 2024-06-20 15:42:48 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2024-06-20 15:42:48 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/pyorbital/etc
[DEBUG: 2024-06-20 15:42:48 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.readers.yaml_reader] Requested orientation for Dataset None is 'native' (default). No flipping is applied.
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Reading ['/data/gholl/checkouts/satpy/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /home/gholl/Arbeit/checkouts-perforce/config/enhancements/generic.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /home/gholl/checkouts/pytroll-dwd-config/enhancements/generic.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/seviri.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Data for DataID(name='IR_108', wavelength=WavelengthRange(min=9.8, central=10.8, max=11.8, unit='µm'), resolution=3000.403165817, calibration=<2>, modifiers=()) will be enhanced with options:
	[{'name': 'stretch', 'method': <function stretch at 0x7f73bb20f600>, 'kwargs': {'stretch': 'linear'}}]
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Applying stretch linear with parameters {}
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Perform a linear contrast stretch.
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Calculate the histogram quantiles: 
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Left and right quantiles: 0.005 0.005
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f740c109a00>
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f73bb4aaef0> created
[DEBUG: 2024-06-20 15:42:49 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f73bb4aaef0>.
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f740c109a00>
[DEBUG: 2024-06-20 15:42:49 : rasterio._io] Path: _UnparsedPath(path='IR_108_20210330_090000.tif'), mode: w, driver: GTiff
[DEBUG: 2024-06-20 15:42:49 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-20 15:42:49 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f740c109a00>
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f73bb4aaef0> options
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f73bb4aaef0>.
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f740c109a00>
[INFO: 2024-06-20 15:42:49 : satpy.writers] Computing and writing results...
2024-06-20 15:42:49,450 - distributed.protocol.pickle - ERROR - Failed to serialize <ToPickle: HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7f73bb25f560>
 0. store-sources-03ce331851719a2514a22c56dc4ea25e
 1. store-map-9809be199e75ba2d686f687a2db7a555
 2. store-4b68c2333fe6b23c4552346ea1966e10
>.
Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
           ^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling
Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/serialize.py", line 363, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/serialize.py", line 78, in pickle_dumps
    frames[0] = pickle.dumps(
                ^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
           ^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/gholl/checkouts/protocode/test-dask-distributed-geotiff.py", line 16, in <module>
    main()
  File "/data/gholl/checkouts/protocode/test-dask-distributed-geotiff.py", line 13, in main
    sc.save_datasets("geotiff")
  File "/data/gholl/checkouts/satpy/satpy/scene.py", line 1293, in save_datasets
    return writer.save_datasets(dataarrays, compute=compute, **save_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/checkouts/satpy/satpy/writers/__init__.py", line 756, in save_datasets
    return compute_writer_results([results])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/checkouts/satpy/satpy/writers/__init__.py", line 594, in compute_writer_results
    da.compute(delayeds)
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/dask/base.py", line 661, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/serialize.py", line 389, in serialize
    raise TypeError(msg, str_x) from exc
TypeError: ('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 3 layers.\n<dask.highlevelgraph.HighLevelGraph object at 0x7f73bb25f560>\n 0. store-sources-03ce331851719a2514a22c56dc4ea25e\n 1. store-map-9809be199e75ba2d686f687a2db7a555\n 2. store-4b68c2333fe6b23c4552346ea1966e10\n>')

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Info:

  • OS: openSUSE Leap 15.3
  • Satpy Version: current satpy main (v0.49.0-102-g8dfc26145)

Additional context

Encountered while working on #2686.
@gerritholl gerritholl mentioned this issue Jun 20, 2024
Open
7 tasks
@djhoese
Copy link
Member

djhoese commented Jun 20, 2024

This is a known limitation of the writer. We need to switch to rioxarray and use more complicated locking to allow for threaded or distributed schedulers.

@gerritholl
Copy link
Member Author

Duplicate of #1762.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug component:writers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gerritholl @djhoese