Native resampler fails for (some) GEO readers due to reduce_data=True #2476
Comments
|
I've had trouble reproducing this, but my biggest confusion is that the AreaDefinition area slices logic should be checking if the two CRSes (of the source area and the target area) are the same: And then doing pretty simple math after that. The fancy geostationary logic only happens if the CRSes are not the same. So we need a case where this is broken and see how/why the CRSes aren't equal. Can someone get me AHI or FCI or some other data that fails this reduce_data=True default case? I don't think I was able to reproduce it on my machine. Has anyone noticed something specific to an operating system for this? |
|
https://sftp.eumetsat.int/public/folder/UsCVknVOOkSyCdgpMimJNQ/User-Materials/Test-Data/MTG/MTG_FCI_L1C_SpectrallyRepresentative_TD-360_May2022/UNCOMPRESSED/RCRC0072.zip you can download a FCI test data full-disk from here |
|
So loading the area from the YAML and loading the area for the vis04 band in your example code shows that the datums for the CRSes are not equal. It is possible they used to be, but PROJ seems to want to take the But for the YAML area it either says it is Or if I add I'll have to ask pyproj for help. |
|
This problem occurs even with import hdf5plugin
from satpy import Scene
from glob import glob
fci_files = glob("/media/nas/x23352/MTG/FCI/L1c-cases/202412100230-norrbotten/*BODY*O_0016_003[89].nc")
sc = Scene(filenames={"fci_l1c_nc": fci_files})
sc.load(["night_microphysics"], generate=False, upper_right_corner="NE", pad_data=False)
ls = sc.resample(resampler="native", reduce_data=False)
ls.save_datasets()Result: |
|
@djhoese I see that the proj issue linked in the pyproj discussion you linked is closed as completed. Does that mean that upgrading proj should solve this problem? |
|
I don't remember most of this but the related PROJ PR was released in PROJ 9.3.0. If someone could test with PROJ 9.3+ with modern FCI data then that'd be nice. I see now that your comment above was still experiencing it. I suppose there is a chance there is something else going on even with new-ish PROJ versions. We'd have to do the same comparisons as before to see if this is CRSes not being equal or something else. |
|
The way you ran your code @gerritholl makes me think you have different amounts of data between the various bands. That is, padding being off and resampling taking the finest resolution area, I would assume some areas aren't equal and that's causing the problems. Can you print out the areas after |
|
I've also seen this with FCI, even when I have all segments. Perhaps some rounding error in the area defs for the different resolutions? I vaguely recall that Himawari used to have a similar issue that was solved with some rounding logic or somehting like that... |
|
Areas: |
|
Where's the extra row coming from on the 1km? 278 * 2 => 556, not 557. |
|
From the data: |
|
Is this a single segment of FCI data? I'm trying to work out why the size is 557...that's not a factor of 5568 or 11136. Do some segments have a size of 556 and others of 557 perhaps, and in this specific case it happens that the size difference is not consistent between bands? |
|
Yes, this is a single segment. The case I posted yesterday consisted of two segments. |
|
How are the pixels aligned between resolutions? For example, on ABI it is meant to be 4 250m pixels perfectly aligned with 1 500m, and similarly 4 500m make up 1 1000m pixel. At least that's how we're "allowed" to think about it. How does this work for this FCI data? |
|
The single segments of FCI data have different sizes (+- 1 row between each other) and indeed the same segment can have different coverage depending on the channel resolution (I notice this only now as well :O). However with padding activated this should not happen, because then the square arrays are always multiple of each other between resolutions. (The FCI padding mechanism is designed to take the different segment sizes into account) And also for FCI, pixels of different resolutions are fully overlapping, same as Dave is describing for ABI. |
@ameraner But how can this be true if there are a different number of rows (unaligned) between resolutions for each segment? @gerritholl So I guess the main question as far as this issue is concerned is do you get this failure when you have all segments and no padding is used? @simonrp84 You mentioned you've seen it, but how long ago was that and what version of PROJ was installed in your environment? And were you padding or not? |
|
It is always true when all segments are merged together and you get the complete full-disc, as then the single segment sizes don't matter anymore (they will always sum up to the full-disc grid size). |
No. Only if I have a subset of segments and I avoid using padding. @ameraner Are the last ten segments (= Q4) guaranteed to be aligned between FDHSI and HRFI even without padding? That will be relevant in particular for MTG-I2 / RSS data. |
|
Good question, I will check. |
It was a good while ago, maybe last year or even earlier, and I vaguely recall that you made a fix for it. Could be wrong though. |
Describe the bug
The
nativeresampler fails when trying to resampler GEO data onto grids (with the same projection).This is solved by deactivateding the data reduction (
scn_r = scn.resample('mtg_fci_fdss_1km', resampler='native', reduce_data=False)).A plausible theory when looking at the code is that the
reduce_datacrops the deep space pixels around the disk, ending up with the slightly smaller array shape that then doesn't match anymore. The slices returned from get_area_slices heresatpy/satpy/scene.py
Lines 914 to 915 in 2ec18f6
(slice(158, 10998, None), slice(157, 10983, None))To Reproduce
Similar failures have been reported by @simonrp84 using AHI data.
Expected behavior
Correct resampling.
Actual results
Additional context
This was reported and discussed with @djhoese around here on Slack https://pytroll.slack.com/archives/C0LNH7LMB/p1682076158103019?thread_ts=1681293614.646929&cid=C0LNH7LMB .
The text was updated successfully, but these errors were encountered: