Can't create I;16S TIFF images

[xan-der]

Moving to a more standard install of MapProxy, using Pillow. Pillow can load 16 bit grayscale TIFF images, but can't create them. Similar bug was in PIL. Steps to reproduce:

# python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image;
>>> filename = "GMTED.tiff";
>>> img = Image.open(filename);
>>> img.mode
'I;16S'
>>> img.size
(256, 256)
>>> img1 = Image.core.new(img.mode, img.size);
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unrecognized mode
>>> 

In the old PIL code, the changes below would fix the problem. I will need to look at the Pillow code to see if it is similar enough to make the same mods.

  Imaging.h - increase buffer for mode to avoid overrun in Storage.c
    char mode[6+1]; /* Band names ("1", "L", "P", "RGB", "RGBA", "CMYK", "YCbCr", "BGR;xy") */
  Access.c - add I;16S to list and update hash code to avoid collision with new value
    #define ACCESS_TABLE_SIZE 25
    #define ACCESS_TABLE_HASH 5543
    ADD("I;16S", line_16, get_pixel_16L, put_pixel_16L);
  Pack.c - add I;16S to storage modes using "copy2"
    {"I;16S",   "I;16S",      16,     copy2},
  Storage.c - change multiple strcmp(mode, ...) to single strncmp(mode, "I;16", 4) so all cases are covered
    } if (strncmp(mode, "I;16", 4) == 0) {
  Unpack.c - add I;16S to the "storage modes" part of the unpackers array
    {"I;16S",   "I;16S",      16,     copy2},
  Convert.c - Add converter references
    { "I", "I;16S", I_I16L },
    { "I;16S", "I", I16L_I },
    { "L", "I;16S", L_I16L },
    { "I;16S", "L", I16L_L },
  ImageMode.py - Add I;16S to mapping modes
        _modes["I;16S"] = ModeDescriptor("I;16S", "I", "L", "L")

[wiredfool]

Pillow has supported some 16bit integer modes for a while. Currently there's support for unsigned int16, in little or bigendian mode. Some of that should patch cleanly, but the Access.c and Imaging.h won't. There's also a PyAccess.py version now to support cffi based pixel access method which would need some attention.

I'm a concerned for the correctness of simply accessing and converting a signed int as a little endian unsigned int. That's likely to be wrong in some cases.

[xan-der]

I'm a concerned for the correctness of simply accessing and converting a signed int as a little endian unsigned int. That's likely to be wrong in some cases.

I totally agree. I'd prefer a real solution to the workaround I posted. As things stand, I can't use MapProxy with 16 bit terrain data because the representation is signed. Both PIL and Pillow throw a ValueError exception when they encounter the I;16S mode.

[wiredfool]

Are there samples of the 16bit terrain data online?

[xan-der]

Yes, a good one is GMTED, available from: http://earthexplorer.usgs.gov/. Available in 30, 15, and 7.5 arcsecond resolutions. You need to register with the USGS site to download.

[gaelysam]

Same error with a 6001x6001 16-bit TIFF image from SRTM. What should I do ?

[wiredfool]

What kind of image is it? Can I get a sample?

[gaelysam]

>>> im = Image.open('/home/gael/Téléchargements/srtm_38_03/srtm_38_03.tif' )
>>> im
<PIL.TiffImagePlugin.TiffImageFile image mode=I;16S size=6001x6001 at 0x7F96516CE6D8>

Address: http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/srtm_38_03.zip (36 MB zip)

[tomderuijter]

Bump!

[s20208413]

How to deal with I;16S mode? I met this error while processing a GeoTIFF file.

$ python
Python 3.6.0 (default, Jan 16 2017, 12:12:55) 
[GCC 6.3.1 20170109] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL
>>> PIL.PILLOW_VERSION
'4.0.0'
>>> from PIL import Image
>>> m = Image.open('1.tif')
>>> m.format
'TIFF'
>>> m.size
(1552, 1024)
>>> m.mode
'I;16S'
>>> m.save('test.tif', 'tiff')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/PIL/Image.py", line 1698, in save
    self.load()
  File "/usr/lib/python3.6/site-packages/PIL/TiffImagePlugin.py", line 1014, in load
    return super(TiffImageFile, self).load()
  File "/usr/lib/python3.6/site-packages/PIL/ImageFile.py", line 180, in load
    self.load_prepare()
  File "/usr/lib/python3.6/site-packages/PIL/ImageFile.py", line 254, in load_prepare
    self.im = Image.core.new(self.mode, self.size)
ValueError: unrecognized mode
>>> 

---

A simple workaround:

>>> m = Image.open('1.tif')
>>> m.format
'TIFF'
>>> m.size
(1552, 1024)
>>> m.mode
'I;16S'
>>> m.getpixel((0, 0))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/PIL/Image.py", line 1212, in getpixel
    self.load()
  File "/usr/lib/python3.6/site-packages/PIL/TiffImagePlugin.py", line 1014, in load
    return super(TiffImageFile, self).load()
  File "/usr/lib/python3.6/site-packages/PIL/ImageFile.py", line 180, in load
    self.load_prepare()
  File "/usr/lib/python3.6/site-packages/PIL/ImageFile.py", line 254, in load_prepare
    self.im = Image.core.new(self.mode, self.size)
ValueError: unrecognized mode
>>> m.mode = 'I'
>>> m.mode
'I'
>>> m.getpixel((0, 0))
1067
>>> 

If set mode manually (m.mode = 'I'), then Image.getpixel() works. But Image.save() still can not work.

[aafaque33]

I am getting same error for Tiff file with Image mode "I" and 16BS type .
I am reading data (raw data) from URL directly. Able to get Tiff Image File object but not able to get-pixels or get-data as the error comes out each time that "unrecognized mode"

How to deal with this error ?

resp2 = req.get('tiff file url')
rawdata = resp2.content
im = Image.open(BytesIO(rawdata))
im

Output

<PIL.TiffImagePlugin.TiffImageFile image mode=I;16BS size=4800x4800 at 0x11E36080>

Error:
   257         if not self.im or\
    258            self.im.mode != self.mode or self.im.size != self.size:
--> 259             self.im = Image.core.new(self.mode, self.size)
    260         # create palette (optional)
    261         if self.mode == "P":

ValueError: unrecognized mode

[ghost]

Hello? Is there any issue fixing this problem? I meet the same problem when merge USGS's TIF data to "RGB"

[aclark4life]

@lixingang Hello! I'd expect there is some issue since this was opened in 2014 and still open …

[radarhere]

The SRTM URL no longer works. However, going to the base site and downloading http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/tiff/srtm_14_02.zip, I find the the TIFF mode to now be I.

[radarhere]

#2748 is the reason for this. If I go back to 4.2.0, the files open as I;16S.

So the problem from @s20208413 of reading I;16S images and trying to save them again should have been fixed by that issue.
The problem from @aafaque33 of reading I;16BS should have been fixed.
The problem from @ghost / @lixingang of opening files and converting to RGB should have been fixed.
So this issue is now just the original problem of creating I;16S TIFF images from scratch.

[radarhere]

When trying to save a I;16S TIFF image, the user at the start of this issue attempted to create an I;16S image. They were hindered by the fact that I;16S is not one of our modes.

I'm reluctant to add another mode, particularly because of #2228.

However, if you create a I;16 image, it can be saved with a SAMPLEFORMAT tag of 2.

from PIL import Image, TiffImagePlugin
im = Image.new("I;16", (100, 100))
im.save("i16s.tiff", tiffinfo={TiffImagePlugin.SAMPLEFORMAT: 2})

That will create an I;16S TIFF image.