More helpful exception when compressing large files in seek mode

[carlossanlop]

When the user opens a stream in Create mode, and passes it to a ZipArchive opened in Update mode, then attempts to add a very large file to the zip, an exception will be thrown on dispose stating Stream was too long, which is confusing for the user, and frustrating if the file was large and this happens so late.

The exception is thrown because we restrict the size of a file to Int32.MaxValue. The restriction is reached because the Update mode requires a stream that allows seeking, so we internally create a MemoryStream so that we can update the archive, but the size limit for the file is Int32.MaxValue.

We do have it documented here, but it's not very obvious for the user to look for answers in that location.

We should explore two improvements:

1. Throw a more helpful error message when this combination of conditions meet (Stream in Create mode, ZipArchive in Update mode, file too large).
2. Try to detect this case much earlier, instead of waiting it to happen on Dispose.

I also opened to suggest a Roslyn analyzer that would help users avoid falling into this: #35815

This has been reported a few times already:

• System.IO.Packaging.ZipPackage returns "Stream too long" exception with big files #42855
• System.IO.Packaging part stream has a memory leak when writing a large stream #23750
• System.IO.Compression: ZipArchive loads entire file in memory on .Dispose #1543
• System.IO.Compression: ZipArchiveEntry always stores uncompressed data in memory #1544
• Add support for Zip64 for archives > 4GB PowerShell/Microsoft.PowerShell.Archive#19

/cc @ericstj

[WinCPP]

@carlossanlop I am contributing after a long time and would like to attempt this.

[danmoseley]

@WinCPP welcome back! I will assign this to you. Let us know if you need help.

[WinCPP]

@WinCPP welcome back! I will assign this to you. Let us know if you need help.

Thanks! And yes, I do need help 🙂

As per the documentation, the file to be added to an archive in update mode cannot be larger than Int32.MaxValue in size. However, the following exception is thrown even when the size is Int32.MaxValue - 1.

      Message: 
System.OutOfMemoryException : Array dimensions exceeded supported range.

      Stack Trace: 
MemoryStream.set_Capacity(Int32 value) line 288
MemoryStream.EnsureCapacity(Int32 value) line 165
MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) line 626
WrappedStream.Write(Byte[] buffer, Int32 offset, Int32 count) line 155
Stream.CopyTo(Stream destination, Int32 bufferSize) line 72
BufferedFileStreamStrategy.CopyTo(Stream destination, Int32 bufferSize) line 945
FileStream.CopyTo(Stream destination, Int32 bufferSize) line 508
Stream.CopyTo(Stream destination) line 51
ZipFileExtensions.DoCreateEntryFromFile(ZipArchive destination, String sourceFileName, String entryName, Nullable`1 compressionLevel) line 123
ZipFileExtensions.CreateEntryFromFile(ZipArchive destination, String sourceFileName, String entryName, CompressionLevel compressionLevel) line 78
ZipFile_ZipArchive_Create.CreateEntryFromLargeFileArchiveUpdateExtension(Boolean withCompressionLevel) line 52

Here is the line that causes the exception (link).

Digging more, I found this discussion about mscorlib code over here. that mentions that the max possible index for a byte array is 0x7FFFFFC7 although the max size can be Int32.MaxValue. When I try with a file of size 0x7FFFFFC7 it works fine, but just a byte more i.e. 0x7FFFFFC8 and I get the above-mentioned exception. So should the logic for this fix use 0x7FFFFFC7 as the maximum allowable size instead of Int32.MaxValue? Appreciate your inputs.

[jeffhandley]

Digging more, I found this discussion about mscorlib code over here. that mentions that the max possible index for a byte array is 0x7FFFFFC7 although the max size can be Int32.MaxValue. When I try with a file of size 0x7FFFFFC7 it works fine, but just a byte more i.e. 0x7FFFFFC8 and I get the above-mentioned exception. So should the logic for this fix use 0x7FFFFFC7 as the maximum allowable size instead of Int32.MaxValue? Appreciate your inputs.

Hi @WinCPP. I'm sorry we missed your question here. I agree with your assessment here that we should cap the size to 0x7FFFFFC7 instead of Int32.MaxValue. Let us know if you would still like to contribute to this.

[WinCPP]

Hi, @jeffhandley thanks for the reply. I am busy with some office work ATM. So I am fine if someone wants to take this ahead.

[jeffhandley]

Thanks, @WinCPP. And sorry again we missed your question earlier this year.