System.IO.Packaging.ZipPackage returns "Stream too long" exception with big files

[qmatteoq]

Description

I'm using the following code, based on the System.IO.Packaging.ZipPackage API, to generate zip files in a WPF application:

private void AddFileToZip(ZipPackage zipPackage, string fileName)
{
    string extension = Path.GetExtension(fileName);
    if (extension == null) return;
 
    PackagePart part = zipPackage.CreatePart(new Uri("/" + Path.GetFileName(fileName), UriKind.Relative),
        "application/" + extension.Substring(1), CompressionOption.Maximum);
    if (part == null) return;
    using (Stream partStream = part.GetStream(FileMode.Create))
    {
        using (Stream fileStream = File.OpenRead(fileName))
        {
            CopyBytes(fileStream, partStream);
            fileStream.Close();
        }
 
        partStream.Close();
    }
}
 
private void CopyBytes(Stream readStream, Stream writeStream)
{
    byte[] bytes = new byte[4096];
 
    int bytesRead = readStream.Read(bytes, 0, bytes.Length);
    while (bytesRead > 0)
    {
        writeStream.Write(bytes, 0, bytesRead);
        bytesRead = readStream.Read(bytes, 0, bytes.Length);
    }
}

This code works fine in the version of the application based on .NET Framework 4.6, regardless of the size of the file to compress. On .NET Core, instead, the operation fails with an IOException and the message "Stream too long" if the file is too big.

Configuration

• .NET Core 3.1 / .NET 5.0 RC1
• Windows 10 19042.51
• x86 and x64

Regression?

Yes, the same code works fine in .NET Framework 4.6.

Other information

You can find a simple repro project here: https://1drv.ms/u/s!AhGLm1VCMcxP7t4yZ5iWm2NwKGf0Dg?e=XajuD2.
This project creates a 2 GB text file in a temporary folder and then tries to zip it using the above code. The project comes with two .csproj files: one for the .NET Framework 4.6 version, which works fine; one for the .NET Core 3.1 version, which generates the IOException.

[carlossanlop]

Thanks for reporting, @qmatteoq . Can you please share the callstacks as well?

[qmatteoq]

Hello @carlossanlop , sorry for the noob question, but which is the preferred way to share the call stack? The one included in the Visual Studio window doesn't seem to contain many information:

[External Code]	
ZipTest.dll!ZipTest.MainWindow.CopyBytes(System.IO.Stream readStream, System.IO.Stream writeStream) Line 70	C#
ZipTest.dll!ZipTest.MainWindow.AddFileToZip(System.IO.Packaging.ZipPackage zipPackage, string fileName) Line 55	C#
ZipTest.dll!ZipTest.MainWindow.OnDoSomething(object sender, System.Windows.RoutedEventArgs e) Line 37	C#
[External Code]	

[danmoseley]

@qmatteoq please disable the "just my code" feature in the debugger options, it should show more of the callstack. Or I believe you can click (maybe right click) on [External Code] and it will give you the option to see it.

[qmatteoq]

Thanks @danmosemsft ! I have enabled "Show external code" in the Call Stack window, let me know if these information are enough:

System.Private.CoreLib.dll!System.IO.MemoryStream.Write(byte[] buffer, int offset, int count)	Unknown
ZipTest.dll!ZipTest.MainWindow.CopyBytes(System.IO.Stream readStream, System.IO.Stream writeStream) Line 70	C#
ZipTest.dll!ZipTest.MainWindow.AddFileToZip(System.IO.Packaging.ZipPackage zipPackage, string fileName) Line 55	C#
ZipTest.dll!ZipTest.MainWindow.OnDoSomething(object sender, System.Windows.RoutedEventArgs e) Line 37	C#
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised)	Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args)	Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseEvent(System.Windows.RoutedEventArgs e)	Unknown
PresentationFramework.dll!System.Windows.Controls.Primitives.ButtonBase.OnClick()	Unknown
PresentationFramework.dll!System.Windows.Controls.Button.OnClick()	Unknown
PresentationFramework.dll!System.Windows.Controls.Primitives.ButtonBase.OnMouseLeftButtonUp(System.Windows.Input.MouseButtonEventArgs e)	Unknown
PresentationCore.dll!System.Windows.UIElement.OnMouseLeftButtonUpThunk(object sender, System.Windows.Input.MouseButtonEventArgs e)	Unknown
PresentationCore.dll!System.Windows.Input.MouseButtonEventArgs.InvokeEventHandler(System.Delegate genericHandler, object genericTarget)	Unknown
PresentationCore.dll!System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate handler, object target)	Unknown
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised)	Unknown
PresentationCore.dll!System.Windows.UIElement.ReRaiseEventAs(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args, System.Windows.RoutedEvent newEvent)	Unknown
PresentationCore.dll!System.Windows.UIElement.OnMouseUpThunk(object sender, System.Windows.Input.MouseButtonEventArgs e)	Unknown
PresentationCore.dll!System.Windows.Input.MouseButtonEventArgs.InvokeEventHandler(System.Delegate genericHandler, object genericTarget)	Unknown
PresentationCore.dll!System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate handler, object target)	Unknown
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised)	Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args)	Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseTrustedEvent(System.Windows.RoutedEventArgs args)	Unknown
PresentationCore.dll!System.Windows.Input.InputManager.ProcessStagingArea()	Unknown
PresentationCore.dll!System.Windows.Input.InputProviderSite.ReportInput(System.Windows.Input.InputReport inputReport)	Unknown
PresentationCore.dll!System.Windows.Interop.HwndMouseInputProvider.ReportInput(System.IntPtr hwnd, System.Windows.Input.InputMode mode, int timestamp, System.Windows.Input.RawMouseActions actions, int x, int y, int wheel)	Unknown
PresentationCore.dll!System.Windows.Interop.HwndMouseInputProvider.FilterMessage(System.IntPtr hwnd, MS.Internal.Interop.WindowMessage msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled)	Unknown
PresentationCore.dll!System.Windows.Interop.HwndSource.InputFilterMessage(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled)	Unknown
WindowsBase.dll!MS.Win32.HwndWrapper.WndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled)	Unknown
WindowsBase.dll!MS.Win32.HwndSubclass.DispatcherCallbackOperation(object o)	Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate callback, object args, int numArgs)	Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.TryCatchWhen(object source, System.Delegate callback, object args, int numArgs, System.Delegate catchHandler)	Unknown
WindowsBase.dll!System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority priority, System.TimeSpan timeout, System.Delegate method, object args, int numArgs)	Unknown
WindowsBase.dll!MS.Win32.HwndSubclass.SubclassWndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam)	Unknown
[Native to Managed Transition]	
user32.dll!UserCallWinProcCheckWow()	Unknown
user32.dll!DispatchMessageWorker()	Unknown
[Managed to Native Transition]	
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame frame)	Unknown
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrame(System.Windows.Threading.DispatcherFrame frame)	Unknown
WindowsBase.dll!System.Windows.Threading.Dispatcher.Run()	Unknown
PresentationFramework.dll!System.Windows.Application.RunDispatcher(object ignore)	Unknown
PresentationFramework.dll!System.Windows.Application.RunInternal(System.Windows.Window window)	Unknown
PresentationFramework.dll!System.Windows.Application.Run()	Unknown
ZipTest.dll!ZipTest.App.Main()	Unknown
[Native to Managed Transition]	
[Inline Frame] hostpolicy.dll!coreclr_t::execute_assembly(int) Line 89	C++
hostpolicy.dll!run_app_for_context(const hostpolicy_context_t & context, int argc, const wchar_t * * argv) Line 246	C++
hostpolicy.dll!run_app(const int argc, const wchar_t * * argv) Line 275	C++
hostpolicy.dll!corehost_main(const int argc, const wchar_t * * argv) Line 398	C++
hostfxr.dll!execute_app(const std::wstring & impl_dll_dir, corehost_init_t * init, const int argc, const wchar_t * * argv) Line 146	C++
[Inline Frame] hostfxr.dll!?A0xb58f0637::read_config_and_execute(const std::wstring &) Line 520	C++
hostfxr.dll!fx_muxer_t::handle_exec_host_command(const std::wstring & host_command, const host_startup_info_t & host_info, const std::wstring & app_candidate, const std::unordered_map<enum known_options,std::vector<std::wstring,std::allocator<std::wstring>>,known_options_hash,std::equal_to<enum known_options>,std::allocator<std::pair<enum known_options const ,std::vector<std::wstring,std::allocator<std::wstring>>>>> & opts, int argc, const wchar_t * * argv, int argoff, host_mode_t mode, wchar_t * result_buffer, int buffer_size, int * required_buffer_size) Line 1001	C++
hostfxr.dll!fx_muxer_t::execute(const std::wstring host_command, const int argc, const wchar_t * * argv, const host_startup_info_t & host_info, wchar_t * result_buffer, int buffer_size, int * required_buffer_size) Line 566	C++
hostfxr.dll!hostfxr_main_startupinfo(const int argc, const wchar_t * * argv, const wchar_t * host_path, const wchar_t * dotnet_root, const wchar_t * app_path) Line 50	C++
ZipTest.exe!exe_start(const int argc, const wchar_t * * argv) Line 236	C++
ZipTest.exe!wmain(const int argc, const wchar_t * * argv) Line 302	C++
[Inline Frame] ZipTest.exe!invoke_main() Line 90	C++
ZipTest.exe!__scrt_common_main_seh() Line 288	C++
kernel32.dll!BaseThreadInitThunk�()	Unknown
ntdll.dll!RtlUserThreadStart�()	Unknown

[danmoseley]

I didn't try it but this looks like #35815.

Package.Open(zipName, FileMode.Create) defaults to FileAccess.ReadWrite. I wonder whether Package.Open(zipName, FileMode.Create, FileAccess.Write) helps.

[danmoseley]

https://docs.microsoft.com/en-us/dotnet/api/system.io.compression.zipfileextensions.createentryfromfile?view=netcore-3.1#System_IO_Compression_ZipFileExtensions_CreateEntryFromFile_System_IO_Compression_ZipArchive_System_String_System_String_System_IO_Compression_CompressionLevel_ warns about this "When ZipArchiveMode.Update is present, the size limit of an entry is limited to Int32.MaxValue. This limit is because update mode uses a MemoryStream internally to allow the seeking required when updating an archive, and MemoryStream has a maximum equal to the size of an int...." but there's no mention in Package.Open docs.

If the above helps I wonder whether you'd be interested in offering a PR to /~https://github.com/dotnet/dotnet-api-docs/blob/master/xml/System.IO.Packaging/Package.xml
to add a similar comment so it appears in https://docs.microsoft.com/en-us/dotnet/api/system.io.packaging.package.open?view=netcore-3.1

[qmatteoq]

Thanks @danmosemsft ! Your suggestion did the trick. I'll be more than happy to open a PR on the documentation.

Thanks a lot for the help!

[danmoseley]

Great. If this seems common we could presumably catch and rethrow with a more helpful message. Or maybe it’s simply fixable @carlossanlop may know whether it is?

[iSazonov]

It is very too old (4 years!) issue PowerShell/Microsoft.PowerShell.Archive#19

I hope this will be improved in 6.0.

Why not use file mappings for large archives?

[carlossanlop]

@qmatteoq if you get a chance to open a PR in dotnet-api-docs to update this file, that would be great.

@danmosemsft since this issue has been reported many times, maybe it would be good to invest in trying to throw a more helpful message, and throw it earlier. I opened this issue to consider the improvement: #43542

Why not use file mappings for large archives?
@iSazonov I am unsure how much effort it would require to detect a large file and switch to a file mapping method. But if you feel interested in exploring this, feel free to open a new issue specifically for that request.

I'll close this since it's been answered.