System.IO.Compression: ZipArchive loads entire file in memory on .Dispose

[qmfrederik]

When you open a ZipArchive in Update mode, the entire zip file will be loaded in memory when the .Dispose method is invoked.

This is because .Dipose calls .WriteFile, which:

• Calls LoadLocalHeaderExtraFieldAndCompressedBytesIfNeeded for all entries, which loads the compressed data into memory for those entries
• Sets the size of the .zip archive to 0, by calling _archiveStream.SetLength(0);
• Writes out all entries one by one.

As a result:

• A lot of memory is used, the compressed data for each entry is loaded into memory
• A lot of unnecessary disk I/O is performed, because all entries are written out again, even if they were not modified.

An alternative may be to incrementally update the zip archive, and only update entries which changes.

[jakubsuchybio]

This is a real pain. We use full .NET Framework and we have some really large zip files where we need to update small files inside, but because of this loading into memory, we are getting OutOfMemoryException, because we have 32b application and we cannot change into 64bit, because of dependent driver DLLs that are in 32b.

[carlossanlop]

Triage:
We should scope this to fix only loading entries that have changes.

[Jlalond]

@carlossanlop Hmm Looks like it's only writing files on creates/updates.

But it looks like it's still iterating over them all , I don't see any properties in ZipArchiveEntry to denote if it's been modified, I'm going to keep doing research once I get home, but I think this would be fun to pick up

[IDisposable]

Anyone working on this? I could pick it up if not.

[ulrichb]

Just found this ticket after creating a duplicate (#94455), and wanted to crosspost our use case where the current behavior is a big issue:

a) We're dealing with potentially large user-provided ZIP files (in the GB range),
b) we need to update entries (actually not directly but via System.IO.Packaging.ZipPackage for OPC file processing), and
c) the whole can happen in parallel.

This means with the current ZipArchive implementation we need to reserve dozens of GBs of virtual memory just for System.IO.Packaging.ZipPackage processing, otherwise we're risking container memory limit violations and therefore OOM exceptions.