System.IO.Compression.Brotli Epic

[joshfree]

Tracking item for System.IO.Compression.Brotli

Related #23613

[akoeplinger]

I'd be interested in getting a conclusion on dotnet/corefxlab#1673 as part of this :)

[danmoseley]

Cost of 5-10 days from @ianhays

[ianhays]

Design

• Design the API for the non-allocating API
• Design the API for BrotliStream
• Add License attributions
• Get final approval from LCA
• Submit issue against CoreFX with API proposal for BrotliStream and Brotli
• Take the issue through API review until approval

Unit Testing

• Abstract out the DeflateStream/GZipStream unit tests to be usable for any compression algorithm+
• Add tests for any missing code coverage to the abstract CompressionStreamUnitTests
• Test BrotliStream using the abstract CompressionStreamUnitTests
• Add tests for the non-allocating API.
• Add tests that ensure interoperability with other Brotli implementations
• Submit new Corefx-testdata package to nuget
• Submit a PR against the github corefx-testdata repo with the new tests
• Merge CoreFX-testdata PR
• Get the tests running in ~5s and move any long tests to Outerloop
• Submit a PR against CoreFX with the updates to the shared tests
• Merge CoreFX test PR
• Add test for giving the decompressor a partially encoded stream that has been flushed- does it throw, and if so, when? Try giving the same data to the google/brotli decoder as well as to Chrome
• Add test for calling BrotliEncoderInstance.Compress() with an empty destination but a nonempty source and vice versa.
• Do lots of manual testing of our brotli streams interoperability with GoogleBrotliExe, Chrome, Firefox, IE, and Edge

Perf Testing

• Abstract out the DeflateStream/GZipStream performance tests to be usable for any compression algorithm
• Test BrotliStream using the abstract CompressionStreamPerfTests
• Add perf tests for the non-allocating API.
• Add the GoogleTestData to the current performance tests
• Compare new perf results between DeflateStream, BrotliStream, and the non-allocating API.

Implementation - Managed Wrappers

• Add new sln/csproj/Configurations/etc. in CoreFX
• Add packaging stuff
• Copy BrotliStream/BrotliStatic from CoreFXLab
• Make any changes necessary to the above to get them to work in CoreFX
• Spanify/Memoryify the implementations
• Override the Async functions
• Do any final cleanup for the managed classes, remove all TODOs, correct all of the error message strings
• Bonus: Make _buffer rented
• Bonus: Add a special-case for one-shot encoding/decoding to use the BrotliDecoderDecompress/BrotliEncoderCompress functions
• Bonus: Add perf tests for the one-shot case and compare them with the compress/flush method
• Update the API with the results of the API Review
• Submit PR to CoreFX
• Respond to any feedback and push updates

Implementation - Windows Native

• Port the c brotli code from Google into clrcompression
• Add brotli into our Cmake files, exporting the functions that we're using in the managed wrappers
• Fix any errors and test that the new imports/exports/details of the new clrcompressions are correct for every build config

Implementation - Unix Native

• Decide whether to apt-get brotli or fit brotli into our current native compression package( which is just a shim right now)
• If we apt-get, then add brotli to the apt-get scripts
• If we ship: Port brotli code to our unix package
• If we ship: Add brotli files and info to the Cmake unix builds
• If we ship: Fix any errors and test that the new imports/exports/details of the new non-shim dlls are correct for every build config
• Build and test on OSX to verify the Unix implementation works there as well

[akoeplinger]

@ianhays so does this mean you decided on using a native brotli lib instead of a managed implementation?

[ianhays]

so does this mean you decided on using a native brotli lib instead of a managed implementation?

Yes, though that does not preclude the eventual inclusion of a manged fallback. The primary drivers behind going the native route were:

• lower time to completion
• performance
• easier to update and take new fixes from google/brotli
• the hardest/most time consuming components of getting the native parts packaged are mostly already done because of zlib

I reasoned that because of the comparatively low cost of the native approach, it would be better to implement that first and have the product in a consumable state sooner rather than later. Then if we determine later that a managed implementation would be better, we can rip out the native one without hassle and replace it with a custom-written managed one. That way we also have a solid, tested baseline against which to test.