Replies: 1 comment
-
Yes, lazy chunks are perfect for that. Look at the format of the lazychunk to understand how bstarts and bsizes are stored. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I want to do partial reads of a remote file compressed with blosc2, transferring as few bytes as possible. Given a list of (offset,extent) ranges in the decompressed output, I'd like to be able to generate a list of (offset, extent) ranges in the compressed file to fetch and then somehow decompress this data with blosc2. I'd like to do this in as few round-trips as possible, ideally one to fetch the header + bstarts, and one to fetch the ranges of compressed data. I have complete control over the generation of the compressed file, and it's also an option to pre-process them somehow (e.g. extracting the header + bstarts out into a separate file for easy lookup).
Is this at all possible with the public api? Is there some example code I should look at? So far I'm thinking about calling
blosc2_schunk_get_chunk
for each chunk during preprocessing, saving that somewhere, and then usingblosc2_io_cb
to define my own reader to read from the downloaded fragments.Beta Was this translation helpful? Give feedback.
All reactions