File object support?

[epicfaace]

I'd like to use indexed_gzip with a file object (in my case, a .tar.gz file that is streamed from Azure Blob Storage). I already am able to create an index file using IndexedGzip (because the file is initially locally stored on creation), but I don't want to download the entire file in order to use the IndexedGzip index to read the archive later. How can I do this?

Related: mxmlnkn/ratarmount#49

[epicfaace]

I can't just pass in BytesIO() to the fileobj argument because IndexedGzip seems to expect that .fileno() is defined on it.

[pauldmccarthy]

Hey @epicfaace, unfortunately this is not possible at this point in time, because the C code which drives indexed_gzip is tightly coupled to reading from a real file (specifically the use of fread, fseek and ftell, on a FILE struct).

Technically there are no barriers to making this possible, as the zlib interface is completely decoupled from the concept of a file. However it would be a fair amount of work to re-organise the indexed_gzip code base to support this, and unfortunately I can only make time for bugfixes and general maintenance at this point in time. PRs welcome though!

[epicfaace]

Thanks for your reply. I'm willing to help change zran.c to get this to work, but I'm trying to better understand how such a change would work / be designed. Do you mind giving some guidance on this? Here's what I'm thinking:

Is the problem that zran.c requires a FILE struct in the first place, or specifically the usage of fread, fseek and ftell?

If it's the former, what kind of data structure would you suggest would be appropriate to pass the file object from Python to C?

If it's the latter, do you know of alternative functions that would work on an in-memory FILE struct that would still work on an in-memory file?

I'm wondering if maybe something like fmemopen might do the trick.

[epicfaace]

Okay, I think I see what would need to be changed. Essentially, I'd need to pass down a file-like object from Python to C, then use code such as what's mentioned here (https://www.geeksforgeeks.org/reading-python-file-like-objects-from-c-python/) to replace the calls to fread, fseek, and ftell. Does that seem like the best way to go about this, or do you have any suggestions as to other things that would need to be changed / better ways to do it?

[pauldmccarthy]

Sorry for the delay; the manpage for fmemopen says that there is no file descriptor associated with, and that fileno will error on, the created stream, so that's unlikely to provide a work-around.

Your suggestion of replacing the FILE * references in zran.(h|c) with a reference to a Python file-like object has promise. I originally kept the zran module as pure C (i.e. de-coupled from Python) to avoid the horridness of writing a Python extension which would be compatible with both Python 2 and 3 - I left that part of the task up to cython. I'm not particularly wedded to the idea of preserving Python 2 compatiblity though - a change such as this could be part of a 2.0 release, leaving the 1.* version compatible with Python 2.

There would be a few complications - for example, in zran_init I use ftell to seek to the end of the file, in order to calculate its compressed size, which is then used in various places for bounds checking. This would obviously not be possible in a streaming scenario, but you mentioned earlier that, in your use-case, you would have already pre-generated an index, so the compressed size could be taken from there.

There would also be the task of managing the various input and output buffers - when preparing a compressed-data buffer before calling zran_read, you would have to ensure that all of the (uncompressed) data you wish to read is present, so that the C code doesn't reach end-of-stream. Perhaps you could query the pre-generated index to calculate how much compressed data needs to be passed in, in order to read a certain amount of uncompressed data.

But I've put a fair amount of effort into the unit tests*, so hopefully they will catch any potential errors which might be introduced. As I've already mentioned, I don't really have a lot of time to work on this, but am happy to try and answer question/help out where possible.

*Effort into making sure that the main code is tested. The quality of the test code leaves something to be desired though..

[epicfaace]

Fixed by #55.