www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 2137] New: Data not compressed on fly when adding to zip archive

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2137

           Summary: Data not compressed on fly when adding to zip archive
           Product: D
           Version: 2.014
          Platform: PC
        OS/Version: All
            Status: NEW
          Severity: minor
          Priority: P2
         Component: Phobos
        AssignedTo: bugzilla digitalmars.com
        ReportedBy: dsimcha yahoo.com


The fact that, when an ArchiveMember is added via addMember() to a ZipArchive,
the data in expandedData is not compressed on the fly, makes it very difficult
to create large zip files without running out of memory.  I have created a
simple workaround  by moving the following code from the build() method to the
addMember() method:

de.crc32 = std.zlib.crc32(0, cast(void[])de.expandedData);
de.compressedData =
cast(ubyte[])std.zlib.compress(cast(void[]))de.expandedData;
de.compressedData = de.compressedData[2 .. de.compressedData.length - 4];
de.compressedSize = de.compressedData.length;
de.expandedSize = de.expandedData.length;

I also added the following to free the memory previously occupied by
expandedData:

de.expandedData=null;

This is actually proving to be a significant issue for me, as I am attempting
to use std.zip to archive large amounts of DNA sequence, which compresses
extremely well, but won't fit in memory uncompressed.


--
Jun 02 2008
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2137






Could you please email me the diffs? Thanks!


--
Aug 14 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2137






Created an attachment (id=269)
 --> (http://d.puremagic.com/issues/attachment.cgi?id=269&action=view)
Diffs to make std.zip compress on fly.


--
Aug 15 2008
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2137






Here they are.  Note that I had changed some style stuff to make the code more
readable for me when I was trying to understand it, so a little of that may
show up in the diffs.  Also, I hadn't bothered to include the switch statement
to get store mode to work because I didn't need it, but this should be trivial
to put back.


--
Aug 15 2008