www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - std.zip and a large archive

reply "AntonSotov" <nepuvive rainmail.biz> writes:
I process archive:
///////////////
import std.stdio, std.zip, std.file;

int main()
{
   auto zip = new ZipArchive(read("c:/test.zip"));
   foreach (item; zip.directory) {
     writeln("processing ", item.name, " ...");
     // processing item...
   }
   return 0;
}
/////////////////
it works well for normal archives.
but how to process zip archive ~1GB ?
it takes a long of RAM.
Jul 19 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 19 July 2014 at 07:55:10 UTC, AntonSotov wrote:
 I process archive:
 ///////////////
 import std.stdio, std.zip, std.file;

 int main()
 {
   auto zip = new ZipArchive(read("c:/test.zip"));
   foreach (item; zip.directory) {
     writeln("processing ", item.name, " ...");
     // processing item...
   }
   return 0;
 }
 /////////////////
 it works well for normal archives.
 but how to process zip archive ~1GB ?
 it takes a long of RAM.
Hmm... it's unfortunate that ZipArchive doesn't take a file descriptor. As a workaround, you can use memory mapping: import std.stdio, std.zip, std.file, std.mmfile; int main() { auto mmfile = new MmFile(File("c:/test.zip", "rb")); auto zip = new ZipArchive(mmfile[]); foreach (item; zip.directory) { writeln("processing ", item.name, " ..."); // processing item... } return 0; }
Jul 19 2014
next sibling parent "AntonSotov" <nepuvive rainmail.biz> writes:
On Saturday, 19 July 2014 at 10:46:31 UTC, Marc Schütz wrote:

 Hmm... it's unfortunate that ZipArchive doesn't take a file 
 descriptor. As a workaround, you can use memory mapping:
auto mmfile = new MmFile("c:/test.zip"); Thank you. it works!
Jul 19 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:

 import std.stdio, std.zip, std.file, std.mmfile;

 int main()
 {
   auto mmfile = new MmFile(File("c:/test.zip", "rb"));
   auto zip = new ZipArchive(mmfile[]);
   foreach (item; zip.directory) {
     writeln("processing ", item.name, " ...");
     // processing item...
   }
   return 0;
 }
This could be added to the ZipArchive online docs. Bye, bearophile
Jul 19 2014
prev sibling parent reply "Domingo Alvarez Duarte" <mingodad gmail.com> writes:
It seems that ti only works for zip files with less than 65000 
entries, zip64 that allow read more than that do seem to be 
implemented.
Jul 19 2014
parent "Domingo Alvarez Duarte" <mingodad gmail.com> writes:
It seems that this only works for zip files with less than 65000
entries, zip64 that allow read more than that do not seem to be
implemented.
Jul 19 2014