digitalmars.D.learn - Help with out of memory when using zip

jicman (88/88) Dec 17 2012 Greetings!

=?UTF-8?B?QWxpIMOHZWhyZWxp?= (6/86) Dec 17 2012 Is this a 32-bit program? It may be similar to the following issue:

jic (8/108) Jan 02 2013 Thanks, Ali.

"jicman" <cabrera_ _wrc.xerox.com> writes:

Greetings!

I have this program that zips a file and everything works 
perfectly, if the
files are small enough.  But, I am having to zip files that are 
getting more
and more extreme in size and I am running out of memory.  Here is 
the output
when executing a small program to zip a folder:

  0:36:59.76>zipafile
Zipping file no 0 huge\0.pdf

Zipping file no 1 huge\1.exe

Zipping file no 2 huge\2.exe

Zipping file no 3 huge\updates\p.zip

Zipping file no 4 huge\updates\q.html

Zipping file no 5 huge\updates\z.exe

finished all files...
  ** PLEASE WAIT FOR ZIP FILE TO BE CREATED **
Error: Out of memory

  0:37:56.26>

This is the program in question...

import std.stdio;
import std.file;
import std.date;
import std.zip;
import std.zlib;
import jic.libs.MyFile;

int main(char[][] args)
{

   char[] folder = r"c:\tmp\huge";
   ZipAFolder(folder);
   return 0;
}

void ZipAFolder(char[] zipFolder)
{
   char[] fdname = std.path.getDirName(zipFolder);
   char[][] allfiles = std.file.listdir(zipFolder,"*");
   char[] zipFile = zipFolder ~ ".zip";
   int FilesCnt = 0;

   std.zip.ZipArchive zr;
   zr = new std.zip.ZipArchive();
   int one = 0;
   foreach(char[] f; allfiles)
   {
     char[] f0 = std.string.replace(f,fdname ~ "\\","");  // 
Filename
     writefln("Zipping file no " ~ std.string.toString(one) ~ " " 
~ f0);

     ArchiveMember am = new ArchiveMember();
     am.compressionMethod = 8;
     am.name = f0;
     //am.expandedData = cast(ubyte[]) f.read();
     am.expandedData = cast(ubyte[]) f.read();
     am.expandedSize = am.expandedData.length;
     FileData fd0 = GetFileInfo(f);
     long usedT0;
     if (fd0.creationTime > fd0.modifiedTime)
     {
       usedT0 = fd0.creationTime;
     }
     else
     {
       usedT0 = fd0.modifiedTime;
     }
     am.time = std.date.toDosFileTime(usedT0);
     zr.addMember(am);

std.string.toString(one) ~ " TO
BE ZIPPED ** ");

     one++;
   }
   writefln("finished all files...");
   writefln(" ** PLEASE WAIT FOR ZIP FILE TO BE CREATED ** ");
   std.file.write(zipFile, cast(byte[])zr.build());
   writefln(" ** " ~ std.string.toString(one) ~ " FILES ZIPPED ** 
");
   writefln(zipFile ~ " created.");
   //return 1;
}

As you can see, the last two writelns do not get printed because 
of the Out
Of Memory.  Any help would be greatly appreciated.  Thanks.

josé

Dec 17 2012

=?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:

On 12/17/2012 10:01 PM, jicman wrote:
 Greetings!

 I have this program that zips a file and everything works perfectly, if the
 files are small enough. But, I am having to zip files that are getting more
 and more extreme in size and I am running out of memory. Here is the output
 when executing a small program to zip a folder:

 0:36:59.76>zipafile
 Zipping file no 0 huge\0.pdf

 Zipping file no 1 huge\1.exe

 Zipping file no 2 huge\2.exe

 Zipping file no 3 huge\updates\p.zip

 Zipping file no 4 huge\updates\q.html

 Zipping file no 5 huge\updates\z.exe

 finished all files...
 ** PLEASE WAIT FOR ZIP FILE TO BE CREATED **
 Error: Out of memory

 0:37:56.26>

 This is the program in question...

 import std.stdio;
 import std.file;
 import std.date;
 import std.zip;
 import std.zlib;
 import jic.libs.MyFile;

 int main(char[][] args)
 {

 char[] folder = r"c:\tmp\huge";
 ZipAFolder(folder);
 return 0;
 }

 void ZipAFolder(char[] zipFolder)
 {
 char[] fdname = std.path.getDirName(zipFolder);
 char[][] allfiles = std.file.listdir(zipFolder,"*");
 char[] zipFile = zipFolder ~ ".zip";
 int FilesCnt = 0;

 std.zip.ZipArchive zr;
 zr = new std.zip.ZipArchive();
 int one = 0;
 foreach(char[] f; allfiles)
 {
 char[] f0 = std.string.replace(f,fdname ~ "\\",""); // Filename
 writefln("Zipping file no " ~ std.string.toString(one) ~ " " ~ f0);

 ArchiveMember am = new ArchiveMember();
 am.compressionMethod = 8;
 am.name = f0;
 //am.expandedData = cast(ubyte[]) f.read();
 am.expandedData = cast(ubyte[]) f.read();
 am.expandedSize = am.expandedData.length;
 FileData fd0 = GetFileInfo(f);
 long usedT0;
 if (fd0.creationTime > fd0.modifiedTime)
 {
 usedT0 = fd0.creationTime;
 }
 else
 {
 usedT0 = fd0.modifiedTime;
 }
 am.time = std.date.toDosFileTime(usedT0);
 zr.addMember(am);

 BE ZIPPED ** ");

 one++;
 }
 writefln("finished all files...");
 writefln(" ** PLEASE WAIT FOR ZIP FILE TO BE CREATED ** ");
 std.file.write(zipFile, cast(byte[])zr.build());
 writefln(" ** " ~ std.string.toString(one) ~ " FILES ZIPPED ** ");
 writefln(zipFile ~ " created.");
 //return 1;
 }

 As you can see, the last two writelns do not get printed because of the Out
 Of Memory. Any help would be greatly appreciated. Thanks.

 josé

Is this a 32-bit program? It may be similar to the following issue:

   http://forum.dlang.org/thread/k95mf2$1aar$1 digitalmars.com?page=1

If so, you are advised to manage your own memory; do not rely on the 
garbage collector.

Ali

Dec 17 2012

"jic" <cabrera wrc.xerox.com> writes:

"Ali �ehreli"  wrote ...
 On 12/17/2012 10:01 PM, jicman wrote:
 Greetings!

 I have this program that zips a file and everything works perfectly, if 
 the
 files are small enough. But, I am having to zip files that are getting 
 more
 and more extreme in size and I am running out of memory. Here is the 
 output
 when executing a small program to zip a folder:

 0:36:59.76>zipafile
 Zipping file no 0 huge\0.pdf

 Zipping file no 1 huge\1.exe

 Zipping file no 2 huge\2.exe

 Zipping file no 3 huge\updates\p.zip

 Zipping file no 4 huge\updates\q.html

 Zipping file no 5 huge\updates\z.exe

 finished all files...
 ** PLEASE WAIT FOR ZIP FILE TO BE CREATED **
 Error: Out of memory

 0:37:56.26>

 This is the program in question...

 import std.stdio;
 import std.file;
 import std.date;
 import std.zip;
 import std.zlib;
 import jic.libs.MyFile;

 int main(char[][] args)
 {

 char[] folder = r"c:\tmp\huge";
 ZipAFolder(folder);
 return 0;
 }

 void ZipAFolder(char[] zipFolder)
 {
 char[] fdname = std.path.getDirName(zipFolder);
 char[][] allfiles = std.file.listdir(zipFolder,"*");
 char[] zipFile = zipFolder ~ ".zip";
 int FilesCnt = 0;

 std.zip.ZipArchive zr;
 zr = new std.zip.ZipArchive();
 int one = 0;
 foreach(char[] f; allfiles)
 {
 char[] f0 = std.string.replace(f,fdname ~ "\\",""); // Filename
 writefln("Zipping file no " ~ std.string.toString(one) ~ " " ~ f0);

 ArchiveMember am = new ArchiveMember();
 am.compressionMethod = 8;
 am.name = f0;
 //am.expandedData = cast(ubyte[]) f.read();
 am.expandedData = cast(ubyte[]) f.read();
 am.expandedSize = am.expandedData.length;
 FileData fd0 = GetFileInfo(f);
 long usedT0;
 if (fd0.creationTime > fd0.modifiedTime)
 {
 usedT0 = fd0.creationTime;
 }
 else
 {
 usedT0 = fd0.modifiedTime;
 }
 am.time = std.date.toDosFileTime(usedT0);
 zr.addMember(am);

 BE ZIPPED ** ");

 one++;
 }
 writefln("finished all files...");
 writefln(" ** PLEASE WAIT FOR ZIP FILE TO BE CREATED ** ");
 std.file.write(zipFile, cast(byte[])zr.build());
 writefln(" ** " ~ std.string.toString(one) ~ " FILES ZIPPED ** ");
 writefln(zipFile ~ " created.");
 //return 1;
 }

 As you can see, the last two writelns do not get printed because of the 
 Out
 Of Memory. Any help would be greatly appreciated. Thanks.


 Is this a 32-bit program? It may be similar to the following issue:

Yes, this is a 32-bit program, but it is D1.0 vs. D2.0

   http://forum.dlang.org/thread/k95mf2$1aar$1 digitalmars.com?page=1

 If so, you are advised to manage your own memory; do not rely on the 
 garbage collector.

Thanks, Ali.

The problem is that the std.zip library does not allow me to break the write 
in pieces.  That is why I am asking for help.  How can I break this line,

std.file.write(zipFile, cast(byte[])zr.build());

to get and write chunks?  Any help would be greatly appreciated.  Thanks.

jos�

Jan 02 2013

D Programming

C/C++ Programming

Other

digitalmars.D.learn - Help with out of memory when using zip