www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How do I use std.zlib to write a struct of data to a binary file with

reply "TJB" <broughtj gmail.com> writes:
Hello,

I am trying to save some data to compressed binary files.  I have 
my data in a struct, which I can write to a binary file just fine:

align(1) struct TradesBin {
   int ttim;
   int prc;
   int siz;
   short g127;
   short corr;
   char[2] cond;
   char ex;
}

auto fout = new BufferedFile(outfile, FileMode.Out);
fout.writeExact(&bin, TradesBin.sizeof);

Where bin is of type TradesBin, filled with data.  The question 
is how do I now do this with zlib to get compression?  
Suggestions and help are mightily appreciated!

TJB
Sep 24 2012
next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
On Mon, 24 Sep 2012 22:10:04 +0200, TJB wrote:


 Hello,
 
 I am trying to save some data to compressed binary files.  I have my
 data in a struct, which I can write to a binary file just fine:
 
 align(1) struct TradesBin {
    int ttim; int prc;
    int siz; short g127; short corr; char[2] cond;
    char ex;
 }
 
 auto fout = new BufferedFile(outfile, FileMode.Out);
 fout.writeExact(&bin, TradesBin.sizeof);
 
 Where bin is of type TradesBin, filled with data.  The question is how
 do I now do this with zlib to get compression?
 Suggestions and help are mightily appreciated!
 
 TJB


Since you're already set up with the Stream interface, try creating a 
MemoryStream instead of a BufferedFile.  Write to the stream just as you 
are now, then use the .data() property (on MemoryStream's superclass, 
TArrayStream) to get an array of raw bytes.  You can feed this array into 
the compress function in std.zlib to produce a new (hopefully shorter) 
array of bytes that contains the compressed data.  Simply write this 
compressed data to a file with `fout.writeExact(compressed.ptr, 
compressed.length)` and you're done.

Since uncompress works better if you know the exact size of the 
uncompressed data, you might also consider writing the uncompressed size 
of the data to your output file before the compressed data.

Justin
Sep 24 2012
parent reply "TJB" <broughtj gmail.com> writes:
 Since you're already set up with the Stream interface, try 
 creating a
 MemoryStream instead of a BufferedFile.  Write to the stream 
 just as you
 are now, then use the .data() property (on MemoryStream's 
 superclass,
 TArrayStream) to get an array of raw bytes.  You can feed this 
 array into
 the compress function in std.zlib to produce a new (hopefully 
 shorter)
 array of bytes that contains the compressed data.  Simply write 
 this
 compressed data to a file with `fout.writeExact(compressed.ptr,
 compressed.length)` and you're done.

 Since uncompress works better if you know the exact size of the
 uncompressed data, you might also consider writing the 
 uncompressed size
 of the data to your output file before the compressed data.

 Justin


Ok, I think I get it.  Can you help me set up the first part, 
writing to the MemoryStream?

TJB
Sep 24 2012
parent reply Justin Whear <justin economicmodeling.com> writes:
On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote:


 Since you're already set up with the Stream interface, try creating a
 MemoryStream instead of a BufferedFile.  Write to the stream just as
 you are now, then use the .data() property (on MemoryStream's
 superclass,
 TArrayStream) to get an array of raw bytes.  You can feed this array
 into the compress function in std.zlib to produce a new (hopefully
 shorter)
 array of bytes that contains the compressed data.  Simply write this
 compressed data to a file with `fout.writeExact(compressed.ptr,
 compressed.length)` and you're done.

 Since uncompress works better if you know the exact size of the
 uncompressed data, you might also consider writing the uncompressed
 size of the data to your output file before the compressed data.

 Justin


 
 Ok, I think I get it.  Can you help me set up the first part,
 writing to the MemoryStream?
 
 TJB


I wrote up a quick example program and pasted it here:  
  http://dpaste.dzfl.pl/f1699d07

Hope that helps you out.
Sep 25 2012
next sibling parent "TJB" <broughtj gmail.com> writes:
On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:


 I wrote up a quick example program and pasted it here:
   http://dpaste.dzfl.pl/f1699d07

 Hope that helps you out.


Justin,

Thanks so much! Very helpful!!!

TJB
Sep 25 2012
prev sibling next sibling parent Justin Whear <justin economicmodeling.com> writes:
On Tue, 25 Sep 2012 18:08:44 +0000, Justin Whear wrote:


 On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote:
 

 Since you're already set up with the Stream interface, try creating a
 MemoryStream instead of a BufferedFile.  Write to the stream just as
 you are now, then use the .data() property (on MemoryStream's
 superclass,
 TArrayStream) to get an array of raw bytes.  You can feed this array
 into the compress function in std.zlib to produce a new (hopefully
 shorter)
 array of bytes that contains the compressed data.  Simply write this
 compressed data to a file with `fout.writeExact(compressed.ptr,
 compressed.length)` and you're done.

 Since uncompress works better if you know the exact size of the
 uncompressed data, you might also consider writing the uncompressed
 size of the data to your output file before the compressed data.

 Justin


 
 Ok, I think I get it.  Can you help me set up the first part,
 writing to the MemoryStream?
 
 TJB


 
 I wrote up a quick example program and pasted it here:
   http://dpaste.dzfl.pl/f1699d07
 
 Hope that helps you out.


I should point out that in my example, the writing to the MemoryStream is 
not strictly necessary.  I could have simply cast the foo array to an 
array of bytes and passed that into the compress function.  But if you're 
already set up for writing raw data into a stream, the MemoryStream will 
help you minimize code changes as well as facilitate more complex 
scenarios.
Sep 25 2012
prev sibling parent "TJB" <broughtj gmail.com> writes:
On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:


 I wrote up a quick example program and pasted it here:
   http://dpaste.dzfl.pl/f1699d07

 Hope that helps you out.


OK, I think I am working this out.  Let's say I want to write the 
data to a gzipped file that will be inter-operable with gzip at 
the command line. I think
I do something like the following:

   auto mem = new MemoryStream();
   mem.writeExact(foo.ptr, foo.length * A.sizeof);

   auto gzdat = new Compress(9, HeaderFormat.gzip);
   gzdat.compress(mem.data);

But then, what do I do with gzdat?  Do I try to write it to a 
file like this:

   auto fout = new std.stream.File("data.bin.gz", FileMode.OutNew);
   fout.writeExact(gzdat.ptr, gzdat.sizeof);

Something is wrong though.  Hmm ...

TJB
Sep 25 2012
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 25-Sep-12 00:10, TJB wrote:

 Hello,

 I am trying to save some data to compressed binary files.  I have my
 data in a struct, which I can write to a binary file just fine:

 align(1) struct TradesBin {


Just a note: recently compiler changed so this align(1) have no effect 
on the fields alignment only on a struct as a whole.
So instead
align(1):
inside of struct should probably work.

    int ttim;
    int prc;
    int siz;
    short g127;
    short corr;
    char[2] cond;
    char ex;
 }

 auto fout = new BufferedFile(outfile, FileMode.Out);
 fout.writeExact(&bin, TradesBin.sizeof);

 Where bin is of type TradesBin, filled with data.  The question is how
 do I now do this with zlib to get compression? Suggestions and help are
 mightily appreciated!

 TJB



-- 
Dmitry Olshansky
Sep 24 2012