www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Read/write structs in streams (Tango, and Strings)

reply Robert Kosek <robert.kosek thewickedflea.com> writes:
Hello again folks,

My background in file-level read/write operations comes from Delphi and 
its handling of types.  So for reading and writing strings they always 
had to be written and read in a specific way, and a length integer 
prepended to the string.  For this reason you can't have a string, 
unless limited in length, in a record (Pascal's version of struct).

Can D read and write structs with dynamically sized strings to and from 
streams?

Here's an example of a struct I'd use:

 struct MyStruct {
   int val1, val2;
   char[] id;
   uint somesuchvalue;
   // Etc...
 }


Can D handle writing a variably-sized struct like this to a stream 
without any other steps?  Or, should I have a "write" method of the 
struct to write it to a stream?  If so, then how do I write an array of 
these structs efficiently?  (Like a batch read, ie: 
Stream.Read(myArrayOfStructs, theirLength))

I guess this is just a general question about how Tango handles streams 
and structs when writing, as well as any D specific quirks or advantages 
there may be.

I'm thinking of making a simple archive format, because I don't like the 
source control packages out there and was thinking of making my own.

Thanks for the answers in advance!

Regards,
Robert
Oct 20 2008
parent reply Robert Kosek <robert.kosek thewickedflea.com> writes:
Robert Kosek Wrote:


 Hello again folks,


I know I asked a really basic question, but I haven't found a real answer
either in the Tango documentation or the D language documentation either.  This
makes it really tough to know how I can save structures to a stream for later
use.

Can someone please tell me if I can read/write structures with dynamic length
strings from/to a stream?  And if not, then how can I read and write a string
to the stream (without the helper stream class)?

Robert
Oct 22 2008
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
"Robert Kosek"  wrote

 Robert Kosek Wrote:


 Hello again folks,


 I know I asked a really basic question, but I haven't found a real answer 
 either in the Tango documentation or the D language documentation either. 
 This makes it really tough to know how I can save structures to a stream 
 for later use.

 Can someone please tell me if I can read/write structures with dynamic 
 length strings from/to a stream?  And if not, then how can I read and 
 write a string to the stream (without the helper stream class)?


I am not completely knowledgable in the stream classes of Tango that do 
serialization, but I don't think there's anything to do what you want.  It 
would be difficult for the stream class to know the exact layout of your 
struct, so how would it know when to output strings?  I think DataStream is 
what you are looking for, but it will only output a block of data.

Again, not completely knowledgable.  You might try asking on the Tango 
forums to get the attention of some of the Tango devs that don't pay 
attention to the NG.

-Steve
Oct 22 2008
parent reply "Bill Baxter" <wbaxter gmail.com> writes:
On Wed, Oct 22, 2008 at 11:24 PM, Steven Schveighoffer
<schveiguy yahoo.com> wrote:

 "Robert Kosek"  wrote

 Robert Kosek Wrote:


 Hello again folks,


 I know I asked a really basic question, but I haven't found a real answer
 either in the Tango documentation or the D language documentation either.
 This makes it really tough to know how I can save structures to a stream
 for later use.

 Can someone please tell me if I can read/write structures with dynamic
 length strings from/to a stream?  And if not, then how can I read and
 write a string to the stream (without the helper stream class)?


 I am not completely knowledgable in the stream classes of Tango that do
 serialization, but I don't think there's anything to do what you want.  It
 would be difficult for the stream class to know the exact layout of your
 struct, so how would it know when to output strings?  I think DataStream is
 what you are looking for, but it will only output a block of data.

 Again, not completely knowledgable.  You might try asking on the Tango
 forums to get the attention of some of the Tango devs that don't pay
 attention to the NG.


Tom S. was working on a nifty ctfe/mixin serialization module.  But I
think it's been on the back burner for a while.  Aarti_pl has one too
in the doost libs.
Neither is really complete.

And yes you need some kind of library if you want to stream arbitrary
structs which may or may not contain pointers to disk.   And no,
neither Tango nor Phobos has such a library.

--bb
Oct 22 2008
parent reply Robert Kosek <robert.kosek thewickedflea.com> writes:
Bill Baxter wrote:

 On Wed, Oct 22, 2008 at 11:24 PM, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:
 [...] 
 And yes you need some kind of library if you want to stream arbitrary
 structs which may or may not contain pointers to disk.   And no,
 neither Tango nor Phobos has such a library.
 
 --bb



Okay, thanks.  To write a struct to a stream should really be like 
writing a buffer.  You point the writer at the memory and tell it how 
large it is, and it writes the block to the stream.  So it should to be 
able to write a pointer-less struct.

That much is good, I can do string mangling/storage another way, probably.

Thanks for the answers Bill and Steven.

Robert
Oct 22 2008
parent "Bill Baxter" <wbaxter gmail.com> writes:
On Thu, Oct 23, 2008 at 12:11 AM, Robert Kosek
<robert.kosek thewickedflea.com> wrote:

 Bill Baxter wrote:

 On Wed, Oct 22, 2008 at 11:24 PM, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:
 [...] And yes you need some kind of library if you want to stream
 arbitrary
 structs which may or may not contain pointers to disk.   And no,
 neither Tango nor Phobos has such a library.

 --bb



 Okay, thanks.  To write a struct to a stream should really be like writing a
 buffer.  You point the writer at the memory and tell it how large it is, and
 it writes the block to the stream.  So it should to be able to write a
 pointer-less struct.


But if you care about portability you need to worry about things like
endianness and the variable size of D's real, creal, and ireal types.
Also difference of size in size_t from 32 bit to 64 bit platforms.  So
even without pointers there can be more to it than just dumping a
chunk of memory into a stream.


 That much is good, I can do string mangling/storage another way, probably.


Yep, if your needs are fairly simple then it's not that difficult to
come up with something.  There is something like an EndianStream in
Tango.  I think it does byteswapping on ints and floats and things for
you.

--bb
Oct 22 2008