www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - structs, templates, alias

reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
Hello.

I have a bunch of disparate binary record types that I need to be able 
to read and write to/from a binary stream. Currently, the way I do this 
is just by drumming out a struct declaration and relying on the tupleof 
property to specify how to read/write.

Ex:

struct Rec0{
   ushort index1;
   ushort index2;
   ushort index3;
   ushort junk;
   double value;
}

Rec rec;
// blit knows!
blit(rec, x"01 00 12 00 05 00 00 00 7F F8 00 00 00 00 00 00");

It works really well for simple cases. However, there are some records 
that have really complicated substructures.

struct Rec1{
   ushort index;
   ushort options;
   ushort nparams;
   Parameter[] params;
}
struct Parameter{
   ParamType pt;
   union{
     STRING str;
     CV param;
   }
}

etc etc

for params, or any substructure dependent on a length field, if the 
length field immediately precedes the structure, I can wrap the two in a 
struct which my read/write routines can treat specially as the next 
contiguous field. Well and good.

What's really annoying is when the length field and the array/structure 
are separated by one or more fields, eg.

struct Rec2{
   ushort index;
   ushort nparams;
   ushort options;
   Parameter[] params;
}

At the moment, I just special case these records. However, I would very 
much like to annotate their fields somehow so that the io routines can 
treat them like any other record. One thing I thought about was using 
templates to express this:

struct Rec1{
   ushort index;
   ushort options;
   Parameters!() params;
}

struct Rec2{
   ushort index;
   ushort nparams;
   ushort options;
   Parameters!(nparams) params;
}

// blit knows it can call member blit on this struct
// one thing I like about this is that it always knows
// about the length field, and can automatically modify
// it when you do something with the parameter array.
struct Parameters(alias k = __dflt__){
   static if(k.stringof == "__dflt__") ushort len;
   else alias k len;

   Parameter[] _params;
   int blit(byte[] data){
     int pos = 0;
     static if(k.stringof == "__dflt__") pos = read(len,data);
     //else assume len has already been read
     params.length = len;
     ...
     // read params
   }
}

That doesn't work, though, as DMD complains about inaccessible frames or 
something. I suppose it probably shouldn't work.

Does anyone have any ideas regarding this problem?
Apr 25 2010
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 25/04/10 16:04, Ellery Newcomer wrote:

 Does anyone have any ideas regarding this problem?


I haven't looked into this fully, but for the "the length field and the 
array/structure are separated by one or more fields" issue, something 
like this should help:
----
foreach( m; MyStruct.tupleof )
{
   static if( is( typeof(m) x : x[] ) )
   {
     // This member is an array, so read the size_t length first, then
     // the elements
   }
   else
   {
     // Treat normally.
   }
}
----

I don't think there's a way of finding if a member is part of an 
anonymous union, the only way around this seems to be to use a named 
union so it is treated as one field.
Apr 25 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 04/25/2010 12:26 PM, Robert Clipsham wrote:

 On 25/04/10 16:04, Ellery Newcomer wrote:

 Does anyone have any ideas regarding this problem?


 I haven't looked into this fully, but for the "the length field and the
 array/structure are separated by one or more fields" issue, something
 like this should help:
 ----
 foreach( m; MyStruct.tupleof )
 {
 static if( is( typeof(m) x : x[] ) )
 {
 // This member is an array, so read the size_t length first, then
 // the elements
 }
 else
 {
 // Treat normally.
 }
 }
 ----


Yeah, that's about what I do. The trouble is getting blit to know which 
field is the length field. I suppose you could pass an index into the 
tuple to the substructure. That still wouldn't fix the substructure 
being able to modify the field length.


 I don't think there's a way of finding if a member is part of an
 anonymous union, the only way around this seems to be to use a named
 union so it is treated as one field.


Well, I assume anything with a union has to be special cased anyways, 
especially when the union has fields of different sizes. Everything's 
always written little endian, and I suspect treating the union as a 
single field would cause problems on mac, etc.
Apr 25 2010
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 25/04/10 19:15, Ellery Newcomer wrote:

 Yeah, that's about what I do. The trouble is getting blit to know which
 field is the length field. I suppose you could pass an index into the
 tuple to the substructure. That still wouldn't fix the substructure
 being able to modify the field length.


I don't know how the input data is formatted, but an array is always in 
the form:
----
struct
{
   size_t length;
   void* ptr;
}
----
So it shouldn't be a problem knowing what offset to write the 
length/elements at.
Apr 25 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 04/25/2010 02:10 PM, Robert Clipsham wrote:

 On 25/04/10 19:15, Ellery Newcomer wrote:

 Yeah, that's about what I do. The trouble is getting blit to know which
 field is the length field. I suppose you could pass an index into the
 tuple to the substructure. That still wouldn't fix the substructure
 being able to modify the field length.


 I don't know how the input data is formatted, but an array is always in
 the form:
 ----
 struct
 {
 size_t length;
 void* ptr;
 }
 ----
 So it shouldn't be a problem knowing what offset to write the
 length/elements at.


Hmm. Either I'm not understanding you or I didn't explain something clearly.

something like this

struct Rec2{
   ushort index;
   ushort nparams;
   ushort options;
   ushort[] params; // this has nparams elements
}

Rec2 rec2 = {index:1, nparams:2, options:~0, params:[4,5]};

would translate to something like

x"01 00 02 00 FF FF 04 00 05 00"

only the elements part of params gets written out, the length part doesn't.
Apr 25 2010
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 25/04/10 20:32, Ellery Newcomer wrote:

 Hmm. Either I'm not understanding you or I didn't explain something
 clearly.

 something like this

 struct Rec2{
 ushort index;
 ushort nparams;
 ushort options;
 ushort[] params; // this has nparams elements
 }

 Rec2 rec2 = {index:1, nparams:2, options:~0, params:[4,5]};

 would translate to something like

 x"01 00 02 00 FF FF 04 00 05 00"

 only the elements part of params gets written out, the length part doesn't.


Ah, I didn't know this. In which case you should be able to store 
nparams for use later when the array is encountered, then parse as 
appropriate?
----
static nparams = 0;
foreach( m; MyStruct.tupleof )
{
   static if( is( typeof(m) x : x[] ) )
   {
     parseParams(nparams);
   }
   else
   {
     static if( MyStruct.tupleof[i].stringof[$-7 $] == "nparams" )
       nparams = m;
   }
}
----
Apr 25 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 04/25/2010 02:50 PM, Robert Clipsham wrote:

 On 25/04/10 20:32, Ellery Newcomer wrote:

 Hmm. Either I'm not understanding you or I didn't explain something
 clearly.

 something like this

 struct Rec2{
 ushort index;
 ushort nparams;
 ushort options;
 ushort[] params; // this has nparams elements
 }

 Rec2 rec2 = {index:1, nparams:2, options:~0, params:[4,5]};

 would translate to something like

 x"01 00 02 00 FF FF 04 00 05 00"

 only the elements part of params gets written out, the length part
 doesn't.


 Ah, I didn't know this. In which case you should be able to store
 nparams for use later when the array is encountered, then parse as
 appropriate?


Yeah. The trick is generalizing it. I have about a bazillion different 
record types, and I want my read/write routines to operate only on the 
type information provided by tupleof as much as possible, so the 
structure of each struct documents what it looks like in a binary form. 
Something like the following could work

struct Rec2{
   ushort index;
   ushort nparams;
   ushort options;
   NoLength!(Parameter[], "nparams") params;
}
...
foreach(i,m; rec2.tupleof){
   static if(isNoLength!(typeof(m))){
     auto len = lenField!(rec2, typeof(m));
     ...
   }else{
     ...
   }
}

The other thing is, once I've read something in, I'm probably going to 
want to manipulate it. As above, to maintain consistency, whenever I do

rec2.params ~= p;

I'd have to make sure to do

rec2.nparams ++;

I'd like to have the second statement come for free (although I suppose 
I should just ignore nparams for anything but reading in and use 
params.length instead), but I haven't come up with any way to get 
NoLength to see nparams.

I bet this would be easier in D2.
Apr 25 2010
parent Robert Clipsham <robert octarineparrot.com> writes:
On 25/04/10 21:21, Ellery Newcomer wrote:

 struct Rec2{
 ushort index;
 ushort nparams;
 ushort options;
 NoLength!(Parameter[], "nparams") params;
 }
 ...
 foreach(i,m; rec2.tupleof){
 static if(isNoLength!(typeof(m))){
 auto len = lenField!(rec2, typeof(m));
 ...
 }else{
 ...
 }
 }

 The other thing is, once I've read something in, I'm probably going to
 want to manipulate it. As above, to maintain consistency, whenever I do

 rec2.params ~= p;

 I'd have to make sure to do

 rec2.nparams ++;

 I'd like to have the second statement come for free (although I suppose
 I should just ignore nparams for anything but reading in and use
 params.length instead), but I haven't come up with any way to get
 NoLength to see nparams.

 I bet this would be easier in D2.


Seems to me it'd be easier to change how the records are laid out so you 
don't have to specify this, I'm guessing you already have the data so 
you can't do this though. If that is the case, perhaps you could have 
something like:

struct Rec2
{
   ushort index;
   ushort nparams;
   ushort options;
   Parameter[] params;
   Meta[] meta;
}

Then ignore the meta field when reading/writing, but use it to store 
extra info such as which field holds the offset for nparams? Another 
possibility would be:

struct Rec2
{
   ushort index;
   ushort nparams;
   ushort options;
   Parameter[] params;
   static size_t offset()
   {
     return nparams.offsetof;
   }
}

You could then call this function to get the offset, and read the length 
from there. As for updating nparams, all you need to do is make sure 
that when you write the struct you make sure you do:

Rec2 rec;
...
// Append to params
...
// Set nparams to the actual length of params.
*(&rec + Rec.offset) = rec.params.length;
Apr 25 2010