www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - buffer to struct type conversion...TArrayStream?

reply Charles Hixson via Digitalmars-d-learn writes:
I've read a chunk of data into a buffer and want to convert it into a 
struct.  The reading routine is in a class that doesn't know about the 
struct, but the size should be exactly the same.  (I.e., I want to use 
the converse procedure to write it.)

Is there a better way to do this than using TArrayStream?

The idea here is to have a file of fixed length records (rather like 
Fortran binary records, except that there's a header record which is a 
different size and which specifies various things about the file).  The 
class (well, struct) that handles the fixed length records only knows 
what the record size is, and a couple of other quite general things.  
The class which uses it holds each record in a fixed length struct with 
no indirections.  So I thought I could just cast the buffer to the 
struct...but this doesn't work.  Every straightforward way I've tried of 
doing it yields:
Error: cannot cast from Node_ to ubyte[]
or something reasonably analogous.  The current version of the 
(non-working) code is:
         ubyte    buf[];
         auto    len    =    btFile.read(nodeId, buf);
         assert    (len == n.self.sizeof);


         n.self    =    to!(Node.Node_)(buf);
         //    TODO    write the code
which yields the error:
Error: template instance std.conv.to!(Node_).to!(ubyte[]) error 
instantiating

Node_ is (approximately, I've renamed aliased values to their base value):
         struct    Node_
         {  ulong    idvalue;
             ulong    keyvalue;
             int        eLen;
             Entry   e[23];
         }
and Entry is (approximately):
struct    Entry
{  ulong    key;
     ulong    d;
     ulong    d2;
}
Mar 19 2015
next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 19 Mar 2015 10:47:05 -0700, Charles Hixson via Digitalmars-d-learn
wrote:

turn it 90 degrees. ;-)

  auto cvt =3D cast(Node_*)buf.ptr;
  n =3D cvt[0];
=
Mar 19 2015
parent Charles Hixson via Digitalmars-d-learn writes:
On 03/19/2015 11:18 AM, ketmar via Digitalmars-d-learn wrote:

 On Thu, 19 Mar 2015 10:47:05 -0700, Charles Hixson via Digitalmars-d-learn
 wrote:

 turn it 90 degrees. ;-)

    auto cvt = cast(Node_*)buf.ptr;
    n = cvt[0];


Whee! Thanks, I don't think I *ever* would have thought of that.  I got 
as far as getting a pointer to buf, but then couldn't figure out how to 
turn the Node_* into a Node.  (Obvious now that you showed me, and I 
even think I've seen that trick done before.)
Mar 19 2015
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 19 March 2015 at 17:48:00 UTC, Charles Hixson wrote:

 I've read a chunk of data into a buffer and want to convert it 
 into a struct.  The reading routine is in a class that doesn't 
 know about the struct, but the size should be exactly the same.
  (I.e., I want to use the converse procedure to write it.)

 Is there a better way to do this than using TArrayStream?

 The idea here is to have a file of fixed length records (rather 
 like Fortran binary records, except that there's a header 
 record which is a different size and which specifies various 
 things about the file).  The class (well, struct) that handles 
 the fixed length records only knows what the record size is, 
 and a couple of other quite general things.  The class which 
 uses it holds each record in a fixed length struct with no 
 indirections.  So I thought I could just cast the buffer to the 
 struct...but this doesn't work.  Every straightforward way I've 
 tried of doing it yields:
 Error: cannot cast from Node_ to ubyte[]
 or something reasonably analogous.  The current version of the 
 (non-working) code is:
         ubyte    buf[];


Please use D-style array declarations, the syntax you're using is 
deprecated:
ubyte[] buf;


         auto    len    =    btFile.read(nodeId, buf);
         assert    (len == n.self.sizeof);


         n.self    =    to!(Node.Node_)(buf);
         //    TODO    write the code
 which yields the error:
 Error: template instance std.conv.to!(Node_).to!(ubyte[]) error 
 instantiating


If you want to reinterpret a byte array as another type, there 
are three ways:

1) Using a cast (unsafe):
    n.self = *cast(Node.Node_*)buf.ptr;

2) With a union:
    union {
        ubyte[Node.Node_.sizeof] asBytes;
        Node.Node_ asNode;
    } buf;
    auto len = btFile.read(nodeId, buf.asBytes[]);
    assert(len == n.self.sizeof);
    n.self = buf.asNode

3) Using std.bitmap.peek(), which also supports conversion 
between big- and little-endian:
    import std.bitmap;
    n.self = buf.peek!(Node.Node_, Endian.bigEndian);

(The examples are untested, it's possible you'll need to make 
some adjustments to make them work.)

You should probably use option 3). It is safe, because it checks 
that the buffer has the right size, and it also allows you to 
specify the endian-ness (many file formats have a standardized 
endian-ness).

While you're at it, you can also try std.bitmanip.read(). It can 
be applied to any range, so you can probably do something like 
this (also untested):

     auto input = btFile.byChunk(4096).joiner;
     while(!input.empty) {
         auto node = input.read!(Node.Node_, Endian.bigEndian);
         // use `node`
     }
Mar 19 2015
next sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 03/19/2015 11:42 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
<schuetzm gmx.net>" wrote:


 Please use D-style array declarations, the syntax you're using is
 deprecated:


I fin the following compiler switches useful:

   -de            show use of deprecated features as errors (halt 
compilation)

   -w             warnings as errors (compilation will halt)

Ali
Mar 19 2015
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 19 March 2015 at 18:42:03 UTC, Marc Schütz wrote:

 3) Using std.bitmap.peek(), which also supports conversion 
 between big- and little-endian:
    import std.bitmap;
    n.self = buf.peek!(Node.Node_, Endian.bigEndian);

 (The examples are untested, it's possible you'll need to make 
 some adjustments to make them work.)

 You should probably use option 3). It is safe, because it 
 checks that the buffer has the right size, and it also allows 
 you to specify the endian-ness (many file formats have a 
 standardized endian-ness).

 While you're at it, you can also try std.bitmanip.read(). It 
 can be applied to any range, so you can probably do something 
 like this (also untested):

     auto input = btFile.byChunk(4096).joiner;
     while(!input.empty) {
         auto node = input.read!(Node.Node_, Endian.bigEndian);
         // use `node`
     }


Urgh... it seems `peek()` and `read()` only work with numerical 
types :-( Is this intentional? It would be quite useful for 
arbitrary types, IMO, even if care must be taken with regard to 
pointers.
Mar 19 2015
parent reply Charles Hixson via Digitalmars-d-learn writes:
On 03/19/2015 12:05 PM, via Digitalmars-d-learn wrote:

 On Thursday, 19 March 2015 at 18:42:03 UTC, Marc Schütz wrote:

 3) Using std.bitmap.peek(), which also supports conversion between 
 big- and little-endian:
    import std.bitmap;
    n.self = buf.peek!(Node.Node_, Endian.bigEndian);

 (The examples are untested, it's possible you'll need to make some 
 adjustments to make them work.)

 You should probably use option 3). It is safe, because it checks that 
 the buffer has the right size, and it also allows you to specify the 
 endian-ness (many file formats have a standardized endian-ness).

 While you're at it, you can also try std.bitmanip.read(). It can be 
 applied to any range, so you can probably do something like this 
 (also untested):

     auto input = btFile.byChunk(4096).joiner;
     while(!input.empty) {
         auto node = input.read!(Node.Node_, Endian.bigEndian);
         // use `node`
     }


 Urgh... it seems `peek()` and `read()` only work with numerical types 
 :-( Is this intentional? It would be quite useful for arbitrary types, 
 IMO, even if care must be taken with regard to pointers.


Note:  Thanks for the prior comment about where to place the array 
markers.  I keep forgetting.

Umnf...I don't plan on needing conversion between Endian versions, but I 
guess that's important.  But more significant if it only handles numeric 
types I wonder about handling arrays of structs, even if those structs 
*are* composed entirely of numerical types. Being secure against a 
future need to handle Endian variations would be good, but not enough to 
both increase the complexity that way and deal with possible unexpected 
errors.

P.S.:  Looking at the std.bitmanip documentation causes me to feel that 
the restriction to integral types was intentional, though admittedly it 
doesn't seem to be stated explicitly, but all of the examples seem to be 
simple arrays.  I'll grant that this could be just to keep things 
simple.  And the wording of the docs causes me to think it would 
probably only work for integral values, as in not even floats.  This is 
reasonable if you are thinking of it as a set of routines for bit 
manipulation.

At all events, I think that it involves excessive overhead (in code 
verbosity) and that I'ld likely need to use a loop to read the array 
within the struct.  A simple bit copy is much more straightforwards (I 
was already checking that the read was the correct length, though I 
haven't decided what to do if that ever fails.  Currently it's an assert 
statement, but this clearly needs to be changed to either an enforce 
statement or to a thrown exception...but I haven't yet thought of a 
valid case where the block would be an incorrect length.
Mar 19 2015
parent "Kagamin" <spam here.lot> writes:
If you don't want to run into alignment issues, you shouldn't 
cast the buffer directly: http://dpaste.dzfl.pl/b522f911871a
Mar 20 2015