www.digitalmars.com         C & C++   DMDScript  

D - Structs & network programming.

reply "Chris Paulson-Ellis" <chrispe tenornetworks.com> writes:
Hi,

In network programming (and many other applications), you often want to map
a layout onto an area of memory, allowing you to access protocol fields in a
packet buffer, for example. In C you cannot portably do this by casting to a
struct because of alignment & packing and because you cannot fix the length
of an integer (to 3 octets, say). Messing with compiler pragmas is only a
partial solution and is not portable, by definition. The problem is usually
solved by using pre-processor macros to copy octects one by one between the
fields of the packet and variables. This is ugly and non-optimal without
custom macros for every structure/field/compiler/processor.

It would be nice if D had support for this common problem. Perhaps a special
type of struct which is always packed and for which the compiler generates
the neccessary code for accessing the fields that have an non-processor
native length or which straddle word boundries that the (non-x86) processor
cannot natively support.

Chris.
Sep 24 2001
parent reply Russell Borogove <kaleja estarcion.com> writes:
Chris Paulson-Ellis wrote:

 In network programming (and many other applications), you often want to map
 a layout onto an area of memory, allowing you to access protocol fields in a
 packet buffer, for example. In C you cannot portably do this by casting to a
 struct because of alignment & packing and because you cannot fix the length
 of an integer (to 3 octets, say). [...snip...]
 
 It would be nice if D had support for this common problem. Perhaps a special
 type of struct which is always packed and for which the compiler generates
 the neccessary code for accessing the fields that have an non-processor
 native length or which straddle word boundries that the (non-x86) processor
 cannot natively support.


The guaranteed alignment problem is covered in D. See:

 http://www.digitalmars.com/d/declaration.html (struct declarations)
 http://www.digitalmars.com/d/attribute.html (align attribute)

It's unclear at this time whether a D compiler is expected to 
generate code to access words with nonstandard alignments (on 
x86, I believe this is done automatically, but on most modern
RISC CPUs, accessing a word or long on an odd address yields 
an exception; since Walter's only working on the x86 compiler 
at this time, he can choose not to choose). 

There's no D-specified solution to dealing with things like 
3-octet/24-bit ints. You'll still have to bit-twiddle for things
like this.

-Russell B
Sep 24 2001
next sibling parent reply Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Russell Borogove wrote:


 There's no D-specified solution to dealing with things like
 3-octet/24-bit ints. You'll still have to bit-twiddle for things
 like this.


A nice, portable (but not performance-optimal) programming trick for this would
be
to use the struct as the binary layout and then implement a simple wrapper class
that did the bit twiddling for you:

class FooPacket
{
   struct FooPacketLayout
   {
       ....
    } binaryLayout;

public:
   int GetParamX() { binaryLayout.....bit twiddling }
};
Sep 24 2001
parent reply Axel Kittenberger <axel dtone.org> writes:
Russ Lewis wrote:


 Russell Borogove wrote:
 

 There's no D-specified solution to dealing with things like
 3-octet/24-bit ints. You'll still have to bit-twiddle for things
 like this.


 
 A nice, portable (but not performance-optimal) programming trick for this
 would be to use the struct as the binary layout and then implement a
 simple wrapper class that did the bit twiddling for you:
 
 class FooPacket
 {
    struct FooPacketLayout
    {
        ....
     } binaryLayout;
 
 public:
    int GetParamX() { binaryLayout.....bit twiddling }
 };
 
 


See it? What have I told what the output of seperated class and struct 
elements in the class is?

Why use a struct inside the class? Why not make the contents fields of the 
class itself?

Im my opinion seperated class and struct elements only tends to confuse 
people, and quickly results into suboptimal results during development.

- Axel
Sep 24 2001
next sibling parent "Chris Paulson-Ellis" <chrispe tenornetworks.com> writes:
There are many ways of encapsulating the bit twiddling. In C++ I've
previously done obscure things with templates and even overloading of the
specified address versions of new & delete! These solutions are even more
obscure than using C pre-processor text manipulation. All these things are
working around deficiencies in the language's ability to abstract a memory
region with an externally determined layout. It is such a common problem
that the language ought to support it (IMHO).

Chris.

"Axel Kittenberger" <axel dtone.org> wrote in message
news:9op7he$j2$1 digitaldaemon.com...

 Russ Lewis wrote:

 [snip]
 A nice, portable (but not performance-optimal) programming trick for




this

 would be to use the struct as the binary layout and then implement a
 simple wrapper class that did the bit twiddling for you:

 class FooPacket
 {
    struct FooPacketLayout
    {
        ....
     } binaryLayout;

 public:
    int GetParamX() { binaryLayout.....bit twiddling }
 };


 See it? What have I told what the output of seperated class and struct
 elements in the class is?

 Why use a struct inside the class? Why not make the contents fields of the
 class itself?

 Im my opinion seperated class and struct elements only tends to confuse
 people, and quickly results into suboptimal results during development.
Sep 25 2001
prev sibling parent Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Axel Kittenberger wrote:


 See it? What have I told what the output of seperated class and struct
 elements in the class is?

 Why use a struct inside the class? Why not make the contents fields of the
 class itself?


Are you were arguing that this is an example why D should integrate structs and
classes, or are you unclear about the code?  Sorry, I didn't quite understand.

If it's the former, I do agree that it would be nice to be able to integrate
structs and classes somehow...though I don't know exactly what it should be.
IMHO, having a struct at the top of the class that defines the binary layout
isn't *too* bad, though it's certainly not great.
Sep 25 2001
prev sibling parent reply "Chris Paulson-Ellis" <chrispe tenornetworks.com> writes:
Ahh - align(1){fields...} is good, although I'm not sure of the value of the
colon syntax...

attribute:
    declaration;
    declaration;

... given the existance of the scoped syntax...

attribute {
    declaration;
    declaration;
}

Especially as the colon could easily be missed (or mistakely assumed to be
present) by the fast scanning eye of a fatigued programmer.

It would be well worth pinning down the behavior of misaligned word accesses
in the language spec before someone comes up with the first non-x86
compiler. I take it no-one has a valuable use for those bus errors? :-)

The nice-to-have 24-bit ints (or whatever) would be the icing on the cake...
and debate is likely to be side tracked into the "can we specify a range in
an int typedef?" question.

Chris.

"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3BAF78FD.82DBF71C estarcion.com...

 [snip]
 The guaranteed alignment problem is covered in D. See:

  http://www.digitalmars.com/d/declaration.html (struct declarations)
  http://www.digitalmars.com/d/attribute.html (align attribute)

 It's unclear at this time whether a D compiler is expected to
 generate code to access words with nonstandard alignments (on
 x86, I believe this is done automatically, but on most modern
 RISC CPUs, accessing a word or long on an odd address yields
 an exception; since Walter's only working on the x86 compiler
 at this time, he can choose not to choose).

 There's no D-specified solution to dealing with things like
 3-octet/24-bit ints. You'll still have to bit-twiddle for things
 like this.

 -Russell B
Sep 25 2001
parent reply Russell Borogove <kaleja estarcion.com> writes:
Chris Paulson-Ellis wrote:

 
 Ahh - align(1){fields...} is good, although I'm not sure of the value of the
 colon syntax...


The value here I think is for "orthogonality" with public: and private:
in C++ class declarations, though I'm not sure I accept that as a good 
thing. Ditto some other attributes.


 It would be well worth pinning down the behavior of misaligned word accesses
 in the language spec before someone comes up with the first non-x86
 compiler. I take it no-one has a valuable use for those bus errors? :-)


The trouble is, there's rabid disagreement as to what the "correct"
behavior of misaligned access should be. 

If you define it as jumping through whatever hoops are required to 
read the data as if it were aligned, you get yelled at for the hidden 
performance hit by RISC[1] zealots porting x86 code. They would 
rather that the program crash, so they can rewrite the offending 
portions to execute more efficiently. This is the valuable use 
for those bus errors, weirdly enough.

If you define it as crashing on all platforms, you get yelled at by 
x86 users porting C code. They would rather that it read the data 
with a one-cycle hit, so they can read, for example, existing file 
formats that happen to include non-aligned header data, without 
jumping through hoops. 

If you define it as crashing on RISC platforms and working on x86, 
you get yelled at by people who need to get their port to a RISC 
platform done ASAP and don't care about performance. On the bright 
side, people coming from a C background won't be surprised by this. 

Thus, in C, under the C philosophy of "it works the way the machine 
says it works", well, it works the way the machine says it works.

-Russell B

[1] Yeah, RISC is a bit of a misnomer here, but it's shorter than 
"every processor family that I can think of developed since the 
introduction of the 68000".
Sep 25 2001
parent reply Axel Kittenberger <axel dtone.org> writes:
I agree thats the same in many language decissions, you just can't make it 
100%. It's the same bad vs. bad that people discussed about the throws 
statement, about brackets, etc.


 [1] Yeah, RISC is a bit of a misnomer here, but it's shorter than
 "every processor family that I can think of developed since the
 introduction of the 68000".


As far I know the new powerpc beeing marketed as a RISC processor is now 
again also able to read misaligned data.
Sep 25 2001
parent reply "Chris Paulson-Ellis" <chrispe tenornetworks.com> writes:
"Axel Kittenberger" <axel dtone.org> wrote in message
news:9orr53$1gtm$1 digitaldaemon.com...

 I agree thats the same in many language decissions, you just can't make it
 100%. It's the same bad vs. bad that people discussed about the throws
 statement, about brackets, etc.


I think that the compiler should generate the neccessary code. The
programmer has documented that there may be a performance hit by putting
align(1){...} in the source. Perhaps another attribute is needed to specify
whether you want mis-aligned accesses to blow up or not. Or perhaps an extra
'argument' to the align attribute.

Chris.
Sep 26 2001
parent Russell Borogove <kaleja estarcion.com> writes:
Chris Paulson-Ellis wrote:

 I think that the compiler should generate the neccessary code. The
 programmer has documented that there may be a performance hit by putting
 align(1){...} in the source. Perhaps another attribute is needed to specify
 whether you want mis-aligned accesses to blow up or not. Or perhaps an extra
 'argument' to the align attribute.


Maybe I only think so for historical reasons, but it seems like if
you're going to make it optional it should be a compiler switch or a 
pragma-type thing. It seems weird at the language level to say "please
don't crash" -- maybe that's an argument in favor of such accesses 
always working, even if slower. 

If misaligned reads are defined as safe, there should be a note to 
non-x86 implementors recommending that possibly-misaligned reads
be emulated via function call or extra instructions inline, rather 
than by trapping the machine exception. If it's defined as safe, 
it's going to get used fairly often. In reading misaligned data 
from a file, for example, I can accept the overhead of read 
emulation, which will be swamped by file access time, but can't 
accept the overhead of a large number of exceptions.

The more I think about it, on modern processors and in modern 
applications, there are a lot of performance issues that aren't 
language defined -- cache sizes and alignments and the like -- and 
you generally want the code to Just Work even if it's non-optimal. 
Perhaps potentially-misaligned reads can generate an optional 
compiler warning, but always work.

-RB
Sep 26 2001