www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Why must butfields sum to a multiple of a byte?

reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
It says for std.bitmanip.bitfields:

"The sum of all bit lengths in one bitfield instantiation must be
exactly 8, 16, 32, or 64."
It has a static assert to verify this. But why does this limitation exist?

I was thinking of cases where I'm wrapping a C++ POD struct which has
bitfields and is packed on a 1-byte boundary. GCC example:

#include <stdio.h>
#pragma pack(1)
struct Foo
{
    unsigned int bits1 : 1;
    unsigned int bits2 : 2;
};

int main()
{
    printf("%d\n", sizeof(Foo));
    return 0;
}

I can't translate this to D using the bitfields mixin since it
requires a multiple of a byte, 3 bits isn't a byte:

import std.stdio;
import std.bitmanip;

align(1) struct Foo
{
    mixin(bitfields!(
        uint, "bits1", 1,
        uint, "bits2", 2
    ));
}

void main() { writeln(Foo.sizeof); }

std\bitmanip.d(151): Error: static assert  "Field widths must sum to
8, 16, 32, or 64"

Is this a language limitation? I guess because 'byte' is the smallest
legal type in D this is what the bitfields mixin uses internally?
Jul 30 2012
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 30 July 2012 at 14:19:56 UTC, Andrej Mitrovic wrote:

 It says for std.bitmanip.bitfields:

 "The sum of all bit lengths in one bitfield instantiation must 
 be
 exactly 8, 16, 32, or 64."
 It has a static assert to verify this. But why does this 
 limitation exist?

 I was thinking of cases where I'm wrapping a C++ POD struct 
 which has
 bitfields and is packed on a 1-byte boundary. GCC example:

 #include <stdio.h>
 #pragma pack(1)
 struct Foo
 {
     unsigned int bits1 : 1;
     unsigned int bits2 : 2;
 };

 int main()
 {
     printf("%d\n", sizeof(Foo));
     return 0;
 }

 I can't translate this to D using the bitfields mixin since it
 requires a multiple of a byte, 3 bits isn't a byte:

 import std.stdio;
 import std.bitmanip;

 align(1) struct Foo
 {
     mixin(bitfields!(
         uint, "bits1", 1,
         uint, "bits2", 2
     ));
 }

 void main() { writeln(Foo.sizeof); }

 std\bitmanip.d(151): Error: static assert  "Field widths must 
 sum to
 8, 16, 32, or 64"

 Is this a language limitation? I guess because 'byte' is the 
 smallest
 legal type in D this is what the bitfields mixin uses 
 internally?


I think you are supposed to explicitly pad the field yourself:

----
align(1) struct Foo
{
    mixin(bitfields!(
        uint, "bits1", 1,
        uint, "bits2", 2,
        uint, "", 5
    ));
}
----
The rationale (I *think*) Is that bitfields wants to know how big 
it should be: While YOU know you want to put it into a "Foo" 
object that is 1 byte big, bitfields has no idea.

Either that, or having the developer provide this by hand made 
the implementation easier enough to justify the burden?

Even then, I think it would have been more convenient to specify 
the total size as a first argument, rather than manually padding 
at the end.

Either way, nothing world shattering for me :D
Jul 30 2012
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, monarch_dodra <monarchdodra gmail.com> wrote:

 I think you are supposed to explicitly pad the field yourself:


You're right, I missed that part of the documentation. :)
Jul 30 2012
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/30/2012 07:19 AM, Andrej Mitrovic wrote:


 I was thinking of cases where I'm wrapping a C++ POD struct which has
 bitfields and is packed on a 1-byte boundary.


It is inherently unportable because unlike D, bitfields in C and C++ 
give too much freedom to the compiler. The compiler can move the fields 
around at will.

Ali
Jul 30 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, Ali =C7ehreli <acehreli yahoo.com> wrote:

 It is inherently unportable because unlike D, bitfields in C and C++
 give too much freedom to the compiler. The compiler can move the fields
 around at will.


I really need to figure out a reliable way to extract this
information, at least for the target platform a library is compiled
on. gccxml gives some information which seems to be somewhat reliable
but according to the authors it's more of a guess than accurate info
(2nd paragraph):
http://www.gccxml.org/pipermail/gccxml/2012-February/001483.html

Still this isn't too shabby, there could be configuration options
where the user could set pack/bitsize/compiler options and similar
stuff which would make a codegenerator more reliable.
Jul 30 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/30/2012 09:44 AM, Andrej Mitrovic wrote:

 On 7/30/12, Ali Çehreli<acehreli yahoo.com>  wrote:

 It is inherently unportable because unlike D, bitfields in C and C++
 give too much freedom to the compiler. The compiler can move the fields
 around at will.


 I really need to figure out a reliable way to extract this
 information, at least for the target platform a library is compiled
 on.


The program should be able to detect the placements of the fields. You 
can expect a certain bit pattern for a given set of values of the bit 
fields, set those values, read them on the D side, detect potential 
inconsistencies, and fail the execution (or hopefully compilation).

Ali
Jul 30 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, Ali =C7ehreli <acehreli yahoo.com> wrote:

 The program should be able to detect the placements of the fields. You
 can expect a certain bit pattern for a given set of values of the bit
 fields, set those values, read them on the D side, detect potential
 inconsistencies, and fail the execution (or hopefully compilation).


Yeah that was exactly my thinking, these things can be automated
although they might slow down the whole codegeneration process a
little bit (not that it matters much since you would rarely have to
regenerate).

Btw, I get an internal std error with this mixin:

import std.bitmanip;
struct Foo
{
    mixin(bitfields!(
        uint, "bits1", 32,
    ));
}

D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76): Error:
shift by 32 is outside the range 0..31

Should I file this?
Jul 30 2012
next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/30/2012 10:15 AM, Andrej Mitrovic wrote:


 import std.bitmanip;
 struct Foo
 {
      mixin(bitfields!(
          uint, "bits1", 32,
      ));
 }

 D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76): Error:
 shift by 32 is outside the range 0..31

 Should I file this?


Yes, it's a bug.

Ali
Jul 30 2012
parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Monday, 30 July 2012 at 17:43:28 UTC, Ali Çehreli wrote:

 On 07/30/2012 10:15 AM, Andrej Mitrovic wrote:


 import std.bitmanip;
 struct Foo
 {
      mixin(bitfields!(
          uint, "bits1", 32,
      ));
 }

 D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76):


 Error:

 shift by 32 is outside the range 0..31

 Should I file this?


 Yes, it's a bug.


  And likely one I'll be working on fairly soon. I've been 
concentrating on the BitArray, but I'll get more of the bitfields 
very very soon.
Jul 30 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:

   And likely one I'll be working on fairly soon. I've been
 concentrating on the BitArray, but I'll get more of the bitfields
 very very soon.


Cool. What about the max limit of 64bits per bitfield instantiation? I
don't suppose this is common in C++ but I wouldn't know..
Jul 30 2012
parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Monday, 30 July 2012 at 23:41:39 UTC, Andrej Mitrovic wrote:

 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:

 And likely one I'll be working on fairly soon. I've been 
 concentrating on the BitArray, but I'll get more of the 
 bitfields very very soon.


 Cool. What about the Max limit of 64bits per bitfield 
 instantiation? I don't suppose this is common in C++ but I 
 wouldn't know..


  The limitation is based on what kind of types you can use. If 
you want a field 1000 bits long, i don't see why not; But 
obviously you can't treat it like an int or bool. (No cheap way 
to make a BigInt here :P), and packing say a struct, it would be 
slow since it has to copy each bit individually, or if it's byte 
aligned, by bytes.

  It assumes the largest type we can currently use which is ulong, 
if cent ever gets properly added, then 128bits will become 
available. If they want to go higher than likely type 'dime' or 
'nickel' or bicent (256bits? :P) can be used once the template is 
modified. Maybe tricent can be 384 and quadcent can be 512.... 
Mmmm :) or it could just be large256, ularge256.

  As for the 'why', is that all of the bitfields work by assuming 
bit-shifting and low level binary operators to do the job. So, 2 
for 4 ints would be

int value;
int a()  property {
   return value & 0x7;
}
void a(int v)  property {
   value &= ~0x7;
   value |= v & 0x7;
}

int b()  property {
   return (value>>4) & 0x7;
}
void a(int v)  property {
   value &= ~(0x7 << 4);
   value |= (v & 0x7) << 4;
}

  That may be flawed but it gives you a basic idea.
Jul 30 2012
next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 00:00:24 UTC, Era Scarecrow wrote:
Corrections:
  So, 2 variables using 4 bit ints would be


 void a(int v)  property {
   value &= ~(0x7 << 4);
   value |= (v & 0x7) << 4;
 }


the second setter should be
  void b(int v)  property {
Jul 30 2012
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:

   It assumes the largest type we can currently use which is ulong


Ah yes, it makes sense now. Thanks for the brain cereal. :p
Jul 30 2012
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 31 July 2012 at 00:44:16 UTC, Andrej Mitrovic wrote:

 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:

   It assumes the largest type we can currently use which is 
 ulong


 Ah yes, it makes sense now. Thanks for the brain cereal. :p


I saw your bug report:
http://d.puremagic.com/issues/show_bug.cgi?id=8474

The bug is only when the field is EXACTLY 32 bits BTW. bitfields 
works quite nice with 33 or whatever. More details in the report.
Jul 31 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:

 The bug is only when the field is EXACTLY 32 bits BTW. bitfields
 works quite nice with 33 or whatever. More details in the report.


Yeah 32 or 64 bits, thanks for changing the title.
Jul 31 2012
parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 15:25:55 UTC, Andrej Mitrovic wrote:

 On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:

 The bug is only when the field is EXACTLY 32 bits BTW. 
 bitfields works quite nice with 33 or whatever. More details 
 in the report.


 Yeah 32 or 64 bits, thanks for changing the title.


  I wonder, is it really a bug? If you are going to have it fill a 
whole size it would fit anyways, why even put it in as a 
bitfield? You could just declare it separately.

  I get the feeling it's not so much a bug as a design feature. If 
you really needed a full size not aligned with whole bytes (or 
padded appropriately) then I could understand, but still...


  And I'm the one that changed the title name. Suddenly I'm 
reminded of south park (movie) and the buttfor.
Jul 31 2012
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:

   I wonder, is it really a bug? If you are going to have it fill a
 whole size it would fit anyways, why even put it in as a
 bitfield? You could just declare it separately.


I don't really know, I'm looking at this from a point of wrapping C++.
I haven't used bitfields myself in my own code.
Jul 31 2012
parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 16:48:37 UTC, Andrej Mitrovic wrote:

 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:

 I wonder, is it really a bug? If you are going to have it fill 
 a whole size it would fit anyways, why even put it in as a 
 bitfield? You could just declare it separately.


 I don't really know, I'm looking at this from a point of 
 wrapping C++. I haven't used bitfields myself in my own code.


  I'd say it's not a bug since C/C++ is free to reorder the fields 
you'd need to tinker with it anyways; HOWEVER if you still need 
to be able to have it then who's to stop you from doing it?

  I think more likely a flag/version or some indicator that you 
didn't make a mistake, such as making them depreciated so it 
complains to you. Kinda like how you can't make assignments in if 
statements or do useless compares, it's an error and helps 
prevent issues that are quite obviously mistakes.
Jul 31 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 07/31/2012 06:57 PM, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 16:48:37 UTC, Andrej Mitrovic wrote:

 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:

 I wonder, is it really a bug? If you are going to have it fill a
 whole size it would fit anyways, why even put it in as a bitfield?
 You could just declare it separately.


 I don't really know, I'm looking at this from a point of wrapping C++.
 I haven't used bitfields myself in my own code.


 I'd say it's not a bug since C/C++ is free to reorder the fields you'd
 need to tinker with it anyways; HOWEVER if you still need to be able to
 have it then who's to stop you from doing it?

 I think more likely a flag/version or some indicator that you didn't
 make a mistake, such as making them depreciated so it complains to you.
 Kinda like how you can't make assignments in if statements or do useless
 compares, it's an error and helps prevent issues that are quite
 obviously mistakes.


This is obviously a mistake in the bitfield implementation. What else
could be concluded from the error message:

std\bitmanip.d(76): Error: shift by 32 is outside the range 0..31

Requesting a 32 bit or 64 bit member on the other hand is not a
mistake, and it is not useless, therefore the analogy breaks down.

(Also IMO, the once-in-a-year wtf that is caused by accidentally
assigning in an if condition does not justify special casing assignment
expressions inside if conditions. Also, what is an useless compare?)
Jul 31 2012
next sibling parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:


 (Also IMO, the once-in-a-year wtf that is caused by 
 accidentally assigning in an if condition does not justify 
 special casing assignment expressions inside if conditions. 
 Also, what is an useless compare?)


  I've noticed in my experience, DMD gives you an error if you do 
a statement that has no effect; IE:

  1 + 2; //statement has no effect
  a == b;//ditto
Jul 31 2012
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 31 July 2012 at 17:17:25 UTC, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 16:59:11 UTC, monarch_dodra wrote:

 Maybe the user needs a 32 bit ulong? This way the ulong only 
 takes 32 bits, but can still be implicitly passed to functions 
 expecting ulongs.


  I would think the bug only showed itself if you did int at 
 32bits and ulong at 64 bits, not ulong at 32bits.


No, the bug shows itself if the first field is 32 bits, 
regardless of (ulong included).

I would add though that requesting a field in bits that is bigger 
than the type of the field should not work (IMO). EG:
--------
struct A
{
     mixin(bitfields!(
           ushort, "a", 24,
           uint,    "",  8
         )
     );
}
--------
I don't see any way how that could make sense...
But it *is* legal in C and C++...
But it does generates warnings...

I think it should static assert in D.

On Tuesday, 31 July 2012 at 17:21:00 UTC, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:


 (Also IMO, the once-in-a-year wtf that is caused by 
 accidentally assigning in an if condition does not justify 
 special casing assignment expressions inside if conditions. 
 Also, what is an useless compare?)


  I've noticed in my experience, DMD gives you an error if you 
 do a statement that has no effect; IE:

  1 + 2; //statement has no effect
  a == b;//ditto


That's part of the standard: Statements that have no effect are 
illegal. This is a good think, IMO. I've seen MANY C++ bugs that 
could have been saved by that.

Regarding the assignment in if. I think it is a good thing. D 
sides with safety. If you *really* want to test assign, you can 
always comma operator it.

--------
void main()
{
   int a = 0, b = 5;
   while(a = --b, a) //YES, I *DO* want to assign!
   {
     write(a);
   }
};
--------I'm sure the compiler will optimize away what it needs.
Jul 31 2012
parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:34:33 UTC, monarch_dodra wrote:

 No, the bug shows itself if the first field is 32 bits, 
 regardless of (ulong included).

 I would add though that requesting a field in bits that is 
 bigger than the type of the field should not work (IMO). EG:
 --------
 struct A
 {
     mixin(bitfields!(
           ushort, "a", 24,
           uint,    "",  8
         )
     );
 }
 --------
 I don't see any way how that could make sense...
 But it *is* legal in C and C++...
 But it does generates warnings...


  Maybe so ushort has extra padding for expansion at some later 
date when they change it to uint?? could put an assert in, but if 
it doesn't break code...


 I think it should static assert in D.


  Glancing over the issue, the [0..31] is a compiler error based 
on bit shifting (not bitfields itself); if the storage type is 
ulong then it shouldn't matter if the first one is a 32bit size 
or not. Unless... Nah, couldn't be... I'll look it over later to 
be sure.


 That's part of the standard: Statements that have no effect are 
 illegal. This is a good think, IMO. I've seen MANY C++ bugs 
 that could have been saved by that.

 Regarding the assignment in if. I think it is a good thing. D 
 sides with safety. If you *really* want to test assign, you can 
 always comma operator it.
Jul 31 2012
prev sibling next sibling parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:


 This is obviously a mistake in the bitfield implementation. 
 What else could be concluded from the error message:

 std\bitmanip.d(76): Error: shift by 32 is outside the range 
 0..31

 Requesting a 32 bit or 64 bit member on the other hand is not a 
 mistake, and it is not useless, therefore the analogy breaks 
 down.


  Well curiously it was easier to fix than I thought (a line for a 
static if, and a modification of the masking)... Was there any 
other bugs that come to mind? Anything of consequence?
Jul 31 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31-Jul-12 22:21, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:


 This is obviously a mistake in the bitfield implementation. What else
 could be concluded from the error message:

 std\bitmanip.d(76): Error: shift by 32 is outside the range 0..31

 Requesting a 32 bit or 64 bit member on the other hand is not a
 mistake, and it is not useless, therefore the analogy breaks down.


   Well curiously it was easier to fix than I thought (a line for a
 static if, and a modification of the masking)... Was there any other
 bugs that come to mind? Anything of consequence?


Great to see things moving. Could you please do a separate pull for 
bitfields it should get merged easier and it seems like a small but 
important bugfix.

-- 
Dmitry Olshansky
Jul 31 2012
next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 20:41:55 UTC, Dmitry Olshansky wrote:

 On 31-Jul-12 22:21, Era Scarecrow wrote:

 Well curiously it was easier to fix than I thought (a line for 
 a static if, and a modification of the masking)... Was there 
 any other bugs that come to mind? Anything of consequence?


 Great to see things moving. Could you please do a separate pull 
 for bitfields it should get merged easier and it seems like a 
 small but important bugfix.


  Guess this means I'll be working on BitArrays a bit later and 
work instead on the bitfields code. What fun... :) I thought I 
had bitfields as separate already, but it's kinda thrown both 
sets of changes in. Once I figure it out I'll get them separated 
and finish work on the bitfields.
Jul 31 2012
prev sibling parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 20:41:55 UTC, Dmitry Olshansky wrote:


 Great to see things moving. Could you please do a separate pull 
 for bitfields it should get merged easier and it seems like a 
 small but important bugfix.


https://github.com/rtcvb32/phobos/commit/620ba57cc0a860245a2bf03f7b7f5d6a1bb58312

  I've updated the next update in my bitfields branch. All 
unittests pass for me.
Aug 01 2012
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 1 August 2012 at 07:24:09 UTC, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 20:41:55 UTC, Dmitry Olshansky 
 wrote:


 Great to see things moving. Could you please do a separate 
 pull for bitfields it should get merged easier and it seems 
 like a small but important bugfix.


 https://github.com/rtcvb32/phobos/commit/620ba57cc0a860245a2bf03f7b7f5d6a1bb58312

  I've updated the next update in my bitfields branch. All 
 unittests pass for me.


I had an (implementation) question for you:
Does the implementation actually require knowing what the size of 
the padding is?

eg:
struct A
{
     int a;
     mixin(bitfields!(
         uint,  "x",    2,
         int,   "y",    3,
         ulong,  "",    3 // <- This line right there
     ));
}

It that highlighted line really mandatory?
I'm fine with having it optional, in case I'd want to have, say, 
a 59 bit padding, but can't the implementation figure it out on 
it's own?
Aug 02 2012
parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 09:03:54 UTC, monarch_dodra wrote:

 I had an (implementation) question for you: Does the 
 implementation actually require knowing what the size of the 
 padding is?

 eg:
 struct A
 {
     int a;
     mixin(bitfields!(
         uint,  "x",    2,
         int,   "y",    3,
         ulong,  "",    3 // <- This line right there
     ));
 }

 It that highlighted line really mandatory?
 I'm fine with having it optional, in case I'd want to have, 
 say, a 59 bit padding, but can't the implementation figure it 
 out on it's own?


  The original code has it set that way, why? Perhaps so you are 
aware and actually have in place where all the bits are assigned 
(even if you aren't using them); Be horrible if you used 
accidently 33 bits and it extended to 64 without telling you 
(Wouldn't it?).

  However, having it fill the size in and ignore the last x bits 
wouldn't be too hard to do, I've been wondering if I should 
remove it.
Aug 02 2012
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 2 August 2012 at 09:14:15 UTC, Era Scarecrow wrote:

 On Thursday, 2 August 2012 at 09:03:54 UTC, monarch_dodra wrote:

 I had an (implementation) question for you: Does the 
 implementation actually require knowing what the size of the 
 padding is?

 eg:
 struct A
 {
    int a;
    mixin(bitfields!(
        uint,  "x",    2,
        int,   "y",    3,
        ulong,  "",    3 // <- This line right there
    ));
 }

 It that highlighted line really mandatory?
 I'm fine with having it optional, in case I'd want to have, 
 say, a 59 bit padding, but can't the implementation figure it 
 out on it's own?


  The original code has it set that way, why? Perhaps so you are 
 aware and actually have in place where all the bits are 
 assigned (even if you aren't using them); Be horrible if you 
 used accidently 33 bits and it extended to 64 without telling 
 you (Wouldn't it?).

  However, having it fill the size in and ignore the last x bits 
 wouldn't be too hard to do, I've been wondering if I should 
 remove it.


Well, I was just trying to figure out the rationale: The most 
obvious one for me being "it is much easier on the 
implementation".

One of the *big* reasons I'm against having a hand chosen 
padding, is that the implementation *should* be able to find out 
what the most efficient padding is on the current machine (could 
be 32 on some, could be 64 on some)

That said, something that could fix the above "problem" could be:
*Bitfields are automatically padded if the final field is not a 
"padding field".
**Padding size is implementation chosen.
*If the final field is a "padding field", then the total size 
must be 8/16/32/64.

EG:
//Case 1
bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
)
//Fine, implementation chosen bitfield size

//Case 2
bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
     ulong, "",   59, //Pad to 64
)
//Fine, imposed 64 bit

//Case 3
bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
     ulong, "",   32, //Pad to 37
)
//ERROR: Padding requests the bitfield to be 37 bits longs

But I'd say that's another development anyways, if we ever decide 
to go this way.
Aug 02 2012
next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 09:26:04 UTC, monarch_dodra wrote:

 Well, I was just trying to figure out the rationale: The most 
 obvious one for me being "it is much easier on the 
 implementation".


  Since the template is recursive and at the end after bit 
counting would know how much it needed, that doesn't seem right; 
far more likely just to be explicit.


 One of the *big* reasons I'm against having a hand chosen 
 padding, is that the implementation *should* be able to find 
 out what the most efficient padding is on the current machine 
 (could be 32 on some, could be 64 on some)


  If your using bitfields, then you are going for space, and to be 
as small as reasonably possible. Especially important for packets 
of information like headers for compression, and making it 
compatible with C/C++'s bitpacking.


 That said, something that could fix the above "problem" could 
 be:
 *Bitfields are automatically padded if the final field is not a 
 "padding field".


  Workable


 **Padding size is implementation chosen.


  I assume you mean by word size (size_t), meaning always 
32/64bit. In that case many applicable cases would go away and be 
useless.


 *If the final field is a "padding field", then the total size 
 must be 8/16/32/64.

 EG:
 //Case 1
 bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
 )
 //Fine, implementation chosen bitfield size


   It would go with a ubyte as is likely obvious, although any of 
the types would work.


 //Case 2
 bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
     ulong, "",   59, //Pad to 64
 )
 //Fine, imposed 64 bit

 //Case 3
 bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
     ulong, "",   32, //Pad to 37
 )
 //ERROR: Padding requests the bitfield to be 37 bits longs

 But I'd say that's another development anyways, if we ever 
 decide to go this way.


  In the end, either explicit in size, or let it round up to the 
size it can accept. If you let it decide, then padding would be 
treated as though it's a variable (but isn't), so...

bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
     ulong, "",   5,
)

  the total size is 9bits, the padding forces it to 16bit 
afterwards. In cases like this it could easily be abused or leave 
it in a confusing state; So likely the padding would have to be 
missing (and assumed) or explicit in size. So

previous one would err, while:

bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1,
//  void, "", 4      //implied by missing, if padded and final 
size not ^2 would statically assert.
)
Aug 02 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/2/12 5:26 AM, monarch_dodra wrote:

 One of the *big* reasons I'm against having a hand chosen padding, is
 that the implementation *should* be able to find out what the most
 efficient padding is on the current machine (could be 32 on some, could
 be 64 on some)


In my neck of the woods they call that "non-portability".

If your code is dependent on the machine's characteristics you use 
version() and whatnot.


Andrei
Aug 02 2012
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 2 August 2012 at 12:38:10 UTC, Andrei Alexandrescu 
wrote:

 On 8/2/12 5:26 AM, monarch_dodra wrote:

 One of the *big* reasons I'm against having a hand chosen 
 padding, is
 that the implementation *should* be able to find out what the 
 most
 efficient padding is on the current machine (could be 32 on 
 some, could
 be 64 on some)


 In my neck of the woods they call that "non-portability".

 If your code is dependent on the machine's characteristics you 
 use version() and whatnot.


Well, isn't that the entire point: Making your code NOT dependent 
on the machine's characteristics?

By forcing the developer to chose the bitfield size (32 or 64), 
you ARE forcing him to make a choice dependent on the machine's 
characteristics. The developer just knows how he wants to pack 
his bits, not how he wants to pad them. Why should the developer 
be burdened with figuring out what the optimal size of his 
bitfield should be?

By leaving the field blank, *that* guarantees portability.
Aug 02 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/2/12 9:48 AM, monarch_dodra wrote:

 On Thursday, 2 August 2012 at 12:38:10 UTC, Andrei Alexandrescu wrote:

 On 8/2/12 5:26 AM, monarch_dodra wrote:

 One of the *big* reasons I'm against having a hand chosen padding, is
 that the implementation *should* be able to find out what the most
 efficient padding is on the current machine (could be 32 on some, could
 be 64 on some)


 In my neck of the woods they call that "non-portability".

 If your code is dependent on the machine's characteristics you use
 version() and whatnot.


 Well, isn't that the entire point: Making your code NOT dependent on the
 machine's characteristics?

 By forcing the developer to chose the bitfield size (32 or 64), you ARE
 forcing him to make a choice dependent on the machine's characteristics.


I think that's backwards.

Andrei
Aug 02 2012
parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 14:52:58 UTC, Andrei Alexandrescu 
wrote:

 On 8/2/12 9:48 AM, monarch_dodra wrote:

 By forcing the developer to chose the bitfield size (32 or 
 64), you ARE forcing him to make a choice dependent on the 
 machine's characteristics.


 I think that's backwards.


  I think specifying bitfields, you're already going really low 
level; meaning you need explicit and full control. As for 
portability, if the size is specified, it comes down to endinness 
or order of the fields.

Zlib header: http://tools.ietf.org/html/rfc1950

A zlib stream has the following structure: CMF, FLaG

CMF:
     bits 0 to 3  CM      Compression method
     bits 4 to 7  CINFO   Compression info
FLaG:
     bits 0 to 4  FCHECK  (check bits for CMF and FLG)
     bit  5       FDICT   (preset dictionary)
     bits 6 to 7  FLEVEL  (compression level)

This easily becomes:

//assumes small-endian
mixin(bitfields(
     ubyte, "CM", 4,
     ubyte, "CINFO", 4,
     ubyte, "FCHECK", 5,
     ubyte, "FDICT", 1,
     ubyte, "FLEVEL", 2)); //should be exactly 16 bits

  Now if a typo was done (wrong number of bits) or endianness is 
questionable, this becomes completely different as the fields 
could be as much as 3-7 bytes off (24 bits, or worse if it 
assumes 64bits, 56 bits off), or cover a much larger area than it 
should for the bits. This is unacceptable, especially if you need 
to get low level with the code.
Aug 02 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/2/12 5:14 AM, Era Scarecrow wrote:

 On Thursday, 2 August 2012 at 09:03:54 UTC, monarch_dodra wrote:

 I had an (implementation) question for you: Does the implementation
 actually require knowing what the size of the padding is?

 eg:
 struct A
 {
 int a;
 mixin(bitfields!(
 uint, "x", 2,
 int, "y", 3,
 ulong, "", 3 // <- This line right there
 ));
 }

 It that highlighted line really mandatory?
 I'm fine with having it optional, in case I'd want to have, say, a 59
 bit padding, but can't the implementation figure it out on it's own?


 The original code has it set that way, why? Perhaps so you are aware and
 actually have in place where all the bits are assigned (even if you
 aren't using them); Be horrible if you used accidently 33 bits and it
 extended to 64 without telling you (Wouldn't it?).


Yes, that's the intent. The user must define exactly how an entire 
ubyte/ushort/uint/ulong is filled, otherwise ambiguities and bugs are 
soon to arrive.


 However, having it fill the size in and ignore the last x bits wouldn't
 be too hard to do, I've been wondering if I should remove it.


Please don't. The effort on the programmer side is virtually nil, and 
keeps things in check. In no case would the use of bitfields() be so 
intensive that the bloat of one line gets any significance.


Andrei
Aug 02 2012
parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 12:35:20 UTC, Andrei Alexandrescu 
wrote:


 Please don't. The effort on the programmer side is virtually 
 nil, and keeps things in check. In no case would the use of 
 bitfields() be so intensive that the bloat of one line gets any 
 significance.>


  If you're using a template or something to fill in the sizes, 
then having to calculate the remainder could be an annoyance; but 
those cases would be small in number.

  I'll agree, and it's best leaving it as it is.

  BTW, Wasn't there a new/reserved type of cent/ucent (128bit)?
Aug 02 2012
prev sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:


 This is obviously a mistake in the bitfield implementation. 
 What else could be concluded from the error message:

 std\bitmanip.d(76): Error: shift by 32 is outside the range 
 0..31

 Requesting a 32 bit or 64 bit member on the other hand is not a 
 mistake, and it is not useless, therefore the analogy breaks 
 down.


  Well curiously it was easier to fix than I thought (a line for a 
static if, and a modification of the masking)... Was there any 
other bugs that come to mind? Anything of consequence?
Jul 31 2012
prev sibling next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 31 July 2012 at 16:16:00 UTC, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 15:25:55 UTC, Andrej Mitrovic wrote:

 On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:

 The bug is only when the field is EXACTLY 32 bits BTW. 
 bitfields works quite nice with 33 or whatever. More details 
 in the report.


 Yeah 32 or 64 bits, thanks for changing the title.


  I wonder, is it really a bug? If you are going to have it fill 
 a whole size it would fit anyways, why even put it in as a 
 bitfield? You could just declare it separately.

  I get the feeling it's not so much a bug as a design feature. 
 If you really needed a full size not aligned with whole bytes 
 (or padded appropriately) then I could understand, but still...


  And I'm the one that changed the title name. Suddenly I'm 
 reminded of south park (movie) and the buttfor.


No, it's a bug. There is no reason for it to fail (and it 
certainly isn't a feature).

Maybe the user wants to pack an "uint, ushort, ubyte, ubyte" 
together in a struct, but doesn't want the rest of that struct's 
members 1-aligned?

Maybe the user needs a 32 bit ulong? This way the ulong only 
takes 32 bits, but can still be implicitly passed to functions 
expecting ulongs.

Maybe the user generated the mixin using another template? For 
example, testing integers from 10 bits wide to 40 bits wide? 
Should he write a special case for 32?

...

Now I'm not saying it is big bug or anything, but it is something 
that should be supported... Or at least explicitly asserted until 
fixed...
Jul 31 2012
parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 16:59:11 UTC, monarch_dodra wrote:

 No, it's a bug. There is no reason for it to fail (and it 
 certainly isn't a feature).


  If I made two fields in a 64bit bitfield, each 32bits int's I'd 
like it to complain; If it's calculated from something else then 
finding the problem may be a little more difficult. But that's 
how my mind works, give you the tools you need to do whatever you 
want including shooting yourself in the foot (although you need 
to work harder to do that than C++).


 Maybe the user wants to pack an "uint, ushort, ubyte, ubyte" 
 together in a struct, but doesn't want the rest of that 
 struct's members 1-aligned?


  That would be the one reason that makes sense; Course can't you 
align sections at 1 byte and then by the default afterwards? 
Course now that I think about it, some systems don't have options 
to do byte alignments and require to access memory at 4/8 byte 
alignments. This makes sense needing to support it.


 Maybe the user needs a 32 bit ulong? This way the ulong only 
 takes 32 bits, but can still be implicitly passed to functions 
 expecting ulongs.


  I would think the bug only showed itself if you did int at 
32bits and ulong at 64 bits, not ulong at 32bits.


 Maybe the user generated the mixin using another template? For 
 example, testing integers from 10 bits wide to 40 bits wide? 
 Should he write a special case for 32?


  Ummm..... Yeah I think I see what your saying; And you're 
probably right a special case would be less easily documented 
(and more a pain) when it shouldn't need a special case.


 Now I'm not saying it is big bug or anything, but it is 
 something that should be supported... Or at least explicitly 
 asserted until fixed...
Jul 31 2012
prev sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/31/2012 09:15 AM, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 15:25:55 UTC, Andrej Mitrovic wrote:

 On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:

 The bug is only when the field is EXACTLY 32 bits BTW. bitfields
 works quite nice with 33 or whatever. More details in the report.


 Yeah 32 or 64 bits, thanks for changing the title.


 I wonder, is it really a bug? If you are going to have it fill a whole
 size it would fit anyways, why even put it in as a bitfield? You could
 just declare it separately.


It can happen in templated code where the width of the first field may 
be a template parameter. I wouldn't want to 'static if (width == 32)'.

But thanks for fixing the bug already! :)

Ali
Jul 31 2012
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 30 July 2012 at 17:15:54 UTC, Andrej Mitrovic wrote:

 On 7/30/12, Ali Çehreli <acehreli yahoo.com> wrote:

 The program should be able to detect the placements of the 
 fields. You
 can expect a certain bit pattern for a given set of values of 
 the bit
 fields, set those values, read them on the D side, detect 
 potential
 inconsistencies, and fail the execution (or hopefully 
 compilation).


 Yeah that was exactly my thinking, these things can be automated
 although they might slow down the whole codegeneration process a
 little bit (not that it matters much since you would rarely 
 have to
 regenerate).

 Btw, I get an internal std error with this mixin:

 import std.bitmanip;
 struct Foo
 {
     mixin(bitfields!(
         uint, "bits1", 32,
     ));
 }

 D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76): 
 Error:
 shift by 32 is outside the range 0..31

 Should I file this?


Yes, that should work.

By the way, what's a "butfield"?
Jul 30 2012
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, monarch_dodra <monarchdodra gmail.com> wrote:

 By the way, what's a "butfield"?


LOL I haven't even noticed.
Jul 30 2012