www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Wouldn't this be better with bitfields?

reply Walter Bright <newshound2 digitalmars.com> writes:
Take a look at this instruction:

https://www.scs.stanford.edu/~zyedidia/arm64/encodingindex.html#addsub_imm

And this bit of implementation:

https://github.com/dlang/dmd/pull/16554/files#diff-230d65c83f6bee1f96ea368513dbbe744372ed550c54ce6f0be171ef0848815dR38

```
/* Add/subtract (immediate)
  * ADD/ADDS/SUB/SUBS Rd,Rn,#imm{, shift}
  * https://www.scs.stanford.edu/~zyedidia/arm64/encodingindex.html#addsub_imm
  */
static uint addsub_imm(uint sf, uint op, uint S, uint sh, uint imm12, ubyte Rn, 
ubyte Rd)
{
     return (sf     << 31) |
            (op     << 30) |
            (S      << 29) |
            (0x22   << 23) |
            (sh     << 22) |
            (imm12  << 10) |
            (Rn     <<  5) |
             Rd;
}
```
That's just to initialize it. Never mind the shift/mask code to read a fields, 
or change just one of the fields. There's a large number of these functions in 
instr.d. Wouldn't it be better with:

```
struct addsub_imm { uint Rd:5, Rn:5, imm12:12, sh:1, x22:6, S:1, op:1, sf: 1; }
```

??
Jul 02
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 03/07/2024 11:32 AM, Walter Bright wrote:

 That's just to initialize it. Never mind the shift/mask code to read a 
 fields, or change just one of the fields. There's a large number of 
 these functions in instr.d. Wouldn't it be better with:
 
 |struct addsub_imm { uint Rd:5, Rn:5, imm12:12, sh:1, x22:6, S:1, op:1, 
 sf: 1; } |
 
 ??


Absolutely!

Especially with the reassurances that changing your target or platform 
won't end up changing what it does!
Jul 02
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/2/2024 6:39 PM, Richard (Rikki) Andrew Cattermole wrote:

 Especially with the reassurances that changing your target or platform won't
end 
 up changing what it does!


It won't. As I've repeatedly stated, if you use uints it won't change.
Jul 02
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Tuesday, 2 July 2024 at 23:32:22 UTC, Walter Bright wrote:

 Take a look at this instruction:

 https://www.scs.stanford.edu/~zyedidia/arm64/encodingindex.html#addsub_imm

 And this bit of implementation:

 https://github.com/dlang/dmd/pull/16554/files#diff-230d65c83f6bee1f96ea368513dbbe744372ed550c54ce6f0be171ef0848815dR38

 ```
 /* Add/subtract (immediate)
  * ADD/ADDS/SUB/SUBS Rd,Rn,#imm{, shift}
  * 
 https://www.scs.stanford.edu/~zyedidia/arm64/encodingindex.html#addsub_imm
  */
 static uint addsub_imm(uint sf, uint op, uint S, uint sh, uint 
 imm12, ubyte Rn, ubyte Rd)
 {
     return (sf     << 31) |
            (op     << 30) |
            (S      << 29) |
            (0x22   << 23) |
            (sh     << 22) |
            (imm12  << 10) |
            (Rn     <<  5) |
             Rd;
 }
 ```
 That's just to initialize it. Never mind the shift/mask code to 
 read a fields, or change just one of the fields. There's a 
 large number of these functions in instr.d. Wouldn't it be 
 better with:

 ```
 struct addsub_imm { uint Rd:5, Rn:5, imm12:12, sh:1, x22:6, 
 S:1, op:1, sf: 1; }
 ```


Is there a guarantee that the bit pattern will match what the CPU 
expects?

I remember someone telling me that ["If you are still concerned 
about the layout, like you want a portable file format, don't use 
bit fields. Use 
std.bitmanip."](https://forum.dlang.org/post/usd5b3$23ni$1 digitalmars.com)

If you are going to use bitfields here, make sure you have tests 
to ensure the bit layout is as expected.

-Steve
Jul 02
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/2/2024 8:26 PM, Steven Schveighoffer wrote:

 Is there a guarantee that the bit pattern will match what the CPU expects?


As I've written many times, if you stick with uint the layout is portable with 
every C compiler I know of.
Jul 03
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Wednesday, 3 July 2024 at 17:58:53 UTC, Walter Bright wrote:

 On 7/2/2024 8:26 PM, Steven Schveighoffer wrote:

 Is there a guarantee that the bit pattern will match what the 
 CPU expects?


 As I've written many times, if you stick with uint the layout 
 is portable with every C compiler I know of.


So my response was mostly tongue in cheek, since the dismissal of 
previous concerns about exact layout was always "just use 
std.bitmanip". I have since relented that I'm OK with the C 
compatibility as long as there is some expectation that "as long 
as you use uint, it's fine".

But... I'm thinking now, why not just specify that? If you use 
uint, this is the explicit layout, and any C compiler that 
doesn't implement that mechanism, D does not support bitfield 
compatibility.

That would go a long way to alleviating any concerns that 
portability would be based on the whim of some C compiler.

This should be fine, because, as you say, everyone already does 
it that way. D has the opportunity to make this official.

-Steve
Jul 03
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 04/07/2024 1:47 PM, Steven Schveighoffer wrote:

 On Wednesday, 3 July 2024 at 17:58:53 UTC, Walter Bright wrote:

 On 7/2/2024 8:26 PM, Steven Schveighoffer wrote:

 Is there a guarantee that the bit pattern will match what the CPU 
 expects?


 As I've written many times, if you stick with uint the layout is 
 portable with every C compiler I know of.


 
 So my response was mostly tongue in cheek, since the dismissal of 
 previous concerns about exact layout was always "just use std.bitmanip". 
 I have since relented that I'm OK with the C compatibility as long as 
 there is some expectation that "as long as you use uint, it's fine".
 
 But... I'm thinking now, why not just specify that? If you use uint, 
 this is the explicit layout, and any C compiler that doesn't implement 
 that mechanism, D does not support bitfield compatibility.
 
 That would go a long way to alleviating any concerns that portability 
 would be based on the whim of some C compiler.
 
 This should be fine, because, as you say, everyone already does it that 
 way. D has the opportunity to make this official.
 
 -Steve


As a consequence it does mean that you shouldn't be doing things like 
tagged pointers with it.

And that's a problem since pretty much all new cpu designs are 64bit.
Jul 04
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/4/2024 8:18 AM, Richard (Rikki) Andrew Cattermole wrote:

 As a consequence it does mean that you shouldn't be doing things like tagged 
 pointers with it.


That's not a problem, either.


 And that's a problem since pretty much all new cpu designs are 64bit.


If you layout your fields the same way as the associated C compiler for that
cpu 
does, you'll be just fine.
Jul 05
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/3/2024 6:47 PM, Steven Schveighoffer wrote:

 But... I'm thinking now, why not just specify that? If you use uint, this is
the 
 explicit layout, and any C compiler that doesn't implement that mechanism, D 
 does not support bitfield compatibility.
 That would go a long way to alleviating any concerns that portability would be 
 based on the whim of some C compiler.
 
 This should be fine, because, as you say, everyone already does it that way. D 
 has the opportunity to make this official.


AFAIK, no C compiler has bothered to document the details of their bit field
layout.

As for D, it specifies that the layout will match that of the "associated C 
compiler".

We could add a "best practices" note in the spec.
Jul 05
parent reply claptrap <clap trap.com> writes:
On Friday, 5 July 2024 at 16:48:29 UTC, Walter Bright wrote:

 On 7/3/2024 6:47 PM, Steven Schveighoffer wrote:

 But... I'm thinking now, why not just specify that? If you use 
 uint, this is the explicit layout, and any C compiler that 
 doesn't implement that mechanism, D does not support bitfield 
 compatibility.
 That would go a long way to alleviating any concerns that 
 portability would be based on the whim of some C compiler.
 
 This should be fine, because, as you say, everyone already 
 does it that way. D has the opportunity to make this official.


 AFAIK, no C compiler has bothered to document the details of 
 their bit field layout.

 As for D, it specifies that the layout will match that of the 
 "associated C compiler".

 We could add a "best practices" note in the spec.


D should not be adding language features that change their 
specification depending on what external C compiler happens to be 
attached to the project.

That is a ridiculous design.

You need to detox your brain of C. First it was C functions 
should be assumed safe, then it was string interpolation should 
be built around printf compatibility, and now it's bitfields 
should change their implementation based on whatever C compiler 
is attached?

The C part of your brain keeps leading you done one bad path 
after another.
Jul 07
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/7/2024 3:21 AM, claptrap wrote:

 D should not be adding language features that change their specification 
 depending on what external C compiler happens to be attached to the project.


It always has - from its inception the struct member layout and alignment 
matches what C does. D needs to be a useful programming language, and
effortless 
compatibility with C data layout makes it much more useful.
Jul 08
parent reply claptrap <clap trap.com> writes:
On Monday, 8 July 2024 at 23:29:15 UTC, Walter Bright wrote:

 On 7/7/2024 3:21 AM, claptrap wrote:

 D should not be adding language features that change their 
 specification depending on what external C compiler happens to 
 be attached to the project.


 It always has - from its inception the struct member layout and 
 alignment matches what C does. D needs to be a useful 
 programming language, and effortless compatibility with C data 
 layout makes it much more useful.


You have completely missed the point. Struct layout doesn't 
change, it's fixed, it doesn't depend on what C compiler is 
attached.

It's not C-compatibility, but letting the under specification of 
C bitfields basically infect D. Its the tail wagging the dog. Or 
leaky implementation. If I have a D program that links a dll or 
static lib also written in D, they could have incompatible 
bitfield layouts.

You could fix this by specifying the layout for D bitfields, and 
using extern(C) when compatibility with the relevant C compiler 
is required. D already does that for classes and functions...
Jul 09
next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Tuesday, 9 July 2024 at 10:57:02 UTC, claptrap wrote:


 You have completely missed the point. Struct layout doesn't 
 change, it's fixed, it doesn't depend on what C compiler is 
 attached.


The layout is fixed, but alignment and padding are implementation 
specific.


 It's not C-compatibility, but letting the under specification 
 of C bitfields basically infect D. Its the tail wagging the 
 dog. Or leaky implementation. If I have a D program that links 
 a dll or static lib also written in D, they could have 
 incompatible bitfield layouts.

 You could fix this by specifying the layout for D bitfields, 
 and using extern(C) when compatibility with the relevant C 
 compiler is required. D already does that for classes and 
 functions...


`extern(C)` does not affect the layout of anything, as far as I 
know. It only impacts name mangling and calling convention. Using 
it to change layout would be a new feature.

When I worked on Derelict and BindBC, I would have loved to have 
had bitfields in D. And I would have expected them to line up 
with the associated C compiler out of the box, because that's 
what already happens with structs.
Jul 09
next sibling parent claptrap <clap trap.com> writes:
On Tuesday, 9 July 2024 at 11:58:14 UTC, Mike Parker wrote:

 On Tuesday, 9 July 2024 at 10:57:02 UTC, claptrap wrote:


 You have completely missed the point. Struct layout doesn't 
 change, it's fixed, it doesn't depend on what C compiler is 
 attached.


 The layout is fixed, but alignment and padding are 
 implementation specific.


 It's not C-compatibility, but letting the under specification 
 of C bitfields basically infect D. Its the tail wagging the 
 dog. Or leaky implementation. If I have a D program that links 
 a dll or static lib also written in D, they could have 
 incompatible bitfield layouts.

 You could fix this by specifying the layout for D bitfields, 
 and using extern(C) when compatibility with the relevant C 
 compiler is required. D already does that for classes and 
 functions...


 `extern(C)` does not affect the layout of anything, as far as I 
 know. It only impacts name mangling and calling convention. 
 Using it to change layout would be a new feature.

 When I worked on Derelict and BindBC, I would have loved to 
 have had bitfields in D. And I would have expected them to line 
 up with the associated C compiler out of the box, because 
 that's what already happens with structs.


Apparently I'm an idiot who has no idea what he's talking about.

Sorry.
Jul 10
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/9/2024 4:58 AM, Mike Parker wrote:

 `extern(C)` does not affect the layout of anything, as far as I know. It only 
 impacts name mangling and calling convention. Using it to change layout would
be 
 a new feature.


There is a case where it does. A struct with no data members is zero size if 
extern(C), otherwise it is size 1. The size 1 comes from C++, which (correctly) 
decided that individual objects should have distinct addresses.

This does have the potential to be memory unsafe, and a struct with no data 
members should probably be rejected in  safe mode.

https://issues.dlang.org/show_bug.cgi?id=24657
Jul 10
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/9/2024 3:57 AM, claptrap wrote:

 You have completely missed the point. Struct layout doesn't change, it's
fixed, 
 it doesn't depend on what C compiler is attached.


I'm sorry to say, it does.



 You could fix this by specifying the layout for D bitfields, and using
extern(C) 
 when compatibility with the relevant C compiler is required.


I've posted in the other thread simple means to lay out portable bitfields.
More 
language features are not required.
Jul 09
parent reply claptrap <clap trap.com> writes:
On Wednesday, 10 July 2024 at 06:47:44 UTC, Walter Bright wrote:

 On 7/9/2024 3:57 AM, claptrap wrote:

 You have completely missed the point. Struct layout doesn't 
 change, it's fixed, it doesn't depend on what C compiler is 
 attached.


 I'm sorry to say, it does.



 You could fix this by specifying the layout for D bitfields, 
 and using extern(C) when compatibility with the relevant C 
 compiler is required.


 I've posted in the other thread simple means to lay out 
 portable bitfields. More language features are not required.


I actually replied to mike yesterday and admitted that I was 
wrong, (I looked at the spec, should have done that before 
opening my mouth) and that I was and idiot, not sure if it got 
moderated because I was rude to myself? Or maybe mike thought I 
was being sarcastic, not sure tbh.

But I was wrong, sorry.
Jul 10
parent reply Dennis <dkorpel gmail.com> writes:
On Wednesday, 10 July 2024 at 08:25:17 UTC, claptrap wrote:

 I was and idiot, not sure if it got moderated because I was 
 rude to myself?


You're currently 'banned' with reason 'spam', meaning every 
message you post has to be approved first. Not sure if that ban 
is accurate.
Jul 10
parent reply claptrap <clap trap.com> writes:
On Wednesday, 10 July 2024 at 10:07:54 UTC, Dennis wrote:

 On Wednesday, 10 July 2024 at 08:25:17 UTC, claptrap wrote:

 I was and idiot, not sure if it got moderated because I was 
 rude to myself?


 You're currently 'banned' with reason 'spam', meaning every 
 message you post has to be approved first. Not sure if that ban 
 is accurate.


I'm not selling penis enlargement products if that helps?

At least not yet anyway.
Jul 10
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/10/2024 2:17 PM, claptrap wrote:

 I'm not selling penis enlargement products if that helps?


Please stop with the inappropriate comments.
Jul 11
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
Walter Bright kirjoitti 3.7.2024 klo 2.32:

 static uint addsub_imm(uint sf, uint op, uint S, uint sh, uint imm12, 
 ubyte Rn, ubyte Rd)
 {
      return (sf     << 31) |
             (op     << 30) |
             (S      << 29) |
             (0x22   << 23) |
             (sh     << 22) |
             (imm12  << 10) |
             (Rn     <<  5) |
              Rd;
 }
 ```
 That's just to initialize it. Never mind the shift/mask code to read a 
 fields, or change just one of the fields. There's a large number of 
 these functions in instr.d. Wouldn't it be better with:
 
 ```
 struct addsub_imm { uint Rd:5, Rn:5, imm12:12, sh:1, x22:6, S:1, op:1, 
 sf: 1; }
 ```
 
 ??


Yes. And we already basically have it:

```D
import std.bitmanip;
struct addsub_imm
{   mixin(bitfields!
     (  ubyte, "Rd", 5,
        ubyte, "Rn", 5,
        uint, "imm12", 12,
        uint, "sh", 1,
        ubyte, "x22", 6,
        uint, "S", 1,
        uint, "op", 1,
        uint, "sf", 1
     ));
}
```

The language builtin would have somewhat nicer syntax though. What if 
bitfields were moved from `std.bitmanip` to `core.*` and the compiler 
would lower any bitfield declarations in struct/class to a `bitfields` 
mixin?
Jul 03
next sibling parent Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Wednesday, 3 July 2024 at 09:24:52 UTC, Dukc wrote:

 What if bitfields were moved from `std.bitmanip` to `core.*` 
 and the compiler would lower any bitfield declarations in 
 struct/class to a `bitfields` mixin?


Then there could even be `extern(C)` bitfields guaranteeing 
compatibility with the associated C compiler, and `extern(D)` 
bitfields guaranteeing portability. (`extern(C++)` bitfields 
would be the same as `extern(C)`, I guess.)

The `extern(C)` bitfields would have to be a compiler intrinsic, 
whereas the `extern(D)` ones could be a lowering to some mixin 
template in `core.bitfield`.
Jul 03
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/3/2024 2:24 AM, Dukc wrote:

 The language builtin would have somewhat nicer syntax though.


A much nicer syntax.


 What if bitfields 
 were moved from `std.bitmanip` to `core.*` and the compiler would lower any 
 bitfield declarations in struct/class to a `bitfields` mixin?


Then they won't be compatible with C bitfields.
Jul 03
parent reply Dukc <ajieskola gmail.com> writes:
Walter Bright kirjoitti 3.7.2024 klo 21.04:

 
 Then they won't be compatible with C bitfields.
 


Were in agreement we should have two bit field types (A well-specified 
one that's `std.bitmanip` one, and the C-compatible one) if we count the 
library implementations. I don't think other people have issues with 
that either.

The question is, why does the C-compatible one have to be the 
language-level one? What's the problem in having `std.bitmanip` as the 
language level field while still providing the C-compatible bitfield via 
a DRuntime or Phobos API?

I have readen your take on this from the latest DIP revision:


 Many methods were proposed to deal with this. The only time the layout matters
is with Usage No. 3, where the layout is imposed externally. For Usage Nos. 1
and 2, the particular layout does not matter. When it does matter, many
alternatives are available:
 
     Use https://dlang.org/phobos/std_bitmanip.html#bitfields
     If one sticks with one field type (such as int), then the layout is
predictable in practice, although not in specification
     Use custom functions to encode/decode fields
     A bit of testing will show what a particular layout is, after which it
will be reliable
 
 The alternatives are easy enough, and Usage No. 3 is unusual enough, that the
extra language complexity to support multiple layouts is not worth it. Note
that the author would use a lot more bitfields if they were in the language.


This doesn't really answer the question though, because in my view at 
least we could just as rightfully write:

The objections revolve around discomfort with the layout being 
incompatible with the associated C compiler. Many methods are there to 
deal with this. The only time this is needed is Usage No. 2, where the 
layout is imposed externally. For Usage No. 1, C compatibility does not 
matter and for 3 it would be less practical to use, being implementation 
dependant. When C compatibility is needed, many alternatives are available:

     Use the C-compatible DRuntime bitfield API
     If one sticks with one field type (such as int), then the layout is 
same as with D in practice, although not in the C specification
     Use custom functions to encode/decode fields
     A bit of investigation will show what the C bitfield layout is, 
after which it can be translated to a D bitfield, possibly with 
`version` statements to deal with differences between compilers.

The alternatives are easy enough, and Usage No. 2 is nonstandard enough, 
that the extra language complexity to support multiple bitfield types at 
language level is not worth it. Note that the author does need to use 
mentioned alternatives a lot anyway to integrate with underspecified C 
types, such as `long`.

So, is there something wrong with this take?
Jul 06
next sibling parent reply Dukc <ajieskola gmail.com> writes:
Dukc kirjoitti 6.7.2024 klo 20.56:

 Note that the author does need to use mentioned alternatives a lot 
 anyway to integrate with underspecified C types, such as `long`.


Actually this is an analogy I want to push. Ask yourself, why is `long` 
defined as exactly 64 bits in D? Why isn't it defined to have the same 
size as associated C `long`?

This is why language-level bitfields should not attempt to mimic C 
either, at least not by default.
Jul 06
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/6/2024 11:05 AM, Dukc wrote:

 Actually this is an analogy I want to push. Ask yourself, why is `long`
defined 
 as exactly 64 bits in D? Why isn't it defined to have the same size as 
 associated C `long`?


Because a 32 bit C long is useless. It's a vestigial remainder from 16 bit 
programming.

Sensible C programmers use "long long" instead.
Jul 06
parent Dukc <ajieskola gmail.com> writes:
On Sunday, 7 July 2024 at 03:58:50 UTC, Walter Bright wrote:

 On 7/6/2024 11:05 AM, Dukc wrote:

 Actually this is an analogy I want to push. Ask yourself, why 
 is `long` defined as exactly 64 bits in D? Why isn't it 
 defined to have the same size as associated C `long`?


 Because a 32 bit C long is useless. It's a vestigial remainder 
 from 16 bit programming.

 Sensible C programmers use "long long" instead.


Yes. In the same way:

Implementation defined behaviour for bitfields is useless. It's 
there just for backwards compatibility for times when there was 
no C standard for it, or whatever.

Sensible C programmers stick with aligning the start of the 
bitfield with alignment of the bitfield type, which makes the 
layout predictable and lets `std.bitmanip` bitfield to be used 
for interfacing with it.
Jul 06
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/6/2024 10:56 AM, Dukc wrote:

 The question is, why does the C-compatible one have to be the language-level 
 one?


Because it's sooo convenient and it makes mixed C/D programs even easier.

User convenience can't be oversold, that's been hammered into me over decades.
Jul 06
parent claptrap <clap trap.com> writes:
On Saturday, 6 July 2024 at 23:36:12 UTC, Walter Bright wrote:

 On 7/6/2024 10:56 AM, Dukc wrote:

 The question is, why does the C-compatible one have to be the 
 language-level one?


 Because it's sooo convenient and it makes mixed C/D programs 
 even easier.

 User convenience can't be oversold, that's been hammered into 
 me over decades.


And you cant find a way to do it that doesn't require D bitfields 
being built on shifting sands?

And it's a niche feature, in a niche situation, really how many 
people are going to be doing mixed C/D programs, and how many of 
them will need compatibility between C and D bitfields.

We should not be letting C idiocy infect the D language.
Jul 07