www.digitalmars.com         C & C++   DMDScript  

digitalmars.dip.development - DIP: add Bit Fields

reply Walter Bright <newshound2 digitalmars.com> writes:
https://github.com/WalterBright/documents/blob/master/bitfields.md
Mar 06
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
"gdc/lcd are lumped together because they produce identical results on 
the same platform."

s/lcd/ldc/

No mention of alignment, total size or the offset.

Needs to mention atomics as not being supported.

No trait to detect a bit field (if its an alias, or by name), nor get 
its size. Kinda required, enough other things in the language can appear 
via an alias sequence or by allMembers that are not able to be taken as 
a pointer of.

The ability to take an alias via getMember needs to be considered if its 
valid to do so.

No mention of tupleof either, what happens there?

No motivation for not including a D layout for it, instead of forcing 
everyone into an unpredictable layout that the system C compiler might 
be using. Moving us off the library type and into a language defined one 
that is predictable is quite attractive so there needs to be an argument 
against this.

Also both the post and its subject do not match the guidelines for this 
forum. So I'm half wondering if it's even worth writing this post 
knowing that Mike could delete it.

https://github.com/dlang/DIPs/blob/master/docs/guidelines-forums.md#dip-development
Mar 06
parent Nick Treleaven <nick geany.org> writes:
On Thursday, 7 March 2024 at 05:00:36 UTC, Richard (Rikki) Andrew 
Cattermole wrote:
 "gdc/lcd are lumped together because they produce identical 
 results on the same platform."

 s/lcd/ldc/

 No mention of alignment, total size or the offset.
Agree, the DIP should mention those. The implementation seems to give each bitfield of a set the same value: ```d struct S { uint a:1, b:1, c:1; } static assert(S.b.alignof == 4); static assert(S.b.offsetof == 0); static assert(S.b.sizeof == 4); ```
 Needs to mention atomics as not being supported.
Yes. Bitfield impl doesn't allow shared, static (or even const) storage classes.
 No trait to detect a bit field (if its an alias, or by name), 
 nor get its size. Kinda required, enough other things in the 
 language can appear via an alias sequence or by allMembers that 
 are not able to be taken as a pointer of.
Possibly this might be enough (using S above): ```d enum E { a, b, c } struct U { int i; alias i this; enum max = 2; } enum isBitfield(alias a) = __traits(isIntegral, typeof(a)) && !is(typeof(a) == enum) && a.max <= uint.sizeof * 8; static assert(isBitfield!(S.b)); static assert(!isBitfield!(E.b)); static assert(!isBitfield!(U())); ```
 The ability to take an alias via getMember needs to be 
 considered if its valid to do so.

 No mention of tupleof either, what happens there?
Impl: ```d void main() { S s; s.tupleof[1]++; __traits(getMember, s, "c")++; writeln(s.tupleof); // 011 } ```
 No motivation for not including a D layout for it, instead of 
 forcing everyone into an unpredictable layout that the system C 
 compiler might be using. Moving us off the library type and 
 into a language defined one that is predictable is quite 
 attractive so there needs to be an argument against this.
Just to mention, the argument in the DIP was portability with C bitfields.
Mar 09
prev sibling next sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Have you ever thought of the idea of using bit specific integers? I once saw this one suggestion on the IRC channel: `alias i1 = __traits(integer, signed: false, 1);` ```D alias u1 = __traits(integer, true, 1); alias u3 = __traits(integer, true, 3); alias u5 = __traits(integer, true, 5); align(1) struct S { u1 isnothrow; // nothrow u1 isnogc; // is nogc u1 isproperty; // can be called without parentheses u1 isref; // returns a reference u1 isreturn; // 'this' is returned by ref u1 isscope; // 'this' is scope u1 isreturninferred; // 'this' is return from inference u1 isscopeinferred; // 'this' is scope from inference u1 islive; // is live u1 incomplete; // return type or default arguments removed u1 inoutParam; // inout on the parameters u1 inoutQual; // inout on the qualifier u5 a; u3 flags; } ```
Mar 07
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
 I've seen plentiful use of it, and have not seen any "do not 
 use bitfields" programming guidelines.
[Bit-fields in C have acquired a poor reputation in the programming community](https://andrewkelley.me/post/a-better-way-to-implement-bit-fields.html) [C bitfields considered harmful](https://www.flamingspork.com/blog/2014/11/14/c-bitfields-considered-harmful/) [So don't go for bitfields "just because"](https://yarchive.net/comp/linux/bitfields.html) [Bitfields seem to be perfect for this, but they're so poorly specified in the standard that common "best practice" is to avoid them.](https://www.reddit.com/r/C_Programming/comments/146tbnp/why_is_using_bitfields_in_high_performance/)
Mar 07
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2024 4:04 AM, Dennis wrote:
 On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
 I've seen plentiful use of it, and have not seen any "do not use bitfields" 
 programming guidelines.
[Bit-fields in C have acquired a poor reputation in the programming community](https://andrewkelley.me/post/a-better-way-to-implement-bit-fields.html)
All Zig is doing is providing a slightly different syntax that does the same thing as C. Zig is layering on a magic "pointer to bit field type" that I attempted to implement in D long ago, but it has too many special cases and turned out to be unworkable. You cannot convert a pointer to bit 3 to a void*, making the special bit pointer type rather useless.
 [C bitfields considered 
 harmful](https://www.flamingspork.com/blog/2014/11/14/c-bitfields-considered-harmful/)
I don't know of any C compiler that puts a single bit flag in a 64 bit int. He didn't show enough code to see the source of his problem. I suspect he doesn't know what he's doing.
 [So don't go for bitfields "just 
 because"](https://yarchive.net/comp/linux/bitfields.html)
Linus suggests using explicit bit masks instead.
 [Bitfields seem to be perfect for this, but they're so poorly specified in the 
 standard that common "best practice" is to avoid 
 them.](https://www.reddit.com/r/C_Programming/comments/146tbnp/why_is_using_bitfields_in_high_performance/)
I.e. the layout is implementation-defined. That's correct. But in reality, it is fixed. There is no way that gcc on Linux is *ever* going to change the bit field layout, because it would break legions of existing code. It is cast in concrete, and is as immutable as it gets. The same for Microsoft C. Furthermore, if you use `int` as the base type for bit fields, the layouts are all the same in all the C compilers I've encountered. For practical purposes, it is defined. If you are still concerned about the layout, like you want a portable file format, don't use bit fields. Use std.bitmanip. But if you want to be compatible with C code that uses bit fields, the builtin bit fields are the way to go. If you like a nice syntax, bit fields cannot be beat. If you want to pack data in your structs and use less memory for your program, bit fields are again a winner. DMD uses far too much memory. Much of the data structs it uses can be packed, but since without bit fields the syntax is just too ugly, there's been little effort to do that. Convenience is everything. If you still don't like bit fields, don't use them. And yes, you can't do volatile, shared, atomic, or pointers to bit fields. You can't do those with std.bitmanip, either. Nor can you do them with manual masking and shifting.
Mar 07
next sibling parent Dennis <dkorpel gmail.com> writes:
On Thursday, 7 March 2024 at 19:42:36 UTC, Walter Bright wrote:
 ...
You can disagree, but 'I have not seen any "do not use bitfields" programming guidelines' (which wasn't a strong argument to begin with) is now false, so I suggest you replace it in the DIP.
Mar 07
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Thursday, 7 March 2024 at 19:42:36 UTC, Walter Bright wrote:
 If you like a nice syntax, bit fields cannot be beat.
There's no need to argue that native bit fields look nicer than manual bit shifts. Of course they do.
 And yes, you can't do volatile, shared, atomic, or pointers to 
 bit fields.
Of course you can't. No need to defend this. The main question is: Why inherit C's controversial inconsistent layout scheme? The DIP keeps going on about C interoperability, but only cites abstract "experience" as an argument. My last use of bit fields was from optimizing `std.uni` tables: https://github.com/dlang/phobos/pull/8880 Wouldn't it be nice if I could use the shiny new bit fields feature for that? Well I can't: the table needs to know the bit offsets, which would be target dependent. I'd have to use the inferior `std.bitmanip`. If I didn't know about the platform dependence, I'd use bit fields incorrectly. It would all work until compiling to a different target results in a corrupted table, without warning. After debugging and finding the cause I'd write a blog post "D bit fields considered harmful" like C users have done before. Why bow down to C and treat D as a second class citizen? The DIP should really substantiate this alleged importance of C-compatibility of non-ImportC D bitfields: Can you name at least one dub package that would benefit from bit fields with C-compatible layout?
Mar 07
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 7, 2024 3:28:34 PM MST Dennis via dip.development wrote:
 Why bow down to C and treat D as a second class citizen? The DIP
 should really substantiate this alleged importance of
 C-compatibility of non-ImportC D bitfields: Can you name at least
 one dub package that would benefit from bit fields with
 C-compatible layout?
Having the ability to declare C-compatible bitfields would definitely be useful for binding some C symbols. I was recently dealing with some symbols on Windows where I basically had to skip the bitfields when writing the bindings, because I don't know of any way to bind to them in D. That being said, I see no reason for bitfields for actual D code to be C-compatible. Unless I'm actually binding to a C type, C-compatibility is irrelevant, and if what C is doing isn't portable, then why would I want it for my D code? - Jonathan M Davis
Mar 07
parent reply Walter Bright <newshound2 digitalmars.com> writes:
You'd want C-compatible bit fields if/when you're building a hybrid D/C/C++ 
binary, like gdc and ldc are.
Mar 07
parent reply Arafel <er.krali gmail.com> writes:
On 8/3/24 3:38, Walter Bright wrote:
 You'd want C-compatible bit fields if/when you're building a hybrid 
 D/C/C++ binary, like gdc and ldc are.
In that case, why not have the best of both worlds and have `extern(D)` (the default, well-defined, perhaps `std.bitmanip`-compatible) and `extern(C)` bit fields, like with classes and C++?
Mar 08
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/8/2024 1:02 AM, Arafel wrote:
 In that case, why not have the best of both worlds and have `extern(D)` (the 
 default, well-defined, perhaps `std.bitmanip`-compatible) and `extern(C)` bit 
 fields, like with classes and C++?
Not a bad idea, but it may be confusing which one is in play, as the extern(C) can be far removed from its effect.
Mar 08
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 09/03/2024 8:28 AM, Walter Bright wrote:
 On 3/8/2024 1:02 AM, Arafel wrote:
 In that case, why not have the best of both worlds and have 
 `extern(D)` (the default, well-defined, perhaps 
 `std.bitmanip`-compatible) and `extern(C)` bit fields, like with 
 classes and C++?
Not a bad idea, but it may be confusing which one is in play, as the extern(C) can be far removed from its effect.
Since fields do not currently have block behaviors for externs, doing this would be a new pattern in the language usage. Otherwise per encapsulation/scope setting of extern may be required. Which is already used, but as it would be required this is new. Either way as per my original comment it needs to be argued which ever way you go, the cost of implementing our own layout shouldn't be an issue considering how many others will be needed > 1. I would love to see this replace the library solution.
Mar 08
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Sometimes, too many options is not helpful. This is one of those situations.
The 
problem with implementation-defined layout is much more of a pedantic one than
a 
pragmatic one, as I've explained elsewhere in this thread.

If one sticks with `int` bit fields, they behave the same across C compilers 
that I've seen. Just like ints are 32 bits.

We don't have an attribute to switch endianess. It's just not worth the added 
complexity.

If anyone is worried about exactly specifying the layout, using masking and 
shifting will guarantee it (absent endianess issues!).

We do not need to solve every detail, just go for the critical ones.
Mar 08
next sibling parent reply Daniel N <no public.email> writes:
On Saturday, 9 March 2024 at 07:51:08 UTC, Walter Bright wrote:
 Sometimes, too many options is not helpful. This is one of 
 those situations. The problem with implementation-defined 
 layout is much more of a pedantic one than a pragmatic one, as 
 I've explained elsewhere in this thread.

 If one sticks with `int` bit fields, they behave the same 
 across C compilers that I've seen. Just like ints are 32 bits.

 We don't have an attribute to switch endianess. It's just not 
 worth the added complexity.

 If anyone is worried about exactly specifying the layout, using 
 masking and shifting will guarantee it (absent endianess 
 issues!).

 We do not need to solve every detail, just go for the critical 
 ones.
Fully agree. There's nothing wrong with implementation-defined. I guess it might be possible to make an even stronger guarantee, specifying that the current behavior is how it must work on windows and unix in order to be compliant with the D language spec. The same way we decided to define the common practise for integer wrapping and two's complement.
Mar 09
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/9/2024 12:38 AM, Daniel N wrote:
 I guess it might be possible to make an even stronger guarantee, specifying
that 
 the current behavior is how it must work on windows and unix in order to be 
 compliant with the D language spec. The same way we decided to define the
common 
 practise for integer wrapping and two's complement.
I've thought about that, and it seemed more straightforward to simply rely on the specified behavior of the associated C compiler. It's like D doesn't specify how floating point arithmetic works, just that it conforms to IEEE754.
Mar 11
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 9 March 2024 at 07:51:08 UTC, Walter Bright wrote:
 The problem with implementation-defined layout is much more of 
 a pedantic one than a pragmatic one, as I've explained 
 elsewhere in this thread.
I've shown that it's a problem when using bit bitfields in D's very own standard library (std.internal.unicode_tables) and asked you to name one dub package that would benefit from D bit fields having C layout, which you didn't do.
Mar 09
next sibling parent reply Daniel N <no public.email> writes:
On Saturday, 9 March 2024 at 09:41:22 UTC, Dennis wrote:
 On Saturday, 9 March 2024 at 07:51:08 UTC, Walter Bright wrote:
 The problem with implementation-defined layout is much more of 
 a pedantic one than a pragmatic one, as I've explained 
 elsewhere in this thread.
I've shown that it's a problem when using bit bitfields in D's very own standard library (std.internal.unicode_tables) and asked you to name one dub package that would benefit from D bit fields having C layout, which you didn't do.
Not on dub, but my closed source makes use of it. Vendor supplied register definitions uses bitfields with C semantics. Also IPC where client/server is written in different languages.
Mar 09
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 9 March 2024 at 10:46:10 UTC, Daniel N wrote:
 Not on dub, but my closed source makes use of it. Vendor 
 supplied register definitions uses bitfields with C semantics. 
 Also IPC where client/server is written in different languages.
That would be great material for the DIP. Can you give a link to such a register definition, and show what your current approach is to create C compatible bit fields for it in D?
Mar 09
parent Daniel N <no public.email> writes:
On Saturday, 9 March 2024 at 10:56:33 UTC, Dennis wrote:
 On Saturday, 9 March 2024 at 10:46:10 UTC, Daniel N wrote:
 Not on dub, but my closed source makes use of it. Vendor 
 supplied register definitions uses bitfields with C semantics. 
 Also IPC where client/server is written in different languages.
That would be great material for the DIP. Can you give a link to such a register definition, and show what your current approach is to create C compatible bit fields for it in D?
The sdk license doesn't allow me to share the files. But you can find similar code here... https://www.renesas.com/us/en/document/apn/sh7450-groupsh7451-group-register-definition-header-file Typically an union with a word sized member is combined with the bitfields to provide either atomic access or convenient access to the fields. Without native D support, I just use C for the lowest layer instead.
Mar 09
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/9/2024 1:41 AM, Dennis wrote:
 I've shown that it's a problem when using bit bitfields in D's very own
standard 
 library (std.internal.unicode_tables)
I presume you mean ``` ulong x; // bit field sizes: 18, 12, 12, 4, 4, 4 ``` https://github.com/dlang/phobos/pull/8880/files#diff-6649807e195867988f8292561430bf14abf6c9cf4cecbea53c7ac2fd436c3623R145 which is 64 bits. That should produce the same result across different C compilers, i.e. it shouldn't be a problem. If I'm missing what the actual problem you've encountered is, please explain.
 and asked you to name one dub package that 
 would benefit from D bit fields having C layout, which you didn't do.
The D compiler itself would, so using it in the D front end will effortlessly work with the C++ gdc and ldc backends.
Mar 11
parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 11 March 2024 at 18:12:58 UTC, Walter Bright wrote:
 On 3/9/2024 1:41 AM, Dennis wrote:
 which is 64 bits. That should produce the same result across 
 different C compilers, i.e. it shouldn't be a problem.
That's great, but it's not what the DIP says currently, which is that the layout (for non-int) is implementation defined matching a C compiler. It makes the comparison with C's varying `int` size, but the difference is that in that case, D sticks to its guns and keeps `int` 32-bits even when C would make it 16-bit. Perhaps the DIP could say: the layout in D is defined as X, which follows what C compilers do in practice. Then programmers can be comfortable relying on the layout.
 The D compiler itself would, so using it in the D front end 
 will effortlessly work with the C++ gdc and ldc backends.
Those are extern(C++) classes, and they already have working bitfields. They wouldn't be impaired by making bit fields in D consistent and predictable.
Mar 11
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/11/2024 11:49 AM, Dennis wrote:
 That's great, but it's not what the DIP says currently, which is that the
layout 
 (for non-int) is implementation defined matching a C compiler.
It's the same story for longs. I had neglected to mention that in the DIP.
 It makes the comparison with C's varying `int` size, but the difference is
that 
 in that case, D sticks to its guns and keeps `int` 32-bits even when C would 
 make it 16-bit. Perhaps the DIP could say: the layout in D is defined as X, 
 which follows what C compilers do in practice.
D doesn't say what the alignment algorithm is, either, just that it matches the associated C compiler. It's always been that way, although it is not an accident. It was deliberate on my part. Nobody seems to notice this, it "just works", as it should. As for 16 bit int D, that is a variant of D, not D. (It should perhaps be called "D16".) This is perfectly reasonable, because you will never be able to port non-trivial D code to D16 without major alterations (DMD itself will never be portable to 16 bit code). This frees the 99.99% of D programmers to not feel a need to contort their code to be "Standard-conformant" to 16 bit targets. I see the pointless self-torture of C programmers doing this all the time. (And in spite of these efforts, the C code is still not portable to those other unusual platforms.) C would be better off if they took the same approach.
 Then programmers can be comfortable relying on the layout.
They already can rely on it, in practice.
 The D compiler itself would, so using it in the D front end will effortlessly 
 work with the C++ gdc and ldc backends.
Those are extern(C++) classes, and they already have working bitfields.
There are currently no bit fields in the DMD front end source code, although there are some emulations of bit fields.
 They wouldn't be impaired by making bit fields in D consistent and predictable.
They are consistent and predictable now. Just like the D and C++ code reliance on ints being 32 bits, the endianess, and the alignment layout of the fields in the structs.
Mar 11
prev sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
I've had a bit of thought about the ABI concerns of having multiple 
layouts possible.

I think its better to ignore the ABI altogether when changing it.

```d
import core.attributes : bitfieldlayout;

struct Foo {
	 bitfieldlayout("packed") {
		int bar:1;
		int zah:2;
	}
}
```

That'll also solve how do we add new ones that isn't C or packed.

C can be default, that need not be changed to something else.
Mar 09
parent Walter Bright <newshound2 digitalmars.com> writes:
I hate to say it, but it's still a complex solution to a rare and frankly 
trivial issue. With some care, one can write portable C bitfields.
Mar 11
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
The inconsistency of C bitfield layout is far less important than it is made
out 
to be. Other inconsistencies C has: char/short/int/long sizes, 
float/double/longdouble behaviors, 1s or 2s complement arithmetic, field 
alignment and endianness. Even the signed-ness of chars is 
implementation-defined. D matches that behavior of the associated C compiler. 
Living with that is normal.

The gdc and ldc backends are written in C++. D bitfields would enable bitfields 
to be used in the D front end, and the backends can use them effortlessly 
(should be no problem for dtoh).

An important feature of D is interoperability with C (and to some extent C++). 
Reducing the friction as much as we reasonably can will lower the barriers to D 
adoption. It means D code can co-exist peacefully in the same binary. When gdc 
and ldc are ported to a different platform, with different C bitfield layouts, 
it will still *effortlessly* work correctly.

I like effortlessly.
Mar 07
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 7, 2024 7:37:02 PM MST Walter Bright via dip.development 
wrote:
 The inconsistency of C bitfield layout is far less important than it is made
 out to be. Other inconsistencies C has: char/short/int/long sizes,
 float/double/longdouble behaviors, 1s or 2s complement arithmetic, field
 alignment and endianness. Even the signed-ness of chars is
 implementation-defined. D matches that behavior of the associated C
 compiler. Living with that is normal.
Yeah, but with the D types, we've mostly tried to fix that (e.g. making char/short/int/long have fixed sizes), and D defines char, wchar, and dchar all as being unsigned. We have enough compatibility to define aliases for the C types and use those in bindings, but normal D code doesn't have to deal with the issue. bviously, there are some cases where we're forced to do the same as C even when we might prefer not to, but we have managed to avoid some of C's mistakes in normal D code in a number of cases while still having extern(C) work.
 The gdc and ldc backends are written in C++. D bitfields would enable
 bitfields to be used in the D front end, and the backends can use them
 effortlessly (should be no problem for dtoh).

 An important feature of D is interoperability with C (and to some extent
 C++). Reducing the friction as much as we reasonably can will lower the
 barriers to D adoption. It means D code can co-exist peacefully in the same
 binary. When gdc and ldc are ported to a different platform, with different
 C bitfield layouts, it will still *effortlessly* work correctly.

 I like effortlessly.
Well, I guess that structs don't technically have extern(C) or extern(D) on them, which makes this harder (structs and classes do with extern(C++), but that doesn't help with C). Otherwise, the obvious thing would be to just use the C behavior in structs / classes that are extern(C) or extern(C++) and use well-defined, cross-platform behavior with extern(D) types, but it's the functions that are extern(*), not the types. However, the next thing that comes to mind would be to use a pragma or attribute to control it. Then you could mark whether the bitfield member would be C-compatible or not. If we really wanted to, we could even reuse extern(C) / extern(C++) (which would then work well with files putting extern(C): at the top like we often do in druntime), though I don't know how desirable that is. Still, if we have a way to tell the compiler whether we want bitfields to be C-compatible or not, then we can have the C behavior where we need it and better behavior for D everywhere else. And it seems like it would be kind of bad to give the C-compatible version the nice syntax for bitfields while requiring that D code that wants consistent behavior use std.bitmanip's bitfelds instead, since that's going to encourage folks to use the C bitfields when they arguably shouldn't be. We need them for compatibility, but if at all reasonably possible, we should be doing the better thing with normal D code. - Jonathan M Davis
Mar 08
next sibling parent reply Arafel <er.krali gmail.com> writes:
On 8/3/24 10:10, Jonathan M Davis wrote:
 Well, I guess that structs don't technically have extern(C) or extern(D) on
 them, which makes this harder (structs and classes do with extern(C++), but
 that doesn't help with C). Otherwise, the obvious thing would be to just use
 the C behavior in structs / classes that are extern(C) or extern(C++) and
 use well-defined, cross-platform behavior with extern(D) types, but it's the
 functions that are extern(*), not the types.
Isn't that already a solved problem? What happens currently when an `extern(D)` struct contains an `extern(C++)` class as a member? Or both `extern(D)` and `extern(C++)` classes? ```d struct S { extern(C++) class C {} extern(D) class D {} C c; D d; } void main() { S s; } ``` I don't see why we can't have something similar with `extern(D)` and `extern(C)` / `extern(C++)` bit fields.
Mar 08
parent Walter Bright <newshound2 digitalmars.com> writes:
The original design of D allowed the compiler to lay out struct fields as it
saw 
fit, and extern(C) had the C layout. This turned out to be rather pointless,
and 
I dropped it.
Mar 08
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
To clarify a bit, C allows ints to be 18 bits (e.g. for a PDP-10). But D
defines 
them as 32 bits. Technically, we're crazy to do that. But practically, we can 
rely on the people who implement C compilers to be sane, and CPU makers have 
long accepted the idea that ints are 32 bits. We are on very safe ground there.

Even chars being signed by default, although popular in the 80s, has
disappeared 
from modern compilers.

We simply do not have to worry about the C bitfield layout being implementation 
defined. No sane person is going to invent another scheme for a C compiler if
at 
all possible.

There are two dominant schemes today, VC and gcc. I'm not aware of any other 
scheme for a modern C compiler (Digital Mars C emulates the VC layout). Even
so, 
VC and gcc lay things out the same way for `int` bit fields. This is never
going 
to change.

Just stick with int bit fields and some common sense, and you're code will be
fine.

Much of D's design (and many other languages' design) is predicted on C
behavior 
not being implementation defined, even though it technically is.

There's no need to solve a non-existent problem.

(Yes, I know, there is a version of D that has 16 bit ints. But that's a
special 
build of the compiler, and it is technically not a D compiler :-/ and the
people 
who use it understand that and it isn't a big problem for them.)
Mar 08
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 3/8/24 20:52, Walter Bright wrote:
 
 There are two dominant schemes today, VC and gcc. I'm not aware of any 
 other scheme for a modern C compiler (Digital Mars C emulates the VC 
 layout). Even so, VC and gcc lay things out the same way for `int` bit 
 fields. This is never going to change.
Well, if you believe this, why not specify the behavior? Similarly, if you believe a specced-out `extern(D)` vs a "companion compiler" `extern(C)` would not make a difference anyway, why is it a concern to you that they could be accidentally confused? And finally, why not just require an _explicit_ specification of the layout on the D side? E.g., something like: ```d int x { int a : 2; int b : 18; int c : 11; int : 1; // sign bit reserved } ``` (x.a, x.b, etc.) or perhaps even: ```d int { int a : 2; int b : 18; int c : 11; int : 1; // sign bit reserved } ``` Then make the compiler enforce that the fields add up to the size of the type. By default this is compatible with what VC and GCC do, and ImportC can just uniformly translate the imported C code to such an explicit layout spec. It will even be intuitive. Just make it portable across machines with the same endianness, like the built-in integer types already are. This optimizes the user experience at the same time: It will usually be obvious how big a struct is, there will not be a need to add together the sizes of bit fields in order to figure out how things are laid out in memory. In any case, the introspection story needs to be fixed. (Even just for the sake of introspecting structs originating in ImportC code.) This would be so much better and improve ImportC at the same time.
Mar 17
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Thank you for your thoughts on this. I've thought a lot about it, too!

In the very rare case of actually needing an exact layout, with 5 minutes of 
effort any layout can be duplicated with C bit fields. If that's too much 
effort, one can write explicit shift/mask code, and encapsulate it into a 
function. (Once the function is inlined, there is zero extra overhead for it.) 
Or one can use std.bitmanip to do that automatically.

Yes, we can add extra syntax and semantics to make this controllable by the 
user, but it's so rarely needed it's over-engineering.

The preceding is true if one wants a specific layout.

But to algorithmically match what the associated C compiler does, that is more 
of a significant problem. It took me a while (with the help of Iain) to 
reverse-engineer it. This is a problem worth solving, and our implementation 
does solve it. For the user, it effortlessly resolves the issue of binary 
interoperability with C code.

Good language design means building things into the core language that are the 
most common use cases, and provide facilities for the uncommon cases. This is 
why slices are builtin, rather than done with metaprogramming.

An earlier version of D had complex numbers builtin. This did have some 
advantages over `struct Complex`. But Andrei told me it wasn't worth it. I 
eventually agreed with him, and took it out. But, for legacy compatibility,
that 
had a very long deprecation period.

Let's spend our limited complexity budget on problems that need to be solved, 
and explicit bitfield layout isn't one of them.
Apr 22
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 23/04/2024 4:16 AM, Walter Bright wrote:
 In the very rare case of actually needing an exact layout, with 5 
 minutes of effort any layout can be duplicated with C bit fields. If 
 that's too much effort, one can write explicit shift/mask code, and 
 encapsulate it into a function. (Once the function is inlined, there is 
 zero extra overhead for it.) Or one can use std.bitmanip to do that 
 automatically.
 
 Yes, we can add extra syntax and semantics to make this controllable by 
 the user, but it's so rarely needed it's over-engineering.
It depends so heavily on what you are doing, as I'm sure you know. If you work with file formats, IPC (sockets, pipes ext.), CPU flags for embedded devices, or performing instruction encoding for fun, you'll end up dealing with this stuff heavily enough that you want language support. If all you need it for is some flags in process then of course it appears to be over-engineering. I'm in the former group often enough that it is pretty uncomforting for me to see that this feature won't exist in those circumstances and this thread shows that I am not alone in this.
Apr 22
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/22/2024 10:28 AM, Richard (Rikki) Andrew Cattermole wrote:
 If you work with file formats, IPC (sockets, pipes ext.), CPU flags for
embedded 
 devices, or performing instruction encoding for fun, you'll end up dealing
with 
 this stuff heavily enough that you want language support.
std.bitmanip.bitfields are the appropriate solution for that. BTW, the backend does a lot of instruction encoding :-) I'd prefer it was written using bitfields, it would make the code more readable. I'm sure I wouldn't have any difficulty using C bitfields for it. I'm sure you wouldn't, either.
Apr 22
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 23/04/2024 12:38 PM, Walter Bright wrote:
 On 4/22/2024 10:28 AM, Richard (Rikki) Andrew Cattermole wrote:
 If you work with file formats, IPC (sockets, pipes ext.), CPU flags 
 for embedded devices, or performing instruction encoding for fun, 
 you'll end up dealing with this stuff heavily enough that you want 
 language support.
std.bitmanip.bitfields are the appropriate solution for that.
This is something I do not want brought over to PhobosV3. Due to its reliance on CTFE, tooling cannot work with it. Its a pretty horrible thing to use I've found. I do not recommend using string mixins to introduce symbols that then get interacted with by the user. Same with mixin templates (I actually need to check to see if it supports eponymous at some point, naming an instance gets you to a lot of cases without the downside of a string mixin). Anything that uses string mixins to create symbols for a user to use, is likely to always cause usability issues for people. Plus with this DIP, the equivalent functionality is in the language, why would we want to duplicate where people need to look for it and have to remember the difference between them in terms of syntax?
 BTW, the backend does a lot of instruction encoding :-)
 
 I'd prefer it was written using bitfields, it would make the code more 
 readable. I'm sure I wouldn't have any difficulty using C bitfields for 
 it. I'm sure you wouldn't, either.
I used that as an example, although it would only really be of use when you're above a register in size. Being able to cast straight to a ``ubyte[]`` and then send off to something else (or the reverse) is a mighty useful thing to have since the cast is free. Both issues of defining LSB..MSB for multibyte along with introspection can be added later without a DIP. So this shouldn't hold it up, I'm giving push back because I know I'll be explaining all this at some point and I'd prefer to not have to.
Apr 22
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 4/22/24 18:16, Walter Bright wrote:
 
 In the very rare case of actually needing an exact layout, with 5 minutes of
effort any layout can be duplicated with C bit fields.
Well, if someone is not aware that a gun can shoot one's own foot, they might do it even if they very well know it is not good to hurt one's foot.
 If that's too much effort, one can write explicit shift/mask code, and
encapsulate it into a function. (Once the function is inlined, there is zero
extra overhead for it.) Or one can use std.bitmanip to do that automatically. 
I suppose you could match C bitfields with a string mixin too. Or the compiler could automatically generate a portable specification corresponding to the C bitfield layout.
 The preceding is true if one wants a specific layout.
 
 But to algorithmically match what the associated C compiler does, that 
 is more of a significant problem. It took me a while (with the help of 
 Iain) to reverse-engineer it. This is a problem worth solving, and our 
 implementation does solve it. For the user, it effortlessly resolves the 
 issue of binary interoperability with C code.
Yes, I agree with this. However, there is a difference between having this feature and making it the _default representation_ of bit fields. It's just about how to best expose that feature for the best overall ergonomics, including introspection.
May 01
prev sibling parent Tim <tim.dlang t-online.de> writes:
On Thursday, 7 March 2024 at 22:28:34 UTC, Dennis wrote:
 Why bow down to C and treat D as a second class citizen? The 
 DIP should really substantiate this alleged importance of 
 C-compatibility of non-ImportC D bitfields: Can you name at 
 least one dub package that would benefit from bit fields with 
 C-compatible layout?
Qt uses bitfields in some headers, e.g. here: https://github.com/qt/qtbase/blob/4e158f6bfa7d0747d8da70b3b15a44b52e35bb8a/src/corelib/kernel/qtimer.h#L239 My bindings emulate the bitfields with a single field and accessor functions: https://github.com/tim-dlang/dqt/blob/2afa4adcec0a6905b11256f403c6136277a6c96a/core/qt/core/timer.d#L217 Bitfields in D could be used instead, but they would need to be compatible with C++ for extern(C++) types. Sometimes bitfield members are disabled with the preprocessor: https://github.com/qt/qtbase/blob/4e158f6bfa7d0747d8da70b3b15a44b52e35bb8a/src/corelib/time/qdatetime.h#L259 This could be translated to version statements in D. Most of the times bitfields for Qt are not a big problem, because they are only private members of the types.
Mar 09
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 3/7/24 20:42, Walter Bright wrote:
 
 I.e. the layout is implementation-defined. That's correct. But in 
 reality, it is fixed. There is no way that gcc on Linux is *ever* going 
 to change the bit field layout, because it would break legions of 
 existing code. It is cast in concrete, and is as immutable as it gets. 
 The same for Microsoft C.
 
 Furthermore, if you use `int` as the base type for bit fields, the 
 layouts are all the same in all the C compilers I've encountered. For 
 practical purposes, it is defined.
 
 If you are still concerned about the layout, like you want a portable 
 file format, don't use bit fields. Use std.bitmanip.
Well, that's kind of a foot gun. Portable by default is better.
Mar 16
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/16/2024 2:32 PM, Timon Gehr wrote:
 On 3/7/24 20:42, Walter Bright wrote:
 If you are still concerned about the layout, like you want a portable file 
 format, don't use bit fields. Use std.bitmanip.
Well, that's kind of a foot gun. Portable by default is better.
One thing about language design - there is no rule that overrides every other rule. This applies to everything. Every design is a balance of competing interests.
Apr 22
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
 The usual way to do bitfields is (culled from the DMD compiler 
 source):
 ```D
 struct S
 {
     enum Flags : uint
     {
         isnothrow       = 0x0001, // nothrow
         isnogc          = 0x0002, // is  nogc
         isproperty      = 0x0004, // can be called without 
 parentheses
         isref           = 0x0008, // returns a reference
         isreturn        = 0x0010, // 'this' is returned by ref
         isscope         = 0x0020, // 'this' is scope
         isreturninferred= 0x0040, // 'this' is return from 
 inference
         isscopeinferred = 0x0080, // 'this' is scope from 
 inference
         islive          = 0x0100, // is  live
         incomplete      = 0x0200, // return type or default 
 arguments removed
         inoutParam      = 0x0400, // inout on the parameters
         inoutQual       = 0x0800, // inout on the qualifier
     }
 
     Flags flags;
 }
 ```
Granted that this way is pretty common, but it's far from optimal. `std.bitmanip.bitfields` does this a lot better, even more so when you have multiple-bit fields. Yes, as the DIP says it lacks C compatibility guarantees, but so does this solution. This example is thus a bit of a straw man. Please just supply an example with a standard library bitfield. In the general use case, portability trumps C compatibility. Therefore, we should and will use `std.bitmanip` bitfields when we don't explicitly have to be C compatible. What the DIP is suggesting should be considered a special bitfield type, not the "normal" case. I do not think any syntactic additions to the language itself are a good idea. We don't have any special syntax for normal bitfields either so it would be strange for C-compatible bitfields to have that. It'd be better that C bitfields would be declared with a mixin somewhere in DRuntime, its usage syntax preferably as close as possible to `std.bitmanip.bitfields`. The compiler can still recognise the call of the mixin as special, but from user POV it should be no different from any other mixin.
Mar 07
prev sibling next sibling parent Nick Treleaven <nick geany.org> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
 One cannot take a reference to or the address of a bitfield
Taking a ref is not detected yet: https://issues.dlang.org/show_bug.cgi?id=24263 I was wondering how to read/write the bits of a bitfield set as their underlying type. So I tried putting a bitfield set inside an anonymous union: ```d import std.stdio; struct S { union { uint u; uint a:1, b:1, c:1; } } static assert(S.sizeof == 4); void main() { S s; s.tupleof.writeln(); // 0000 s.u++; s.tupleof.writeln(); // 1111 } ``` I was a bit surprised that setting `u` to 1 sets all 3 bitfields to 1 as well - I was assuming the union had 2 uint members where the 2nd one didn't have a name. Then I realized I needed to put the bitfields inside an anonymous struct, because otherwise each one was a union member. Anyway doing that does work.
Mar 09
prev sibling next sibling parent reply Martin Tschierschke <mt smartdolphin.de> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Typo:
Abstract
This proposal is for supporting bitfields in **C** the same way 
they are supported in **C**.
Mar 11
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/11/2024 2:46 AM, Martin Tschierschke wrote:
 On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Typo:
 Abstract
 This proposal is for supporting bitfields in **C** the same way they are 
 supported in **C**.
haha
Mar 11
parent reply Martin Tschierschke <mt smartdolphin.de> writes:
On Monday, 11 March 2024 at 17:57:58 UTC, Walter Bright wrote:
 On 3/11/2024 2:46 AM, Martin Tschierschke wrote:
 On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Typo:
 Abstract
 This proposal is for supporting bitfields in **C** the same 
 way they are supported in **C**.
haha
Sorry, but this is an ERROR on __your__ page, not a joke.
Mar 12
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, March 12, 2024 1:31:49 AM MDT Martin Tschierschke via 
dip.development wrote:
 On Monday, 11 March 2024 at 17:57:58 UTC, Walter Bright wrote:
 On 3/11/2024 2:46 AM, Martin Tschierschke wrote:
 On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Typo:
 Abstract
 This proposal is for supporting bitfields in **C** the same
 way they are supported in **C**.
haha
Sorry, but this is an ERROR on __your__ page, not a joke.
I expect that he was just laughing at his mistake. - Jonathan M Davis
Mar 12
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/6/2024 8:26 PM, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Permanent link: https://github.com/WalterBright/documents/blob/9b91ca5242e3e49ed807ee696675c078986aa51e/bitfields.md
Mar 14
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 3/7/24 05:26, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Still not a fan of this potentially not being portable between different compilers implementing the C calling convention on the same platform. But let's put that aside for a moment. Let's say D bitfields are stored in memory in a non-portable way. How would a serialization library detect via introspection that these are bitfields and serialize them accordingly (in a portable way)? In general, how to introspect on bitfields?
Mar 16
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/16/2024 2:40 PM, Timon Gehr wrote:
 In general, how to introspect on bitfields?
Currently, allMembers will get you the name of the bitfield, and you can use .offsetof and .alignof on it. The number of bits can be derived from the .max property. You'll know it's a bitfield if the .max property is less than the type.max. From those, the bit offsets can be determined, but one could reasonably ask for an easier way to get that.
Apr 21
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
Most of the complaints by the seasoned pros here are of the form: C bitfields are an implementation-defined mess, and we shouldn't copy them. I don't think this means we can't copy the *good parts*. If I understand correctly, it's because C can do odd things with corner cases, not because it's doing insane things like storing bitfields in every other bit, etc. So if "almost all C compilers" do the same thing, why not just define that thing, and say D does it this way? Why say we depend 100% on C compiler implementation whims? If the corner cases are problematic, ban the corner cases? Pick the "right way", define it, and say "This works for most cases in all the C compilers we use, and if it doesn't, you have to figure it out yourself". I don't understand the problem with that. ImportC bitfields *don't have to be the same*. As you once told me, bitfields are not generally used for public API, because of the complications. So why do we need C compatibility in the first place? C does some really dumb things (hindsight, etc.), and D didn't copy all those dumb things. We should continue to not do dumb things. Bitfields should have `__traits` to go with them (e.g. `__traits(bitSize, x)` should return `4` for a 4-bit value). There is zero reason to hide what the compiler knows. I don't know if there are any more things needed, as you can use `typeof` to get the underlying type, and getting an address to the full bitfield can be done with a union. There are other things that aren't specified in the proposal, such as the properties that work on other basic types: `sizeof`, `offsetof`, `alignof`. -Steve
Mar 22
next sibling parent Daniel N <no public.email> writes:
On Saturday, 23 March 2024 at 02:32:20 UTC, Steven Schveighoffer 
wrote:
 So if "almost all C compilers" do the same thing, why not just 
 define that thing, and say D does it this way? Why say we 
 depend 100% on C compiler implementation whims? If the corner 
 cases are problematic, ban the corner cases?
Walter already replied to my identical suggestion: https://forum.dlang.org/post/usnh0c$2180$1 digitalmars.com
Mar 23
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/22/2024 7:32 PM, Steven Schveighoffer wrote:
 So if "almost all C compilers" do the same thing, why not just define that 
 thing, and say D does it this way? Why say we depend 100% on C compiler 
 implementation whims? If the corner cases are problematic, ban the corner
cases?
I don't see an actual use case for our own layout. They may indeed be whims on the part of the C compiler, but we deliberately pick which C compiler is the "associated C compiler", and why pick a nutburger compiler?
 Pick the "right way", define it,
There isn't a right way.
 and say "This works for most cases in all the C 
 compilers we use, and if it doesn't, you have to figure it out yourself". I 
 don't understand the problem with that. ImportC bitfields *don't have to be
the 
 same*. As you once told me, bitfields are not generally used for public API, 
 because of the complications. So why do we need C compatibility in the first
place?
One of the uses of bitfields in D is interoperation with C code. It's not unreasonable to have a program that is half in D and half in C. Having different bitfield layouts would just be an aggravation. It would be surprising.
 C does some really dumb things (hindsight, etc.), and D didn't copy all those 
 dumb things. We should continue to not do dumb things.
We copy the dumb way C passes arguments to functions, too. Why? Because it's convenient, easy to explain, nobody questions it, etc. D in the past had a slightly better way of calling C code, and we abandoned that because the nuisance of being different simply was not worth it. Changing it to match C made the problems and the endless effort explaining the difference just go away. The value of "it is the same as C" is quite large.
 There are other things that aren't specified in the proposal, such as the 
 properties that work on other basic types: `sizeof`, `offsetof`, `alignof`.
ok
Apr 22
prev sibling next sibling parent reply MrSmith <mrsmith33 yandex.ru> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
My list of wishes: 1. bitfields imported from C should use `extern(C)`, this way they can use whatever layout C compiler uses. 2. D bitfields should be `extern(D)`, and work like `bitfields` mixin. 3. Allow specifying default value. Here is how C3 language does bitfields: https://c3lang.github.io/c3docs/types/#bitstructs It has guaranteed layout, allows specifying endinanness, optional overlaping of fields. It allows static arrays to be used as a backing type. Its bitfields are done trough a special kind of struct - `bitstruct`.
Mar 25
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/25/2024 9:39 AM, MrSmith wrote:
 My list of wishes:
 1. bitfields imported from C should use `extern(C)`, this way they can use 
 whatever layout C compiler uses.
 2. D bitfields should be `extern(D)`, and work like `bitfields` mixin.
 3. Allow specifying default value.
 
 Here is how C3 language does bitfields: 
 https://c3lang.github.io/c3docs/types/#bitstructs
 It has guaranteed layout, allows specifying endinanness, optional overlaping
of 
 fields. It allows static arrays to be used as a backing type. Its bitfields
are 
 done trough a special kind of struct - `bitstruct`.
Thank you for the link to C3. I read it. Some problems are simply not worth solving, and C3 goes too far. You can get a consistent layout by simply declaring your bitfields as 'int'. I know this is not specified anywhere, but it works.
Apr 22
prev sibling next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:
 https://github.com/WalterBright/documents/blob/master/bitfields.md
I don't know if already mentioned. It's "workaround aura" not "workaround aurora" in ImportC paragraph of the rationale. Aurora means dawn.
Apr 02
prev sibling parent Mike Parker <aldacron gmail.com> writes:
Threads for review of drafts two and three have been opened 
subsequent to this one. Please leave further feedback on the 
thread for the third draft, linked below. This thread is now 
closed.

https://forum.dlang.org/post/v193hc$b9c$1 digitalmars.com
May 21