digitalmars.dip.development - DIP: add Bit Fields
- Walter Bright (1/1) Mar 06 https://github.com/WalterBright/documents/blob/master/bitfields.md
- Richard (Rikki) Andrew Cattermole (21/21) Mar 06 "gdc/lcd are lumped together because they produce identical results on
- Nick Treleaven (47/64) Mar 09 Agree, the DIP should mention those. The implementation seems to
- ryuukk_ (27/28) Mar 07 Have you ever thought of the idea of using bit specific integers?
- Dennis (11/14) Mar 07 [Bit-fields in C have acquired a poor reputation in the
- Walter Bright (30/45) Mar 07 All Zig is doing is providing a slightly different syntax that does the ...
- Dennis (4/5) Mar 07 You can disagree, but 'I have not seen any "do not use bitfields"
- Dennis (23/26) Mar 07 There's no need to argue that native bit fields look nicer than
- Jonathan M Davis (10/15) Mar 07 Having the ability to declare C-compatible bitfields would definitely be
- Walter Bright (2/2) Mar 07 You'd want C-compatible bit fields if/when you're building a hybrid D/C/...
- Arafel (4/6) Mar 08 In that case, why not have the best of both worlds and have `extern(D)`
- Walter Bright (3/6) Mar 08 Not a bad idea, but it may be confusing which one is in play, as the ext...
- Richard (Rikki) Andrew Cattermole (9/17) Mar 08 Since fields do not currently have block behaviors for externs, doing
- Walter Bright (10/10) Mar 08 Sometimes, too many options is not helpful. This is one of those situati...
- Daniel N (7/20) Mar 09 Fully agree. There's nothing wrong with implementation-defined.
- Walter Bright (4/8) Mar 11 I've thought about that, and it seemed more straightforward to simply re...
- Dennis (5/8) Mar 09 I've shown that it's a problem when using bit bitfields in D's
- Daniel N (4/12) Mar 09 Not on dub, but my closed source makes use of it. Vendor supplied
- Dennis (4/7) Mar 09 That would be great material for the DIP. Can you give a link to
- Daniel N (9/16) Mar 09 The sdk license doesn't allow me to share the files.
- Walter Bright (11/15) Mar 11 I presume you mean
- Dennis (13/18) Mar 11 That's great, but it's not what the DIP says currently, which is
- Walter Bright (21/32) Mar 11 D doesn't say what the alignment algorithm is, either, just that it matc...
- Richard (Rikki) Andrew Cattermole (14/14) Mar 09 I've had a bit of thought about the ABI concerns of having multiple
- Walter Bright (2/2) Mar 11 I hate to say it, but it's still a complex solution to a rare and frankl...
- Walter Bright (15/15) Mar 07 The inconsistency of C bitfield layout is far less important than it is ...
- Jonathan M Davis (33/48) Mar 08 Yeah, but with the D types, we've mostly tried to fix that (e.g. making
- Arafel (17/23) Mar 08 Isn't that already a solved problem? What happens currently when an
- Walter Bright (3/3) Mar 08 The original design of D allowed the compiler to lay out struct fields a...
- Walter Bright (20/20) Mar 08 To clarify a bit, C allows ints to be 18 bits (e.g. for a PDP-10). But D...
- Timon Gehr (38/43) Mar 17 Well, if you believe this, why not specify the behavior?
- Walter Bright (23/23) Apr 22 Thank you for your thoughts on this. I've thought a lot about it, too!
- Richard (Rikki) Andrew Cattermole (10/19) Apr 22 It depends so heavily on what you are doing, as I'm sure you know.
- Walter Bright (6/9) Apr 22 std.bitmanip.bitfields are the appropriate solution for that.
- Richard (Rikki) Andrew Cattermole (23/35) Apr 22 This is something I do not want brought over to PhobosV3.
- Timon Gehr (10/20) May 01 Well, if someone is not aware that a gun can shoot one's own foot, they
- Tim (13/18) Mar 09 Qt uses bitfields in some headers, e.g. here:
- Timon Gehr (2/15) Mar 16 Well, that's kind of a foot gun. Portable by default is better.
- Walter Bright (3/8) Apr 22 One thing about language design - there is no rule that overrides every ...
- Dukc (20/49) Mar 07 Granted that this way is pretty common, but it's far from
- Nick Treleaven (32/34) Mar 09 Taking a ref is not detected yet:
- Martin Tschierschke (2/6) Mar 11
- Walter Bright (2/12) Mar 11 haha
- Martin Tschierschke (2/14) Mar 12 Sorry, but this is an ERROR on __your__ page, not a joke.
- Jonathan M Davis (4/16) Mar 12 I expect that he was just laughing at his mistake.
- Walter Bright (3/4) Mar 14 Permanent link:
- Timon Gehr (8/9) Mar 16 Still not a fan of this potentially not being portable between different...
- Walter Bright (7/8) Apr 21 Currently, allMembers will get you the name of the bitfield, and you can...
- Steven Schveighoffer (32/33) Mar 22 Most of the complaints by the seasoned pros here are of the form:
- Daniel N (4/8) Mar 23 Walter already replied to my identical suggestion:
- Walter Bright (15/28) Apr 22 I don't see an actual use case for our own layout. They may indeed be wh...
- MrSmith (13/14) Mar 25 My list of wishes:
- Walter Bright (5/16) Apr 22 Thank you for the link to C3. I read it. Some problems are simply not wo...
- Patrick Schluter (4/5) Apr 02 I don't know if already mentioned. It's "workaround aura" not
- Mike Parker (5/5) May 21 Threads for review of drafts two and three have been opened
"gdc/lcd are lumped together because they produce identical results on the same platform." s/lcd/ldc/ No mention of alignment, total size or the offset. Needs to mention atomics as not being supported. No trait to detect a bit field (if its an alias, or by name), nor get its size. Kinda required, enough other things in the language can appear via an alias sequence or by allMembers that are not able to be taken as a pointer of. The ability to take an alias via getMember needs to be considered if its valid to do so. No mention of tupleof either, what happens there? No motivation for not including a D layout for it, instead of forcing everyone into an unpredictable layout that the system C compiler might be using. Moving us off the library type and into a language defined one that is predictable is quite attractive so there needs to be an argument against this. Also both the post and its subject do not match the guidelines for this forum. So I'm half wondering if it's even worth writing this post knowing that Mike could delete it. https://github.com/dlang/DIPs/blob/master/docs/guidelines-forums.md#dip-development
Mar 06
On Thursday, 7 March 2024 at 05:00:36 UTC, Richard (Rikki) Andrew Cattermole wrote:"gdc/lcd are lumped together because they produce identical results on the same platform." s/lcd/ldc/ No mention of alignment, total size or the offset.Agree, the DIP should mention those. The implementation seems to give each bitfield of a set the same value: ```d struct S { uint a:1, b:1, c:1; } static assert(S.b.alignof == 4); static assert(S.b.offsetof == 0); static assert(S.b.sizeof == 4); ```Needs to mention atomics as not being supported.Yes. Bitfield impl doesn't allow shared, static (or even const) storage classes.No trait to detect a bit field (if its an alias, or by name), nor get its size. Kinda required, enough other things in the language can appear via an alias sequence or by allMembers that are not able to be taken as a pointer of.Possibly this might be enough (using S above): ```d enum E { a, b, c } struct U { int i; alias i this; enum max = 2; } enum isBitfield(alias a) = __traits(isIntegral, typeof(a)) && !is(typeof(a) == enum) && a.max <= uint.sizeof * 8; static assert(isBitfield!(S.b)); static assert(!isBitfield!(E.b)); static assert(!isBitfield!(U())); ```The ability to take an alias via getMember needs to be considered if its valid to do so. No mention of tupleof either, what happens there?Impl: ```d void main() { S s; s.tupleof[1]++; __traits(getMember, s, "c")++; writeln(s.tupleof); // 011 } ```No motivation for not including a D layout for it, instead of forcing everyone into an unpredictable layout that the system C compiler might be using. Moving us off the library type and into a language defined one that is predictable is quite attractive so there needs to be an argument against this.Just to mention, the argument in the DIP was portability with C bitfields.
Mar 09
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdHave you ever thought of the idea of using bit specific integers? I once saw this one suggestion on the IRC channel: `alias i1 = __traits(integer, signed: false, 1);` ```D alias u1 = __traits(integer, true, 1); alias u3 = __traits(integer, true, 3); alias u5 = __traits(integer, true, 5); align(1) struct S { u1 isnothrow; // nothrow u1 isnogc; // is nogc u1 isproperty; // can be called without parentheses u1 isref; // returns a reference u1 isreturn; // 'this' is returned by ref u1 isscope; // 'this' is scope u1 isreturninferred; // 'this' is return from inference u1 isscopeinferred; // 'this' is scope from inference u1 islive; // is live u1 incomplete; // return type or default arguments removed u1 inoutParam; // inout on the parameters u1 inoutQual; // inout on the qualifier u5 a; u3 flags; } ```
Mar 07
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdI've seen plentiful use of it, and have not seen any "do not use bitfields" programming guidelines.[Bit-fields in C have acquired a poor reputation in the programming community](https://andrewkelley.me/post/a-better-way-to-implement-bit-fields.html) [C bitfields considered harmful](https://www.flamingspork.com/blog/2014/11/14/c-bitfields-considered-harmful/) [So don't go for bitfields "just because"](https://yarchive.net/comp/linux/bitfields.html) [Bitfields seem to be perfect for this, but they're so poorly specified in the standard that common "best practice" is to avoid them.](https://www.reddit.com/r/C_Programming/comments/146tbnp/why_is_using_bitfields_in_high_performance/)
Mar 07
On 3/7/2024 4:04 AM, Dennis wrote:On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:All Zig is doing is providing a slightly different syntax that does the same thing as C. Zig is layering on a magic "pointer to bit field type" that I attempted to implement in D long ago, but it has too many special cases and turned out to be unworkable. You cannot convert a pointer to bit 3 to a void*, making the special bit pointer type rather useless.https://github.com/WalterBright/documents/blob/master/bitfields.mdI've seen plentiful use of it, and have not seen any "do not use bitfields" programming guidelines.[Bit-fields in C have acquired a poor reputation in the programming community](https://andrewkelley.me/post/a-better-way-to-implement-bit-fields.html)[C bitfields considered harmful](https://www.flamingspork.com/blog/2014/11/14/c-bitfields-considered-harmful/)I don't know of any C compiler that puts a single bit flag in a 64 bit int. He didn't show enough code to see the source of his problem. I suspect he doesn't know what he's doing.[So don't go for bitfields "just because"](https://yarchive.net/comp/linux/bitfields.html)Linus suggests using explicit bit masks instead.[Bitfields seem to be perfect for this, but they're so poorly specified in the standard that common "best practice" is to avoid them.](https://www.reddit.com/r/C_Programming/comments/146tbnp/why_is_using_bitfields_in_high_performance/)I.e. the layout is implementation-defined. That's correct. But in reality, it is fixed. There is no way that gcc on Linux is *ever* going to change the bit field layout, because it would break legions of existing code. It is cast in concrete, and is as immutable as it gets. The same for Microsoft C. Furthermore, if you use `int` as the base type for bit fields, the layouts are all the same in all the C compilers I've encountered. For practical purposes, it is defined. If you are still concerned about the layout, like you want a portable file format, don't use bit fields. Use std.bitmanip. But if you want to be compatible with C code that uses bit fields, the builtin bit fields are the way to go. If you like a nice syntax, bit fields cannot be beat. If you want to pack data in your structs and use less memory for your program, bit fields are again a winner. DMD uses far too much memory. Much of the data structs it uses can be packed, but since without bit fields the syntax is just too ugly, there's been little effort to do that. Convenience is everything. If you still don't like bit fields, don't use them. And yes, you can't do volatile, shared, atomic, or pointers to bit fields. You can't do those with std.bitmanip, either. Nor can you do them with manual masking and shifting.
Mar 07
On Thursday, 7 March 2024 at 19:42:36 UTC, Walter Bright wrote:...You can disagree, but 'I have not seen any "do not use bitfields" programming guidelines' (which wasn't a strong argument to begin with) is now false, so I suggest you replace it in the DIP.
Mar 07
On Thursday, 7 March 2024 at 19:42:36 UTC, Walter Bright wrote:If you like a nice syntax, bit fields cannot be beat.There's no need to argue that native bit fields look nicer than manual bit shifts. Of course they do.And yes, you can't do volatile, shared, atomic, or pointers to bit fields.Of course you can't. No need to defend this. The main question is: Why inherit C's controversial inconsistent layout scheme? The DIP keeps going on about C interoperability, but only cites abstract "experience" as an argument. My last use of bit fields was from optimizing `std.uni` tables: https://github.com/dlang/phobos/pull/8880 Wouldn't it be nice if I could use the shiny new bit fields feature for that? Well I can't: the table needs to know the bit offsets, which would be target dependent. I'd have to use the inferior `std.bitmanip`. If I didn't know about the platform dependence, I'd use bit fields incorrectly. It would all work until compiling to a different target results in a corrupted table, without warning. After debugging and finding the cause I'd write a blog post "D bit fields considered harmful" like C users have done before. Why bow down to C and treat D as a second class citizen? The DIP should really substantiate this alleged importance of C-compatibility of non-ImportC D bitfields: Can you name at least one dub package that would benefit from bit fields with C-compatible layout?
Mar 07
On Thursday, March 7, 2024 3:28:34 PM MST Dennis via dip.development wrote:Why bow down to C and treat D as a second class citizen? The DIP should really substantiate this alleged importance of C-compatibility of non-ImportC D bitfields: Can you name at least one dub package that would benefit from bit fields with C-compatible layout?Having the ability to declare C-compatible bitfields would definitely be useful for binding some C symbols. I was recently dealing with some symbols on Windows where I basically had to skip the bitfields when writing the bindings, because I don't know of any way to bind to them in D. That being said, I see no reason for bitfields for actual D code to be C-compatible. Unless I'm actually binding to a C type, C-compatibility is irrelevant, and if what C is doing isn't portable, then why would I want it for my D code? - Jonathan M Davis
Mar 07
You'd want C-compatible bit fields if/when you're building a hybrid D/C/C++ binary, like gdc and ldc are.
Mar 07
On 8/3/24 3:38, Walter Bright wrote:You'd want C-compatible bit fields if/when you're building a hybrid D/C/C++ binary, like gdc and ldc are.In that case, why not have the best of both worlds and have `extern(D)` (the default, well-defined, perhaps `std.bitmanip`-compatible) and `extern(C)` bit fields, like with classes and C++?
Mar 08
On 3/8/2024 1:02 AM, Arafel wrote:In that case, why not have the best of both worlds and have `extern(D)` (the default, well-defined, perhaps `std.bitmanip`-compatible) and `extern(C)` bit fields, like with classes and C++?Not a bad idea, but it may be confusing which one is in play, as the extern(C) can be far removed from its effect.
Mar 08
On 09/03/2024 8:28 AM, Walter Bright wrote:On 3/8/2024 1:02 AM, Arafel wrote:Since fields do not currently have block behaviors for externs, doing this would be a new pattern in the language usage. Otherwise per encapsulation/scope setting of extern may be required. Which is already used, but as it would be required this is new. Either way as per my original comment it needs to be argued which ever way you go, the cost of implementing our own layout shouldn't be an issue considering how many others will be needed > 1. I would love to see this replace the library solution.In that case, why not have the best of both worlds and have `extern(D)` (the default, well-defined, perhaps `std.bitmanip`-compatible) and `extern(C)` bit fields, like with classes and C++?Not a bad idea, but it may be confusing which one is in play, as the extern(C) can be far removed from its effect.
Mar 08
Sometimes, too many options is not helpful. This is one of those situations. The problem with implementation-defined layout is much more of a pedantic one than a pragmatic one, as I've explained elsewhere in this thread. If one sticks with `int` bit fields, they behave the same across C compilers that I've seen. Just like ints are 32 bits. We don't have an attribute to switch endianess. It's just not worth the added complexity. If anyone is worried about exactly specifying the layout, using masking and shifting will guarantee it (absent endianess issues!). We do not need to solve every detail, just go for the critical ones.
Mar 08
On Saturday, 9 March 2024 at 07:51:08 UTC, Walter Bright wrote:Sometimes, too many options is not helpful. This is one of those situations. The problem with implementation-defined layout is much more of a pedantic one than a pragmatic one, as I've explained elsewhere in this thread. If one sticks with `int` bit fields, they behave the same across C compilers that I've seen. Just like ints are 32 bits. We don't have an attribute to switch endianess. It's just not worth the added complexity. If anyone is worried about exactly specifying the layout, using masking and shifting will guarantee it (absent endianess issues!). We do not need to solve every detail, just go for the critical ones.Fully agree. There's nothing wrong with implementation-defined. I guess it might be possible to make an even stronger guarantee, specifying that the current behavior is how it must work on windows and unix in order to be compliant with the D language spec. The same way we decided to define the common practise for integer wrapping and two's complement.
Mar 09
On 3/9/2024 12:38 AM, Daniel N wrote:I guess it might be possible to make an even stronger guarantee, specifying that the current behavior is how it must work on windows and unix in order to be compliant with the D language spec. The same way we decided to define the common practise for integer wrapping and two's complement.I've thought about that, and it seemed more straightforward to simply rely on the specified behavior of the associated C compiler. It's like D doesn't specify how floating point arithmetic works, just that it conforms to IEEE754.
Mar 11
On Saturday, 9 March 2024 at 07:51:08 UTC, Walter Bright wrote:The problem with implementation-defined layout is much more of a pedantic one than a pragmatic one, as I've explained elsewhere in this thread.I've shown that it's a problem when using bit bitfields in D's very own standard library (std.internal.unicode_tables) and asked you to name one dub package that would benefit from D bit fields having C layout, which you didn't do.
Mar 09
On Saturday, 9 March 2024 at 09:41:22 UTC, Dennis wrote:On Saturday, 9 March 2024 at 07:51:08 UTC, Walter Bright wrote:Not on dub, but my closed source makes use of it. Vendor supplied register definitions uses bitfields with C semantics. Also IPC where client/server is written in different languages.The problem with implementation-defined layout is much more of a pedantic one than a pragmatic one, as I've explained elsewhere in this thread.I've shown that it's a problem when using bit bitfields in D's very own standard library (std.internal.unicode_tables) and asked you to name one dub package that would benefit from D bit fields having C layout, which you didn't do.
Mar 09
On Saturday, 9 March 2024 at 10:46:10 UTC, Daniel N wrote:Not on dub, but my closed source makes use of it. Vendor supplied register definitions uses bitfields with C semantics. Also IPC where client/server is written in different languages.That would be great material for the DIP. Can you give a link to such a register definition, and show what your current approach is to create C compatible bit fields for it in D?
Mar 09
On Saturday, 9 March 2024 at 10:56:33 UTC, Dennis wrote:On Saturday, 9 March 2024 at 10:46:10 UTC, Daniel N wrote:The sdk license doesn't allow me to share the files. But you can find similar code here... https://www.renesas.com/us/en/document/apn/sh7450-groupsh7451-group-register-definition-header-file Typically an union with a word sized member is combined with the bitfields to provide either atomic access or convenient access to the fields. Without native D support, I just use C for the lowest layer instead.Not on dub, but my closed source makes use of it. Vendor supplied register definitions uses bitfields with C semantics. Also IPC where client/server is written in different languages.That would be great material for the DIP. Can you give a link to such a register definition, and show what your current approach is to create C compatible bit fields for it in D?
Mar 09
On 3/9/2024 1:41 AM, Dennis wrote:I've shown that it's a problem when using bit bitfields in D's very own standard library (std.internal.unicode_tables)I presume you mean ``` ulong x; // bit field sizes: 18, 12, 12, 4, 4, 4 ``` https://github.com/dlang/phobos/pull/8880/files#diff-6649807e195867988f8292561430bf14abf6c9cf4cecbea53c7ac2fd436c3623R145 which is 64 bits. That should produce the same result across different C compilers, i.e. it shouldn't be a problem. If I'm missing what the actual problem you've encountered is, please explain.and asked you to name one dub package that would benefit from D bit fields having C layout, which you didn't do.The D compiler itself would, so using it in the D front end will effortlessly work with the C++ gdc and ldc backends.
Mar 11
On Monday, 11 March 2024 at 18:12:58 UTC, Walter Bright wrote:On 3/9/2024 1:41 AM, Dennis wrote: which is 64 bits. That should produce the same result across different C compilers, i.e. it shouldn't be a problem.That's great, but it's not what the DIP says currently, which is that the layout (for non-int) is implementation defined matching a C compiler. It makes the comparison with C's varying `int` size, but the difference is that in that case, D sticks to its guns and keeps `int` 32-bits even when C would make it 16-bit. Perhaps the DIP could say: the layout in D is defined as X, which follows what C compilers do in practice. Then programmers can be comfortable relying on the layout.The D compiler itself would, so using it in the D front end will effortlessly work with the C++ gdc and ldc backends.Those are extern(C++) classes, and they already have working bitfields. They wouldn't be impaired by making bit fields in D consistent and predictable.
Mar 11
On 3/11/2024 11:49 AM, Dennis wrote:That's great, but it's not what the DIP says currently, which is that the layout (for non-int) is implementation defined matching a C compiler.It's the same story for longs. I had neglected to mention that in the DIP.It makes the comparison with C's varying `int` size, but the difference is that in that case, D sticks to its guns and keeps `int` 32-bits even when C would make it 16-bit. Perhaps the DIP could say: the layout in D is defined as X, which follows what C compilers do in practice.D doesn't say what the alignment algorithm is, either, just that it matches the associated C compiler. It's always been that way, although it is not an accident. It was deliberate on my part. Nobody seems to notice this, it "just works", as it should. As for 16 bit int D, that is a variant of D, not D. (It should perhaps be called "D16".) This is perfectly reasonable, because you will never be able to port non-trivial D code to D16 without major alterations (DMD itself will never be portable to 16 bit code). This frees the 99.99% of D programmers to not feel a need to contort their code to be "Standard-conformant" to 16 bit targets. I see the pointless self-torture of C programmers doing this all the time. (And in spite of these efforts, the C code is still not portable to those other unusual platforms.) C would be better off if they took the same approach.Then programmers can be comfortable relying on the layout.They already can rely on it, in practice.There are currently no bit fields in the DMD front end source code, although there are some emulations of bit fields.The D compiler itself would, so using it in the D front end will effortlessly work with the C++ gdc and ldc backends.Those are extern(C++) classes, and they already have working bitfields.They wouldn't be impaired by making bit fields in D consistent and predictable.They are consistent and predictable now. Just like the D and C++ code reliance on ints being 32 bits, the endianess, and the alignment layout of the fields in the structs.
Mar 11
I've had a bit of thought about the ABI concerns of having multiple layouts possible. I think its better to ignore the ABI altogether when changing it. ```d import core.attributes : bitfieldlayout; struct Foo { bitfieldlayout("packed") { int bar:1; int zah:2; } } ``` That'll also solve how do we add new ones that isn't C or packed. C can be default, that need not be changed to something else.
Mar 09
I hate to say it, but it's still a complex solution to a rare and frankly trivial issue. With some care, one can write portable C bitfields.
Mar 11
The inconsistency of C bitfield layout is far less important than it is made out to be. Other inconsistencies C has: char/short/int/long sizes, float/double/longdouble behaviors, 1s or 2s complement arithmetic, field alignment and endianness. Even the signed-ness of chars is implementation-defined. D matches that behavior of the associated C compiler. Living with that is normal. The gdc and ldc backends are written in C++. D bitfields would enable bitfields to be used in the D front end, and the backends can use them effortlessly (should be no problem for dtoh). An important feature of D is interoperability with C (and to some extent C++). Reducing the friction as much as we reasonably can will lower the barriers to D adoption. It means D code can co-exist peacefully in the same binary. When gdc and ldc are ported to a different platform, with different C bitfield layouts, it will still *effortlessly* work correctly. I like effortlessly.
Mar 07
On Thursday, March 7, 2024 7:37:02 PM MST Walter Bright via dip.development wrote:The inconsistency of C bitfield layout is far less important than it is made out to be. Other inconsistencies C has: char/short/int/long sizes, float/double/longdouble behaviors, 1s or 2s complement arithmetic, field alignment and endianness. Even the signed-ness of chars is implementation-defined. D matches that behavior of the associated C compiler. Living with that is normal.Yeah, but with the D types, we've mostly tried to fix that (e.g. making char/short/int/long have fixed sizes), and D defines char, wchar, and dchar all as being unsigned. We have enough compatibility to define aliases for the C types and use those in bindings, but normal D code doesn't have to deal with the issue. bviously, there are some cases where we're forced to do the same as C even when we might prefer not to, but we have managed to avoid some of C's mistakes in normal D code in a number of cases while still having extern(C) work.The gdc and ldc backends are written in C++. D bitfields would enable bitfields to be used in the D front end, and the backends can use them effortlessly (should be no problem for dtoh). An important feature of D is interoperability with C (and to some extent C++). Reducing the friction as much as we reasonably can will lower the barriers to D adoption. It means D code can co-exist peacefully in the same binary. When gdc and ldc are ported to a different platform, with different C bitfield layouts, it will still *effortlessly* work correctly. I like effortlessly.Well, I guess that structs don't technically have extern(C) or extern(D) on them, which makes this harder (structs and classes do with extern(C++), but that doesn't help with C). Otherwise, the obvious thing would be to just use the C behavior in structs / classes that are extern(C) or extern(C++) and use well-defined, cross-platform behavior with extern(D) types, but it's the functions that are extern(*), not the types. However, the next thing that comes to mind would be to use a pragma or attribute to control it. Then you could mark whether the bitfield member would be C-compatible or not. If we really wanted to, we could even reuse extern(C) / extern(C++) (which would then work well with files putting extern(C): at the top like we often do in druntime), though I don't know how desirable that is. Still, if we have a way to tell the compiler whether we want bitfields to be C-compatible or not, then we can have the C behavior where we need it and better behavior for D everywhere else. And it seems like it would be kind of bad to give the C-compatible version the nice syntax for bitfields while requiring that D code that wants consistent behavior use std.bitmanip's bitfelds instead, since that's going to encourage folks to use the C bitfields when they arguably shouldn't be. We need them for compatibility, but if at all reasonably possible, we should be doing the better thing with normal D code. - Jonathan M Davis
Mar 08
On 8/3/24 10:10, Jonathan M Davis wrote:Well, I guess that structs don't technically have extern(C) or extern(D) on them, which makes this harder (structs and classes do with extern(C++), but that doesn't help with C). Otherwise, the obvious thing would be to just use the C behavior in structs / classes that are extern(C) or extern(C++) and use well-defined, cross-platform behavior with extern(D) types, but it's the functions that are extern(*), not the types.Isn't that already a solved problem? What happens currently when an `extern(D)` struct contains an `extern(C++)` class as a member? Or both `extern(D)` and `extern(C++)` classes? ```d struct S { extern(C++) class C {} extern(D) class D {} C c; D d; } void main() { S s; } ``` I don't see why we can't have something similar with `extern(D)` and `extern(C)` / `extern(C++)` bit fields.
Mar 08
The original design of D allowed the compiler to lay out struct fields as it saw fit, and extern(C) had the C layout. This turned out to be rather pointless, and I dropped it.
Mar 08
To clarify a bit, C allows ints to be 18 bits (e.g. for a PDP-10). But D defines them as 32 bits. Technically, we're crazy to do that. But practically, we can rely on the people who implement C compilers to be sane, and CPU makers have long accepted the idea that ints are 32 bits. We are on very safe ground there. Even chars being signed by default, although popular in the 80s, has disappeared from modern compilers. We simply do not have to worry about the C bitfield layout being implementation defined. No sane person is going to invent another scheme for a C compiler if at all possible. There are two dominant schemes today, VC and gcc. I'm not aware of any other scheme for a modern C compiler (Digital Mars C emulates the VC layout). Even so, VC and gcc lay things out the same way for `int` bit fields. This is never going to change. Just stick with int bit fields and some common sense, and you're code will be fine. Much of D's design (and many other languages' design) is predicted on C behavior not being implementation defined, even though it technically is. There's no need to solve a non-existent problem. (Yes, I know, there is a version of D that has 16 bit ints. But that's a special build of the compiler, and it is technically not a D compiler :-/ and the people who use it understand that and it isn't a big problem for them.)
Mar 08
On 3/8/24 20:52, Walter Bright wrote:There are two dominant schemes today, VC and gcc. I'm not aware of any other scheme for a modern C compiler (Digital Mars C emulates the VC layout). Even so, VC and gcc lay things out the same way for `int` bit fields. This is never going to change.Well, if you believe this, why not specify the behavior? Similarly, if you believe a specced-out `extern(D)` vs a "companion compiler" `extern(C)` would not make a difference anyway, why is it a concern to you that they could be accidentally confused? And finally, why not just require an _explicit_ specification of the layout on the D side? E.g., something like: ```d int x { int a : 2; int b : 18; int c : 11; int : 1; // sign bit reserved } ``` (x.a, x.b, etc.) or perhaps even: ```d int { int a : 2; int b : 18; int c : 11; int : 1; // sign bit reserved } ``` Then make the compiler enforce that the fields add up to the size of the type. By default this is compatible with what VC and GCC do, and ImportC can just uniformly translate the imported C code to such an explicit layout spec. It will even be intuitive. Just make it portable across machines with the same endianness, like the built-in integer types already are. This optimizes the user experience at the same time: It will usually be obvious how big a struct is, there will not be a need to add together the sizes of bit fields in order to figure out how things are laid out in memory. In any case, the introspection story needs to be fixed. (Even just for the sake of introspecting structs originating in ImportC code.) This would be so much better and improve ImportC at the same time.
Mar 17
Thank you for your thoughts on this. I've thought a lot about it, too! In the very rare case of actually needing an exact layout, with 5 minutes of effort any layout can be duplicated with C bit fields. If that's too much effort, one can write explicit shift/mask code, and encapsulate it into a function. (Once the function is inlined, there is zero extra overhead for it.) Or one can use std.bitmanip to do that automatically. Yes, we can add extra syntax and semantics to make this controllable by the user, but it's so rarely needed it's over-engineering. The preceding is true if one wants a specific layout. But to algorithmically match what the associated C compiler does, that is more of a significant problem. It took me a while (with the help of Iain) to reverse-engineer it. This is a problem worth solving, and our implementation does solve it. For the user, it effortlessly resolves the issue of binary interoperability with C code. Good language design means building things into the core language that are the most common use cases, and provide facilities for the uncommon cases. This is why slices are builtin, rather than done with metaprogramming. An earlier version of D had complex numbers builtin. This did have some advantages over `struct Complex`. But Andrei told me it wasn't worth it. I eventually agreed with him, and took it out. But, for legacy compatibility, that had a very long deprecation period. Let's spend our limited complexity budget on problems that need to be solved, and explicit bitfield layout isn't one of them.
Apr 22
On 23/04/2024 4:16 AM, Walter Bright wrote:In the very rare case of actually needing an exact layout, with 5 minutes of effort any layout can be duplicated with C bit fields. If that's too much effort, one can write explicit shift/mask code, and encapsulate it into a function. (Once the function is inlined, there is zero extra overhead for it.) Or one can use std.bitmanip to do that automatically. Yes, we can add extra syntax and semantics to make this controllable by the user, but it's so rarely needed it's over-engineering.It depends so heavily on what you are doing, as I'm sure you know. If you work with file formats, IPC (sockets, pipes ext.), CPU flags for embedded devices, or performing instruction encoding for fun, you'll end up dealing with this stuff heavily enough that you want language support. If all you need it for is some flags in process then of course it appears to be over-engineering. I'm in the former group often enough that it is pretty uncomforting for me to see that this feature won't exist in those circumstances and this thread shows that I am not alone in this.
Apr 22
On 4/22/2024 10:28 AM, Richard (Rikki) Andrew Cattermole wrote:If you work with file formats, IPC (sockets, pipes ext.), CPU flags for embedded devices, or performing instruction encoding for fun, you'll end up dealing with this stuff heavily enough that you want language support.std.bitmanip.bitfields are the appropriate solution for that. BTW, the backend does a lot of instruction encoding :-) I'd prefer it was written using bitfields, it would make the code more readable. I'm sure I wouldn't have any difficulty using C bitfields for it. I'm sure you wouldn't, either.
Apr 22
On 23/04/2024 12:38 PM, Walter Bright wrote:On 4/22/2024 10:28 AM, Richard (Rikki) Andrew Cattermole wrote:This is something I do not want brought over to PhobosV3. Due to its reliance on CTFE, tooling cannot work with it. Its a pretty horrible thing to use I've found. I do not recommend using string mixins to introduce symbols that then get interacted with by the user. Same with mixin templates (I actually need to check to see if it supports eponymous at some point, naming an instance gets you to a lot of cases without the downside of a string mixin). Anything that uses string mixins to create symbols for a user to use, is likely to always cause usability issues for people. Plus with this DIP, the equivalent functionality is in the language, why would we want to duplicate where people need to look for it and have to remember the difference between them in terms of syntax?If you work with file formats, IPC (sockets, pipes ext.), CPU flags for embedded devices, or performing instruction encoding for fun, you'll end up dealing with this stuff heavily enough that you want language support.std.bitmanip.bitfields are the appropriate solution for that.BTW, the backend does a lot of instruction encoding :-) I'd prefer it was written using bitfields, it would make the code more readable. I'm sure I wouldn't have any difficulty using C bitfields for it. I'm sure you wouldn't, either.I used that as an example, although it would only really be of use when you're above a register in size. Being able to cast straight to a ``ubyte[]`` and then send off to something else (or the reverse) is a mighty useful thing to have since the cast is free. Both issues of defining LSB..MSB for multibyte along with introspection can be added later without a DIP. So this shouldn't hold it up, I'm giving push back because I know I'll be explaining all this at some point and I'd prefer to not have to.
Apr 22
On 4/22/24 18:16, Walter Bright wrote:In the very rare case of actually needing an exact layout, with 5 minutes of effort any layout can be duplicated with C bit fields.Well, if someone is not aware that a gun can shoot one's own foot, they might do it even if they very well know it is not good to hurt one's foot.If that's too much effort, one can write explicit shift/mask code, and encapsulate it into a function. (Once the function is inlined, there is zero extra overhead for it.) Or one can use std.bitmanip to do that automatically.I suppose you could match C bitfields with a string mixin too. Or the compiler could automatically generate a portable specification corresponding to the C bitfield layout.The preceding is true if one wants a specific layout. But to algorithmically match what the associated C compiler does, that is more of a significant problem. It took me a while (with the help of Iain) to reverse-engineer it. This is a problem worth solving, and our implementation does solve it. For the user, it effortlessly resolves the issue of binary interoperability with C code.Yes, I agree with this. However, there is a difference between having this feature and making it the _default representation_ of bit fields. It's just about how to best expose that feature for the best overall ergonomics, including introspection.
May 01
On Thursday, 7 March 2024 at 22:28:34 UTC, Dennis wrote:Why bow down to C and treat D as a second class citizen? The DIP should really substantiate this alleged importance of C-compatibility of non-ImportC D bitfields: Can you name at least one dub package that would benefit from bit fields with C-compatible layout?Qt uses bitfields in some headers, e.g. here: https://github.com/qt/qtbase/blob/4e158f6bfa7d0747d8da70b3b15a44b52e35bb8a/src/corelib/kernel/qtimer.h#L239 My bindings emulate the bitfields with a single field and accessor functions: https://github.com/tim-dlang/dqt/blob/2afa4adcec0a6905b11256f403c6136277a6c96a/core/qt/core/timer.d#L217 Bitfields in D could be used instead, but they would need to be compatible with C++ for extern(C++) types. Sometimes bitfield members are disabled with the preprocessor: https://github.com/qt/qtbase/blob/4e158f6bfa7d0747d8da70b3b15a44b52e35bb8a/src/corelib/time/qdatetime.h#L259 This could be translated to version statements in D. Most of the times bitfields for Qt are not a big problem, because they are only private members of the types.
Mar 09
On 3/7/24 20:42, Walter Bright wrote:I.e. the layout is implementation-defined. That's correct. But in reality, it is fixed. There is no way that gcc on Linux is *ever* going to change the bit field layout, because it would break legions of existing code. It is cast in concrete, and is as immutable as it gets. The same for Microsoft C. Furthermore, if you use `int` as the base type for bit fields, the layouts are all the same in all the C compilers I've encountered. For practical purposes, it is defined. If you are still concerned about the layout, like you want a portable file format, don't use bit fields. Use std.bitmanip.Well, that's kind of a foot gun. Portable by default is better.
Mar 16
On 3/16/2024 2:32 PM, Timon Gehr wrote:On 3/7/24 20:42, Walter Bright wrote:One thing about language design - there is no rule that overrides every other rule. This applies to everything. Every design is a balance of competing interests.If you are still concerned about the layout, like you want a portable file format, don't use bit fields. Use std.bitmanip.Well, that's kind of a foot gun. Portable by default is better.
Apr 22
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdThe usual way to do bitfields is (culled from the DMD compiler source): ```D struct S { enum Flags : uint { isnothrow = 0x0001, // nothrow isnogc = 0x0002, // is nogc isproperty = 0x0004, // can be called without parentheses isref = 0x0008, // returns a reference isreturn = 0x0010, // 'this' is returned by ref isscope = 0x0020, // 'this' is scope isreturninferred= 0x0040, // 'this' is return from inference isscopeinferred = 0x0080, // 'this' is scope from inference islive = 0x0100, // is live incomplete = 0x0200, // return type or default arguments removed inoutParam = 0x0400, // inout on the parameters inoutQual = 0x0800, // inout on the qualifier } Flags flags; } ```Granted that this way is pretty common, but it's far from optimal. `std.bitmanip.bitfields` does this a lot better, even more so when you have multiple-bit fields. Yes, as the DIP says it lacks C compatibility guarantees, but so does this solution. This example is thus a bit of a straw man. Please just supply an example with a standard library bitfield. In the general use case, portability trumps C compatibility. Therefore, we should and will use `std.bitmanip` bitfields when we don't explicitly have to be C compatible. What the DIP is suggesting should be considered a special bitfield type, not the "normal" case. I do not think any syntactic additions to the language itself are a good idea. We don't have any special syntax for normal bitfields either so it would be strange for C-compatible bitfields to have that. It'd be better that C bitfields would be declared with a mixin somewhere in DRuntime, its usage syntax preferably as close as possible to `std.bitmanip.bitfields`. The compiler can still recognise the call of the mixin as special, but from user POV it should be no different from any other mixin.
Mar 07
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdOne cannot take a reference to or the address of a bitfieldTaking a ref is not detected yet: https://issues.dlang.org/show_bug.cgi?id=24263 I was wondering how to read/write the bits of a bitfield set as their underlying type. So I tried putting a bitfield set inside an anonymous union: ```d import std.stdio; struct S { union { uint u; uint a:1, b:1, c:1; } } static assert(S.sizeof == 4); void main() { S s; s.tupleof.writeln(); // 0000 s.u++; s.tupleof.writeln(); // 1111 } ``` I was a bit surprised that setting `u` to 1 sets all 3 bitfields to 1 as well - I was assuming the union had 2 uint members where the 2nd one didn't have a name. Then I realized I needed to put the bitfields inside an anonymous struct, because otherwise each one was a union member. Anyway doing that does work.
Mar 09
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdTypo:Abstract This proposal is for supporting bitfields in **C** the same way they are supported in **C**.
Mar 11
On 3/11/2024 2:46 AM, Martin Tschierschke wrote:On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:hahahttps://github.com/WalterBright/documents/blob/master/bitfields.mdTypo:Abstract This proposal is for supporting bitfields in **C** the same way they are supported in **C**.
Mar 11
On Monday, 11 March 2024 at 17:57:58 UTC, Walter Bright wrote:On 3/11/2024 2:46 AM, Martin Tschierschke wrote:Sorry, but this is an ERROR on __your__ page, not a joke.On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:hahahttps://github.com/WalterBright/documents/blob/master/bitfields.mdTypo:Abstract This proposal is for supporting bitfields in **C** the same way they are supported in **C**.
Mar 12
On Tuesday, March 12, 2024 1:31:49 AM MDT Martin Tschierschke via dip.development wrote:On Monday, 11 March 2024 at 17:57:58 UTC, Walter Bright wrote:I expect that he was just laughing at his mistake. - Jonathan M DavisOn 3/11/2024 2:46 AM, Martin Tschierschke wrote:Sorry, but this is an ERROR on __your__ page, not a joke.On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:hahahttps://github.com/WalterBright/documents/blob/master/bitfields.mdTypo:Abstract This proposal is for supporting bitfields in **C** the same way they are supported in **C**.
Mar 12
On 3/6/2024 8:26 PM, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdPermanent link: https://github.com/WalterBright/documents/blob/9b91ca5242e3e49ed807ee696675c078986aa51e/bitfields.md
Mar 14
On 3/7/24 05:26, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdStill not a fan of this potentially not being portable between different compilers implementing the C calling convention on the same platform. But let's put that aside for a moment. Let's say D bitfields are stored in memory in a non-portable way. How would a serialization library detect via introspection that these are bitfields and serialize them accordingly (in a portable way)? In general, how to introspect on bitfields?
Mar 16
On 3/16/2024 2:40 PM, Timon Gehr wrote:In general, how to introspect on bitfields?Currently, allMembers will get you the name of the bitfield, and you can use .offsetof and .alignof on it. The number of bits can be derived from the .max property. You'll know it's a bitfield if the .max property is less than the type.max. From those, the bit offsets can be determined, but one could reasonably ask for an easier way to get that.
Apr 21
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdMost of the complaints by the seasoned pros here are of the form: C bitfields are an implementation-defined mess, and we shouldn't copy them. I don't think this means we can't copy the *good parts*. If I understand correctly, it's because C can do odd things with corner cases, not because it's doing insane things like storing bitfields in every other bit, etc. So if "almost all C compilers" do the same thing, why not just define that thing, and say D does it this way? Why say we depend 100% on C compiler implementation whims? If the corner cases are problematic, ban the corner cases? Pick the "right way", define it, and say "This works for most cases in all the C compilers we use, and if it doesn't, you have to figure it out yourself". I don't understand the problem with that. ImportC bitfields *don't have to be the same*. As you once told me, bitfields are not generally used for public API, because of the complications. So why do we need C compatibility in the first place? C does some really dumb things (hindsight, etc.), and D didn't copy all those dumb things. We should continue to not do dumb things. Bitfields should have `__traits` to go with them (e.g. `__traits(bitSize, x)` should return `4` for a 4-bit value). There is zero reason to hide what the compiler knows. I don't know if there are any more things needed, as you can use `typeof` to get the underlying type, and getting an address to the full bitfield can be done with a union. There are other things that aren't specified in the proposal, such as the properties that work on other basic types: `sizeof`, `offsetof`, `alignof`. -Steve
Mar 22
On Saturday, 23 March 2024 at 02:32:20 UTC, Steven Schveighoffer wrote:So if "almost all C compilers" do the same thing, why not just define that thing, and say D does it this way? Why say we depend 100% on C compiler implementation whims? If the corner cases are problematic, ban the corner cases?Walter already replied to my identical suggestion: https://forum.dlang.org/post/usnh0c$2180$1 digitalmars.com
Mar 23
On 3/22/2024 7:32 PM, Steven Schveighoffer wrote:So if "almost all C compilers" do the same thing, why not just define that thing, and say D does it this way? Why say we depend 100% on C compiler implementation whims? If the corner cases are problematic, ban the corner cases?I don't see an actual use case for our own layout. They may indeed be whims on the part of the C compiler, but we deliberately pick which C compiler is the "associated C compiler", and why pick a nutburger compiler?Pick the "right way", define it,There isn't a right way.and say "This works for most cases in all the C compilers we use, and if it doesn't, you have to figure it out yourself". I don't understand the problem with that. ImportC bitfields *don't have to be the same*. As you once told me, bitfields are not generally used for public API, because of the complications. So why do we need C compatibility in the first place?One of the uses of bitfields in D is interoperation with C code. It's not unreasonable to have a program that is half in D and half in C. Having different bitfield layouts would just be an aggravation. It would be surprising.C does some really dumb things (hindsight, etc.), and D didn't copy all those dumb things. We should continue to not do dumb things.We copy the dumb way C passes arguments to functions, too. Why? Because it's convenient, easy to explain, nobody questions it, etc. D in the past had a slightly better way of calling C code, and we abandoned that because the nuisance of being different simply was not worth it. Changing it to match C made the problems and the endless effort explaining the difference just go away. The value of "it is the same as C" is quite large.There are other things that aren't specified in the proposal, such as the properties that work on other basic types: `sizeof`, `offsetof`, `alignof`.ok
Apr 22
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdMy list of wishes: 1. bitfields imported from C should use `extern(C)`, this way they can use whatever layout C compiler uses. 2. D bitfields should be `extern(D)`, and work like `bitfields` mixin. 3. Allow specifying default value. Here is how C3 language does bitfields: https://c3lang.github.io/c3docs/types/#bitstructs It has guaranteed layout, allows specifying endinanness, optional overlaping of fields. It allows static arrays to be used as a backing type. Its bitfields are done trough a special kind of struct - `bitstruct`.
Mar 25
On 3/25/2024 9:39 AM, MrSmith wrote:My list of wishes: 1. bitfields imported from C should use `extern(C)`, this way they can use whatever layout C compiler uses. 2. D bitfields should be `extern(D)`, and work like `bitfields` mixin. 3. Allow specifying default value. Here is how C3 language does bitfields: https://c3lang.github.io/c3docs/types/#bitstructs It has guaranteed layout, allows specifying endinanness, optional overlaping of fields. It allows static arrays to be used as a backing type. Its bitfields are done trough a special kind of struct - `bitstruct`.Thank you for the link to C3. I read it. Some problems are simply not worth solving, and C3 goes too far. You can get a consistent layout by simply declaring your bitfields as 'int'. I know this is not specified anywhere, but it works.
Apr 22
On Thursday, 7 March 2024 at 04:26:56 UTC, Walter Bright wrote:https://github.com/WalterBright/documents/blob/master/bitfields.mdI don't know if already mentioned. It's "workaround aura" not "workaround aurora" in ImportC paragraph of the rationale. Aurora means dawn.
Apr 02
Threads for review of drafts two and three have been opened subsequent to this one. Please leave further feedback on the thread for the third draft, linked below. This thread is now closed. https://forum.dlang.org/post/v193hc$b9c$1 digitalmars.com
May 21