www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D as an extension language for C

reply Walter Bright <newshound2 digitalmars.com> writes:
https://twitter.com/WalterBright/status/1656574787874095104

I love making discoveries of emergent behavior like this! It's certainly saved 
me a lot of work.
May 11 2023
parent reply Coco <Coco gmail.com> writes:
On Thursday, 11 May 2023 at 18:11:32 UTC, Walter Bright wrote:

 https://twitter.com/WalterBright/status/1656574787874095104

 I love making discoveries of emergent behavior like this! It's 
 certainly saved me a lot of work.


Interesting, yes... or at least.. perhaps.

But personally, I think the D language's obsession with the C 
language.. has already gone too far.

Rust and Swift are showing where languages should be heading.
May 11 2023
parent reply user <user blah.com> writes:
On Thursday, 11 May 2023 at 22:08:30 UTC, Coco wrote:

 On Thursday, 11 May 2023 at 18:11:32 UTC, Walter Bright wrote:

 https://twitter.com/WalterBright/status/1656574787874095104

 I love making discoveries of emergent behavior like this! It's 
 certainly saved me a lot of work.


 Interesting, yes... or at least.. perhaps.

 But personally, I think the D language's obsession with the C 
 language.. has already gone too far.

 Rust and Swift are showing where languages should be heading.



Couldn't help but observe that

D started as better Java by aot compiling similar code

Then D wanted to become better C++ with ranges, structs instead 
of classes, nogc, etc

Now D wants to be betterC, C extension, etc

Future: D as better assembly

/s
:-)
May 12 2023
next sibling parent reply zjh <fqbqrr 163.com> writes:
On Friday, 12 May 2023 at 22:07:43 UTC, user wrote:

 On Thursday, 11 May 2023 at 22:08:30 UTC, Coco wrote:





 Couldn't help but observe that

 D started as better Java by aot compiling similar code

 Then D wanted to become better C++ with ranges, structs instead 
 of classes, nogc, etc

 Now D wants to be betterC, C extension, etc

 Future: D as better assembly

 /s
 :-)



If D can be a good glue language, it is also good. D can 
interface with `C++, rust, C, and asm`  etc.
May 12 2023
next sibling parent zjh <fqbqrr 163.com> writes:
On Saturday, 13 May 2023 at 00:35:04 UTC, zjh wrote:

 If D can be a good glue language, it is also good. D can 
 interface with `C++, rust, C, and asm`  etc.



I have a `weak` ecosystem, but I can borrow that I can 
`interface` with, so there's nothing wrong with it.
Looking at the new language, which one does not support 
interfacing with `C`?
May 12 2023
prev sibling parent Sergey <kornburn yandex.ru> writes:
On Saturday, 13 May 2023 at 00:35:04 UTC, zjh wrote:

 On Friday, 12 May 2023 at 22:07:43 UTC, user wrote:

 On Thursday, 11 May 2023 at 22:08:30 UTC, Coco wrote:





 Couldn't help but observe that

 D started as better Java by aot compiling similar code

 Then D wanted to become better C++ with ranges, structs 
 instead of classes, nogc, etc

 Now D wants to be betterC, C extension, etc

 Future: D as better assembly

 /s
 :-)



 If D can be a good glue language, it is also good. D can 
 interface with `C++, rust, C, and asm`  etc.


C is already in that place. FFI C has almost every language..
May 13 2023
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2023 3:07 PM, user wrote:

 Future: D as better assembly


It already is. D has a better assembler than Gnu, and VC doesn't even have an 
assembler.

Perhaps I haven't clarified the purpose of ImportC.

D was designed from the beginning to leverage off of existing C code. D's 
foreign function interface with C is excellent.

But D itself cannot read C .h files nor .c files (a capability that C++ has and 
has used to great advantage). .h files must be laboriously converted to D.
We've 
done a lot of the usual system .h files conversions, such as core.stdc.stdio, 
they are in druntime.

The Deimos project is to collect crowd-sourced conversions of C .h files.

The fundamental problem with hand conversions is they are tedious, boring work 
that nobody wants to do. They also need redoing whenever the vendor of the .h 
file changes them. As the D community grows, this simply does not scale.

Then there were three attempts at programmatic conversion of .h to .d files,
one 
written by me, one by Jacob, and one by Atila. Those brought us to 3rd base.

To do a home run, the compiler needs to understand C code directly - this is 
ImportC.

Using D as an extension language for ImportC enabled me to easily get working a 
number of C features (such as variadic arguments) that would otherwise have 
required a fair chunk of explicit code to implement. Instead, I was able have
it 
use D's existing implementation code (which is non-trivial) in core.stdc.stdarg.

I.e. ImportC is an enabler of D use.
May 13 2023
next sibling parent Dave P. <dave287091 gmail.com> writes:
On Saturday, 13 May 2023 at 18:09:33 UTC, Walter Bright wrote:

 On 5/12/2023 3:07 PM, user wrote:

 Future: D as better assembly


 […]
 Using D as an extension language for ImportC enabled me to 
 easily get working a number of C features (such as variadic 
 arguments) that would otherwise have required a fair chunk of 
 explicit code to implement. Instead, I was able have it use D's 
 existing implementation code (which is non-trivial) in 
 core.stdc.stdarg.

 I.e. ImportC is an enabler of D use.


Unfortunately for that example, dmd’s implementation of c varargs 
is buggy.
https://issues.dlang.org/show_bug.cgi?id=23409
May 13 2023
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Saturday, 13 May 2023 at 18:09:33 UTC, Walter Bright wrote:

 D was designed from the beginning to leverage off of existing C 
 code. D's foreign function interface with C is excellent.


I thought it was C++ you wanted to leverage.


 But D itself cannot read C .h files nor .c files (a capability 
 that C++ has and has used to great advantage). .h files must be 
 laboriously converted to D. We've done a lot of the usual 
 system .h files conversions, such as core.stdc.stdio, they are 
 in druntime.

 The Deimos project is to collect crowd-sourced conversions of C 
 .h files.

 The fundamental problem with hand conversions is they are 
 tedious, boring work that nobody wants to do. They also need 
 redoing whenever the vendor of the .h file changes them. As the 
 D community grows, this simply does not scale.

 Then there were three attempts at programmatic conversion of .h 
 to .d files, one written by me, one by Jacob, and one by Atila. 
 Those brought us to 3rd base.

 To do a home run, the compiler needs to understand C code 
 directly - this is ImportC.


You keep mention on this forum that you keep on hitting corner 
cases with importC which s expected. C is old and there many 
quirks, non-standard and compiler specific code out there. I 
would still gone the converter route as it would instead 
translate the .h file to native D which you then can use natively 
within D. For example you haven't enabled C macros to be used 
within D, with a converter you could have converted the macro to 
D code (mixin?). I think that would have scaled better than 
trying to retrofit a C compiler into D compiler. Also why should 
C be a first class citizen and not C++, Rust etc. Since there are 
plenty of languages out there a converter project for each 
language is more convenient.

I'm a bit afraid that importC will be a feature that needs 
constant patching and consume time.
May 14 2023
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2023 1:21 AM, IGotD- wrote:

 I thought it was C++ you wanted to leverage.


I knew back then the impracticality of importing C++ code.



 You keep mention on this forum that you keep on hitting corner cases with 
 importC which s expected. C is old and there many quirks, non-standard and 
 compiler specific code out there. I would still gone the converter route as it 
 would instead translate the .h file to native D which you then can use
natively 
 within D.


We've had 3 converters so far.


 For example you haven't enabled C macros to be used within D, with a 
 converter you could have converted the macro to D code (mixin?). I think that 
 would have scaled better than trying to retrofit a C compiler into D compiler. 


There are a number of C features which do not translate into D. But with
ImportC 
I can make them work seamlessly.


 Also why should C be a first class citizen and not C++, Rust etc.


C is the lingua franka, not C++ or Rust.


 Since there
 are plenty of languages out there a converter project for each language is more
 convenient.


I can only do so much.



 I'm a bit afraid that importC will be a feature that needs constant patching
and 
 consume time.


C changes a lot slower than other languages.
May 14 2023
parent reply IGotD- <nise nise.com> writes:
On Sunday, 14 May 2023 at 23:56:55 UTC, Walter Bright wrote:

 There are a number of C features which do not translate into D. 
 But with ImportC I can make them work seamlessly.


What are the C features that cannot be expressed in D?
May 15 2023
parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 15 May 2023 at 20:19:05 UTC, IGotD- wrote:

 On Sunday, 14 May 2023 at 23:56:55 UTC, Walter Bright wrote:

 There are a number of C features which do not translate into 
 D. But with ImportC I can make them work seamlessly.


 What are the C features that cannot be expressed in D?


VLA's and flexible arrays.
May 16 2023
next sibling parent reply IGotD- <nise nise.com> writes:
On Tuesday, 16 May 2023 at 07:48:42 UTC, Patrick Schluter wrote:

 On Monday, 15 May 2023 at 20:19:05 UTC, IGotD- wrote:

 What are the C features that cannot be expressed in D?


 VLA's and flexible arrays.


A C flexible array can be translated to a slice. I have done this 
myself and was amused how good fit it was. In 99% of the cases 
there is some upper limit for the flexible array and a slice is 
an extra guard against out of bounds accesses that C doesn't have.

VLA should be a part of the FFI and not importC.

The only thing I can think of that isn't supported are bitfields. 
The question is if this could have been solved with meta 
programming.
May 16 2023
next sibling parent Andrew <andrewlalisofficial gmail.com> writes:
On Tuesday, 16 May 2023 at 11:04:53 UTC, IGotD- wrote:

 The only thing I can think of that isn't supported are 
 bitfields. The question is if this could have been solved with 
 meta programming.


Bit fields are already something that's provided in Phobos: 
https://dlang.org/phobos/std_bitmanip.html#bitfields

I can't attest to how good they are, but I think it's the right 
approach; rather choose to implement something in the std lib 
than the language itself to avoid unnecessary complexity.
May 16 2023
prev sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 16 May 2023 at 11:04:53 UTC, IGotD- wrote:

 The only thing I can think of that isn't supported are 
 bitfields. The question is if this could have been solved with 
 meta programming.


ImportC can do C bitfields, but since they're layout is undefined 
it is useless anyway.

(Walter also copied importC's awful bitfields into D, but I hope 
that mistake is reverted.)
May 16 2023
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2023 6:36 AM, Adam D Ruppe wrote:

 ImportC can do C bitfields, but since they're layout is undefined it is
useless 
 anyway.


Pedantically, the C layout is not undefined, it is implementation defined. D's 
bitfields, however, are defined to match the layout of the associated C
compiler.

But as a practical matter, if you stick with `int` and `unsigned` bit field 
types, the layout is consistent across compilers with 32 bit ints. Any other 
differences would only be a problem if data is moved between systems. The C
code 
will still work.

It's analogous to the size of `int` being implementation defined, along with
the 
size of `char`, `long`, `long long`, the signedness of char being
implementation 
defined, whether it's ones or twos complement, and so on. But C remained useful.

mportC also implements a number of half-specified associated C compiler 
extensions, because (sadly) nearly all of the C header files unnecessarily rely 
on these extensions.

Ironically, by making D bitfields match the associated C compiler's, it
arguably 
makes D bitfields *more* portable.



 (Walter also copied importC's awful bitfields into D, but I hope that mistake
is 
 reverted.)


There's always https://dlang.org/phobos/std_bitmanip.html#bitfields which do 
have a portable defined format.

Note that nobody has implemented in Phobos a bitfield setup that matches the 
associated C compiler's bit fields, making such C constructs difficult to
access 
from D. They're easy now!
May 16 2023
next sibling parent Daniel N <no public.email> writes:
On Wednesday, 17 May 2023 at 03:52:33 UTC, Walter Bright wrote:

 On 5/16/2023 6:36 AM, Adam D Ruppe wrote:

 ImportC can do C bitfields, but since they're layout is 
 undefined it is useless anyway.



 Ironically, by making D bitfields match the associated C 
 compiler's, it arguably makes D bitfields *more* portable.


I agree 100%. Usually unhappy users are more vokal than happy 
users, so I just wanted to say thanks Walter, I for one love this 
feature!
May 17 2023
prev sibling next sibling parent reply max haughton <maxhaton gmail.com> writes:
On Wednesday, 17 May 2023 at 03:52:33 UTC, Walter Bright wrote:

 On 5/16/2023 6:36 AM, Adam D Ruppe wrote:

 ImportC can do C bitfields, but since they're layout is 
 undefined it is useless anyway.


 Pedantically, the C layout is not undefined, it is 
 implementation defined. D's bitfields, however, are defined to 
 match the layout of the associated C compiler.


This is what makes them a major footgun. Other than being 
underpowered for the complexity they add (I want the compiler to 
help me do discontinuous bitpacking not just easy stuff), you 
can't rely on the layout for basically anything other than 
space-efficiency, so you can't guarantee something will actually 
round-trip (say) over the network or even a filesystem/buffer on 
the same computer.

This, for example, is why one of the ELF specifications 
explicitly recommends not using bitfields to implement it.
May 17 2023
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/17/2023 2:53 AM, max haughton wrote:

 This, for example, is why one of the ELF specifications explicitly recommends 
 not using bitfields to implement it.


That's also why people implementing ELF use typedefs instead of ints, etc., and 
have special code to deal with endianness, which is also implementation defined.

Despite all this implementation defined behavior in C, there are some things 
that while pedantically implementation defined, are in practice consistently 
defined across diverse compilers. The int bitfields, for example.
May 17 2023
prev sibling parent reply Johan <j j.nl> writes:
On Wednesday, 17 May 2023 at 03:52:33 UTC, Walter Bright wrote:

 On 5/16/2023 6:36 AM, Adam D Ruppe wrote:

 ImportC can do C bitfields, but since they're layout is 
 undefined it is useless anyway.


 Pedantically, the C layout is not undefined, it is 
 implementation defined. D's bitfields, however, are defined to 
 match the layout of the associated C compiler.


What defines the "associated C compiler"?
It's the first time I hear it for D.

thanks,
   Johan
May 17 2023
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/17/2023 4:25 AM, Johan wrote:

 What defines the "associated C compiler"?


The D implementation.


 It's the first time I hear it for D.


It's referred to in the D spec.

https://dlang.org/glossary.html#acc
May 17 2023
prev sibling parent reply Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Tuesday, 16 May 2023 at 13:36:46 UTC, Adam D Ruppe wrote:

 On Tuesday, 16 May 2023 at 11:04:53 UTC, IGotD- wrote:

 The only thing I can think of that isn't supported are 
 bitfields. The question is if this could have been solved with 
 meta programming.


 ImportC can do C bitfields, but since they're layout is 
 undefined it is useless anyway.

 (Walter also copied importC's awful bitfields into D, but I 
 hope that mistake is reverted.)


Yes, the syntax is awful. It should have been `__vector(bool[n])`:
```d
struct minifloat
{
     __vector(bool[3]) significand;
     __vector(bool[4]) exponent;
     __vector(bool[1]) sign;
}
```

C++ leads the way with `std::vector<bool>` and its packed bits. 😁

No, in all seriousness, `__vector(bool[n])` wouldn’t be nearly as 
bad a syntax as C’s bit-fields. And giving `__vector(bool[n])` 
special semantics is nowhere close to daring.
May 24 2023
parent Adam D Ruppe <destructionator gmail.com> writes:
On Wednesday, 24 May 2023 at 14:28:52 UTC, Quirin Schroll wrote:

 Yes, the syntax is awful. It should have been 
 `__vector(bool[n])`:


How would that interact with things like `sign.sizeof`?

I wrote about some of these things in more detail here btw:

http://dpldocs.info/this-week-in-d/Blog.Posted_2022_09_12.html#bitfields
May 24 2023
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/16/2023 12:48 AM, Patrick Schluter wrote:

 On Monday, 15 May 2023 at 20:19:05 UTC, IGotD- wrote:

 On Sunday, 14 May 2023 at 23:56:55 UTC, Walter Bright wrote:

 There are a number of C features which do not translate into D. But with 
 ImportC I can make them work seamlessly.


 What are the C features that cannot be expressed in D?


 
 VLA's and flexible arrays.


Some more things:

1. struct tags and non-tag names co-existing in the same scope

2. enum member names get "promoted" to the surrounding scope, losing their
enum-ness

3. bit fields (since fixed by adding bitfields to D!)

4. C's everything-in-one-global-namespace issue

5. default initialization isn't always zero in D

6. different rules for `const` (ImportC admittedly fakes it)


If you look at the frontend source code, the semantic routines have lots of "if 
it's C do this instead".

Of course, "cannot" is an absolute word. There are always ways to make
something 
work, but the result may be just too ugly or otherwise unpalatable. With 
ImportC, we just let C be C instead of trying to bash it into D code.
May 16 2023
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 13 May 2023 at 18:09:33 UTC, Walter Bright wrote:

 [snip]
 Perhaps I haven't clarified the purpose of ImportC.

 D was designed from the beginning to leverage off of existing C 
 code. D's foreign function interface with C is excellent.

 But D itself cannot read C .h files nor .c files (a capability 
 that C++ has and has used to great advantage). .h files must be 
 laboriously converted to D. We've done a lot of the usual 
 system .h files conversions, such as core.stdc.stdio, they are 
 in druntime.

 [snip]

 To do a home run, the compiler needs to understand C code 
 directly - this is ImportC.
 [snip]


Is ImportC the final destination? Or is ImportC a step along the 
way to a final destination?
May 14 2023
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/14/2023 4:39 PM, jmh530 wrote:

 Is ImportC the final destination? Or is ImportC a step along the way to a
final 
 destination?


Not sure what you mean. It is an end in itself.
May 14 2023
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Sunday, 14 May 2023 at 23:57:32 UTC, Walter Bright wrote:

 On 5/14/2023 4:39 PM, jmh530 wrote:

 Is ImportC the final destination? Or is ImportC a step along 
 the way to a final destination?


 Not sure what you mean. It is an end in itself.


You said before: "But D itself cannot read C .h files nor .c 
files (a capability that C++ has and has used to great advantage)"

importC lets you import .c file in D, but not a .h file. To 
import a .h file in D, I have to create a .c file and include 
only the .h file header and then import the .c file. Would you be 
able to streamline that further so that the .h file can be 
imported?

Maybe behind the scenes, it would look for a .d file, then a .c 
file, then try to create a blank .c file and include a .h? I 
don't know what makes the most sense, but think about it this 
way: for me to say some random C library works for D, I need to 
create a separate project for it with a .c file that includes the 
.h. If you can import the .h file directly, then it would just 
work (TM).
May 15 2023
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/15/2023 5:20 AM, jmh530 wrote:

 importC lets you import .c file in D, but not a .h file. To import a .h file
in 
 D, I have to create a .c file and include only the .h file header and then 
 import the .c file. Would you be able to streamline that further so that the
.h 
 file can be imported?


The .h file limitation is due to a disagreement between myself and Iain about 
the best way forward for it.
May 15 2023
parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 15 May 2023 at 19:13:49 UTC, Walter Bright wrote:

 On 5/15/2023 5:20 AM, jmh530 wrote:

 importC lets you import .c file in D, but not a .h file. To 
 import a .h file in D, I have to create a .c file and include 
 only the .h file header and then import the .c file. Would you 
 be able to streamline that further so that the .h file can be 
 imported?


 The .h file limitation is due to a disagreement between myself 
 and Iain about the best way forward for it.


I hope some agreement is reached.

Looking over the (multiple) github discussions, I was partial to 
the solution of `import "foo.h"` combined with a custom search 
path for `.h` files [1]. The one to use `import("foo.h")` looks a 
bit too much like an import expression.

You made a comment about the import syntax getting kludged [2], 
but it would be a little less kludge-y if you could import any 
file like `import "foo.d"` or `import "foo.c"`, but then only 
allow .d/.c/.i files to be imported like `import foo`. Further, I 
recall you previously commenting on how sometimes you have to 
make the compiler more complicated in order to make things work 
how users expect them to. This might be a situation like that. 
Though I can't say what the best solution is, I think most would 
agree that it would be good to have something better than 
creating a .c file to import a .h file.

Not a lot of code has been written yet that uses importC so there 
is still time to get it right.

[1] https://github.com/dlang/dmd/pull/14700
[2] 
https://github.com/dlang/dmd/pull/14864#issuecomment-1432256522
May 16 2023