www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - built in import c header files

reply 12345swordy <alexanderheistermann gmail.com> writes:
I remember reading a reddit comment saying that if d were to 
import c header files directly without any 3rd party libraries it 
would be a game changer. The question is: how difficult to 
implement a C AST parser for the dmd front end?


-Alex
Oct 06 2020
next sibling parent reply Meta <jared771 gmail.com> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


You're in luck: https://github.com/atilaneves/dpp
Oct 06 2020
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


 You're in luck: https://github.com/atilaneves/dpp


What part of "built in and not part of a 3rd party library" do 
you not understand?
Oct 06 2020
parent reply norm <normical gmail.com> writes:
On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:

 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front end?


 -Alex


 You're in luck: https://github.com/atilaneves/dpp


 What part of "built in and not part of a 3rd party library" do 
 you not understand?


You could easily work it out for yourself by looking at any C 
parser and the DMD front. But why bother with C, it is trivial to 
generate C bindings from headers. C++ is where it becomes 
complicated and where it would be useful.
Oct 06 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 7 October 2020 at 01:29:05 UTC, norm wrote:

 On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:

 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front end?


 -Alex


 You're in luck: https://github.com/atilaneves/dpp


 What part of "built in and not part of a 3rd party library" do 
 you not understand?


 You could easily work it out for yourself by looking at any C 
 parser and the DMD front.



 But why bother with C, it is trivial to generate C bindings 
 from headers.


It really isn't. Can it be easy? Yes. Is it easy in the general 
case? No.

I've tried calling a large C codebase from D before. It was a 
nightmare. It's very hard to do manual wrapping when you have 
thousands of headers in hundreds of directories and everything 
depends on everything.
Oct 07 2020
parent reply norm <normical gmail.com> writes:
On Wednesday, 7 October 2020 at 10:04:57 UTC, Atila Neves wrote:

 On Wednesday, 7 October 2020 at 01:29:05 UTC, norm wrote:

 On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:

 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy 
 wrote:

 I remember reading a reddit comment saying that if d were 
 to import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front end?


 -Alex


 You're in luck: https://github.com/atilaneves/dpp


 What part of "built in and not part of a 3rd party library" 
 do you not understand?


 You could easily work it out for yourself by looking at any C 
 parser and the DMD front.



 But why bother with C, it is trivial to generate C bindings 
 from headers.




Really? I have not had any issues before automatically creating C 
bindings for D using codegen. Sure, occasionally a macro will 
bite but it isn't that often nor difficult to fix.  They were not 
the kernel but they were not small code bases without macros 
either.

TBH though now I just use dpp so I don't see the value of bolting 
on a "C" parser to the D front end. C++ I have struggled with 
that and it requires an annoying amount of manual editing. DPP 
works well but even it falls over for some C++ code bases I've 
tried.
Oct 07 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 7 October 2020 at 11:14:02 UTC, norm wrote:

 On Wednesday, 7 October 2020 at 10:04:57 UTC, Atila Neves wrote:

 On Wednesday, 7 October 2020 at 01:29:05 UTC, norm wrote:

 On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:

 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy 
 wrote:

 I remember reading a reddit comment saying that if d were 
 to import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front 
 end?


 -Alex


 You're in luck: https://github.com/atilaneves/dpp


 What part of "built in and not part of a 3rd party library" 
 do you not understand?


 You could easily work it out for yourself by looking at any C 
 parser and the DMD front.



 But why bother with C, it is trivial to generate C bindings 
 from headers.




 Really? I have not had any issues before automatically creating 
 C bindings for D using codegen. Sure, occasionally a macro will 
 bite but it isn't that often nor difficult to fix.  They were 
 not the kernel but they were not small code bases without 
 macros either.


Imagine this and extrapolate to hundreds of headers:

---
// hdr0.h
struct struct0 {
     struct1* s1;
};
---

---
// hdr1.h
#define SILLY_MACRO struct2;
struct struct1 {
     SILLY_MACRO* s2;
}
----

---
// hdr2.h
struct struct2 {
     struct3* s3;
}
---



 TBH though now I just use dpp so I don't see the value of 
 bolting on a "C" parser to the D front end. C++ I have 
 struggled with that and it requires an annoying amount of 
 manual editing. DPP works well but even it falls over for some 
 C++ code bases I've tried.


dpp doesn't officially support C++ yet, because... it doesn't 
work yet. There are several reasons for this.

The first, and most annoying, is that libclang isn't all it's 
cracked up to be. There's a *lot* of functionality missing for 
C++ that the compiler frontend itself knows about but doesn't 
expose via the C API.

The second, and maybe unfixable reason, is how to translate 
C++-only concepts to D. Consider std::is_reference_v. That can't 
be written in D, not even by hand. What if that gets used in 
SFINAE? What about expression templates given how operator 
overloading in D is a lot different? I still don't know how to 
get around that.
Oct 08 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
There's another huge problem with C files.

It's the predefined macros. Every compiler generates a different set of them, 
and in each compiler the set varies by compiler version, target, switch 
settings, etc.

Most are undocumented.

I think gcc has somewhere near a thousand of them.

Naturally, every non-trivial C program winds up totally dependent on them.
Apr 17 2021
prev sibling next sibling parent reply Max Haughton <maxhaton gmail.com> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


Not impossible (it's worth saying that you can literally do this 
as a library via mixins), but having a c parser in the compiler 
seems like too much work when it's already fairly trivial to do. 
If D had a team of full time staff on payroll, then it could be 
made to work but it adds so much surface area to the compiler.
Oct 06 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/6/2020 7:56 PM, Max Haughton wrote:

 Not impossible (it's worth saying that you can literally do this as a library 
 via mixins), but having a c parser in the compiler seems like too much work
when 
 it's already fairly trivial to do. If D had a team of full time staff on 
 payroll, then it could be made to work but it adds so much surface area to the 
 compiler.


You'll also have problems with all the nutburger C extensions implemented by 
various compilers.
Apr 17 2021
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Saturday, 17 April 2021 at 21:32:24 UTC, Walter Bright wrote:

 On 10/6/2020 7:56 PM, Max Haughton wrote:

 Not impossible (it's worth saying that you can literally do 
 this as a library via mixins), but having a c parser in the 
 compiler seems like too much work when it's already fairly 
 trivial to do. If D had a team of full time staff on payroll, 
 then it could be made to work but it adds so much surface area 
 to the compiler.


 You'll also have problems with all the nutburger C extensions 
 implemented by various compilers.


There is no need for the dmd frontend to implement that. The ldc2 
and the gdc compilers can relay on their c++ counterparts to 
provided the c extensions.

-Alex
Apr 17 2021
parent reply Max Haughton <maxhaton gmail.com> writes:
On Saturday, 17 April 2021 at 22:12:41 UTC, 12345swordy wrote:

 On Saturday, 17 April 2021 at 21:32:24 UTC, Walter Bright wrote:

 On 10/6/2020 7:56 PM, Max Haughton wrote:

 Not impossible (it's worth saying that you can literally do 
 this as a library via mixins), but having a c parser in the 
 compiler seems like too much work when it's already fairly 
 trivial to do. If D had a team of full time staff on payroll, 
 then it could be made to work but it adds so much surface 
 area to the compiler.


 You'll also have problems with all the nutburger C extensions 
 implemented by various compilers.


 There is no need for the dmd frontend to implement that. The 
 ldc2 and the gdc compilers can relay on their c++ counterparts 
 to provided the c extensions.

 -Alex


No they can't or at least GCC *maybe* could but LDC is a separate 
project entirely from clang.
Apr 17 2021
parent 12345swordy <alexanderheistermann gmail.com> writes:
On Saturday, 17 April 2021 at 23:22:47 UTC, Max Haughton wrote:

 On Saturday, 17 April 2021 at 22:12:41 UTC, 12345swordy wrote:

 On Saturday, 17 April 2021 at 21:32:24 UTC, Walter Bright 
 wrote:

 On 10/6/2020 7:56 PM, Max Haughton wrote:

 Not impossible (it's worth saying that you can literally do 
 this as a library via mixins), but having a c parser in the 
 compiler seems like too much work when it's already fairly 
 trivial to do. If D had a team of full time staff on 
 payroll, then it could be made to work but it adds so much 
 surface area to the compiler.


 You'll also have problems with all the nutburger C extensions 
 implemented by various compilers.


 There is no need for the dmd frontend to implement that. The 
 ldc2 and the gdc compilers can relay on their c++ counterparts 
 to provided the c extensions.

 -Alex


 No they can't or at least GCC *maybe* could but LDC is a 
 separate project entirely from clang.


Actually they can. It is a question of money/time/resources not 
programming ability here.
Regardless the dmd front-end just need to strictly implement the 
c99 standard(no language extension by the compiler of any sorts) 
to get the ball rolling.

-Alex
Apr 17 2021
prev sibling next sibling parent reply Alexis Kyaw <alexis kyaw.de> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


This would indeed be a total game changer.

I just looked at D in the course of checking out newer 
programming languages and it turns out that I think none of them 
will be really successful replacing C (or C++) because there's no 
direct compatibility with C (Priority 1) and C++ (Priority 2) and 
ideally not only using the C (and potentially C++) binaries 
directly, but also creating header files to allow using the D 
binaries smoothly "on the other side".

There is just some much existing code that you would totally on 
the winner side with this and your totally out in the desert 
without it. Just like the biggest argument for Java is it's 
ecosystem, this would make all C (and maybe even C++) libraries 
part of the D ecosystem.

For this to be really an argument, it must be an in-built feature 
of the language toolchain and a priority in all future 
developments.

Just imagine what this would mean: a clean way forward from 
existing C code to write bulletproof applications on top of all 
the existing code. And a way to use the new code from new 
libraries written in D without any hassle.

A potential external tool would a C to D converter which also 
allows easy migration of C code. Just imagine converting a 
complete Linux implementation to D, compile it and having a full 
D-based Linux.

Then you could start migrating the code base to use the advanced 
features of D and start a new age for the whole industry.

This would make the world listen.

Actually it looks as if this is not such an ambitious goal for D, 
so I'm surprised it's not under way. If had time I'd maybe look 
at it myself, unfortunately I have project work and 3 children, 
leaving really very little time for extra projects.
Apr 17 2021
next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 17 April 2021 at 08:36:41 UTC, Alexis Kyaw wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


 This would indeed be a total game changer.

 I just looked at D in the course of checking out newer 
 programming languages and it turns out that I think none of 
 them will be really successful replacing C (or C++) because 
 there's no direct compatibility with C (Priority 1) and C++ 
 (Priority 2) and ideally not only using the C (and potentially 
 C++) binaries directly, but also creating header files to allow 
 using the D binaries smoothly "on the other side".

 There is just some much existing code that you would totally on 
 the winner side with this and your totally out in the desert 
 without it. Just like the biggest argument for Java is it's 
 ecosystem, this would make all C (and maybe even C++) libraries 
 part of the D ecosystem.

 For this to be really an argument, it must be an in-built 
 feature of the language toolchain and a priority in all future 
 developments.

 Just imagine what this would mean: a clean way forward from 
 existing C code to write bulletproof applications on top of all 
 the existing code. And a way to use the new code from new 
 libraries written in D without any hassle.

 A potential external tool would a C to D converter which also 
 allows easy migration of C code. Just imagine converting a 
 complete Linux implementation to D, compile it and having a 
 full D-based Linux.

 Then you could start migrating the code base to use the 
 advanced features of D and start a new age for the whole 
 industry.

 This would make the world listen.

 Actually it looks as if this is not such an ambitious goal for 
 D, so I'm surprised it's not under way. If had time I'd maybe 
 look at it myself, unfortunately I have project work and 3 
 children, leaving really very little time for extra projects.


+1

Have been experimenting with converting C(++) to D. It's possible 
to some extent.

*dream hat on*

I've thought about this, and what would be best for D is if code 
could be converted. Why? Because otherwise we still just have 
built a lot of stuff upon the old, meaning that the % of D in the 
world hasn't really increased much.

Also, any modifications would be hard etc etc.

What's lacking tho is good tooling around D, like static analysis 
etc etc.

But, if C(++) could be converted to D, *some* parts of the tools 
might be reusable, and you would get a compounding effect after a 
while.

For instance, even *if* C(++) and D could be seamlessly mixed in 
the same source file, you would still end up with much C(++) code.

I still think dpp, dstep etc are 100% needed, and more work 
should definitely be put in it, but the endgame would be to 
somehow increase the amount of D, right? (Btw, what's the state 
of cpp2d?)

Yes, that would mean you end up with two code bases, but that's 
kind of the point in this case. There are obviously + and - with 
this solution. But maybe the endgame deluxe would be to be able 
to reuse the parts that work well and convert the parts that 
doesn't. Best of both worlds.

*dream hat off* <- note
Apr 17 2021
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 17 April 2021 at 08:36:41 UTC, Alexis Kyaw wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


 This would indeed be a total game changer.


Why would one implement a C parser (never mind a C++ one) when 
libclang exists?
Apr 21 2021
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


I've been doing this in both of my compiled languages. (fcc 
first, built-in, and now in cx as a macro.)

For an example of it in action, check out my glfw sample 
https://github.com/FeepingCreature/Cx/blob/master/demos/glfw.cx , 
or the macro code for it 
https://github.com/FeepingCreature/Cx/blob/master/src/cx/macros/cimport.cx .

Three things that jump out:

1. preprocessed headers with `gcc -dD -E` are a lot easier to 
work with, because gcc helpfully unrolls *most* of #define 
definitions, but leaves enough standing so that you can generate 
enums for all the #defines. This is why it's a useful thing to 
have unrestricted IO support, including arbitrary command 
execution, in CTFE. :)

2. "Header C" is actually pretty easy to parse if you just 
studiously ignore all the weird things, weird inside-out type 
format aside.

3. 10% of effort to parse C gets you 99% of value: because the 
vast majority of typing work involved in making a C header 
binding is trivially simple to parse from the header, and the 
rest you can always just do by hand, or (if your language has 
macros) expand as you need it. I presume glibc headers are more 
complicated, but I have been massively positively surprised how 
much of GLFW and GL I could get running with a day or two of 
coding work on this. To a first approximation, once I added 
function pointers, it pretty much all just worked.

I can wholeheartedly recommend any language do this. It unlocks a 
lot of value and removes a lot of manual work.
Apr 18 2021
parent reply evilrat <evilrat666 gmail.com> writes:
On Monday, 19 April 2021 at 06:51:54 UTC, FeepingCreature wrote:

 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:

 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex


 I've been doing this in both of my compiled languages. (fcc 
 first, built-in, and now in cx as a macro.)

 For an example of it in action, check out my glfw sample 
 https://github.com/FeepingCreature/Cx/blob/master/demos/glfw.cx 
 , or the macro code for it 
 https://github.com/FeepingCreature/Cx/blob/master/src/cx/macros/cimport.cx .

 ...

 I can wholeheartedly recommend any language do this. It unlocks 
 a lot of value and removes a lot of manual work.


Yeah, just until it fails to convert something and blocks 
compilation, where no manual fixes is possible or not even 
allowed by design.
Apr 19 2021
parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 19 April 2021 at 09:09:21 UTC, evilrat wrote:

 On Monday, 19 April 2021 at 06:51:54 UTC, FeepingCreature wrote:

 I can wholeheartedly recommend any language do this. It 
 unlocks a lot of value and removes a lot of manual work.


 Yeah, just until it fails to convert something and blocks 
 compilation, where no manual fixes is possible or not even 
 allowed by design.


Yeah I think it differs between using it for system headers and 
using it for library headers, that are probably going to be 
relatively standardized across distributions and compiler 
versions.

That said, at least the p much worst thing that can happen is a 
compiler error. With manual headers, you can end up on a weird 
system where the layout of a header struct is different, or a 
function is implemented as a header macro that calls a different 
function, and then you get linker errors or miscompilation. If 
anything, the headers are probably closer to the "source of 
truth" about the API.
Apr 19 2021