www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - My idea for seamless interop to C++

reply Markus <contact markus-lanner.com> writes:
Hi there!

For months I've been thinking on how to escape C++.
Here are my thoughts about that matter:
https://gitlab.com/qwertzui11/d-to-cpp/-/blob/master/README.md

What do think about that idea. Is it worth a shot? I'm thinking 
about implementing such a tool.

Have fun,
Markus
Nov 18 2020
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 18 November 2020 at 14:04:36 UTC, Markus wrote:

 Hi there!

 For months I've been thinking on how to escape C++.
 Here are my thoughts about that matter:
 https://gitlab.com/qwertzui11/d-to-cpp/-/blob/master/README.md

 What do think about that idea. Is it worth a shot? I'm thinking 
 about implementing such a tool.

 Have fun,
 Markus


I think I'm missing a few steps.

How does this line work
fun->doFunStuff();
if MarkdownToHtmlImpl only inherits from MarkdownToHtml and not 
FunUseCase?

Wouldn't you want to make a FunUseCase that has 
MarkdownToHtmlImpl (or MarkdownToHtml) as a member?
Nov 18 2020
parent Markus <contact markus-lanner.com> writes:
On Wednesday, 18 November 2020 at 15:10:53 UTC, jmh530 wrote:

 On Wednesday, 18 November 2020 at 14:04:36 UTC, Markus wrote:
 How does this line work
 fun->doFunStuff();
 if MarkdownToHtmlImpl only inherits from MarkdownToHtml and not 
 FunUseCase?

 Wouldn't you want to make a FunUseCase that has 
 MarkdownToHtmlImpl (or MarkdownToHtml) as a member?


You are completly right. I failed in the example. I fixed it.

It should of course look like this

```cpp
int main(int, char *[]) {
   std::shared_ptr<MarkdownToHtml> converter =
       std::make_shared<MarkdownToHtmlImpl>();
   std::shared_ptr<FunUseCase> fun = FunUseCase.create(converter);
   fun->doFunStuff();
   return 0;
}
```
Nov 18 2020
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Wednesday, 18 November 2020 at 14:04:36 UTC, Markus wrote:

 Hi there!

 For months I've been thinking on how to escape C++.
 Here are my thoughts about that matter:
 https://gitlab.com/qwertzui11/d-to-cpp/-/blob/master/README.md

 What do think about that idea. Is it worth a shot? I'm thinking 
 about implementing such a tool.

 Have fun,
 Markus


Kudos for the detailed comparison between the different C++ 
alternatives!

That said, I'm sorry to disappoint, but I don't think this would 
be a a viable approach (translating D source code to C++ (not 
just headers) while preserving the high-level structure). D is 
also a complex language (just the complexity is much less and 
much better managed compared to C++) and doing a semantically 
correct translation (while even preserving high-level concepts, 
such as OOP) would be quite hard. D's template system is more 
powerful (and more sane) than the one of C++, so you can't 
translate any D code that use templates basically (which is 
pretty much the whole standard and runtime library).

I believe there are 5 approaches / areas worth investing to:
1. dmd's built-in extern (C/C++) ABI and header generation [1] 
[6] [7] [8]
2. dpp [2]
3. dstep [3] (I really wish Atilla would collaborate with Jacob 
on merging dpp and dstep)
4. cling [4] - as far as I know Symmetry have build some sort of 
bridge between D and C++ based on this.
5. Calypso - using Qt from D is nothing sort of astonishing! 
Unfortunately all the work is done by single developer. If others 
would join, hopefully the project could progress much faster.

[1]: https://dlang.org/changelog/2.091.0.html#headers
[2]: https://github.com/atilaneves/dpp
[3]: https://github.com/jacob-carlborg
[4]: https://github.com/root-project/cling
[5]: https://wiki.dlang.org/Calypso

[6]: https://dlang.org/changelog/pending.html#dtoh-improvements | 
permalink: 
https://github.com/dlang/dmd/blob/82cbfec6c72a5ece1b2a9a72d18adcae145d10c7/changelog/dtoh-improvements.dd
[7]: https://dlang.org/changelog/2.094.0.html#cxx-header-modes
[8]: https://dlang.org/changelog/2.092.0.html#cpp_abi_tags

I'm curious to know more on your thoughts on dmd's dtoh.

You can find some sample generated code here:

Big example:
https://github.com/MoonlightSentinel/dmd/blob/ecce5075cbdef1e41b502ccfc01d188ecce4ebc6/src/dmd/frontend.h

Smaller examples:
https://github.com/MoonlightSentinel/dmd/blob/72e5188bd66e4bd8b23f12f0979d66e8354c7d2a/test/compilable/dtoh_cpp98_compat.d

https://github.com/MoonlightSentinel/dmd/blob/72e5188bd66e4bd8b23f12f0979d66e8354c7d2a/test/compilable/dtoh_enum.d

https://github.com/MoonlightSentinel/dmd/blob/72e5188bd66e4bd8b23f12f0979d66e8354c7d2a/test/compilable/dtoh_extern_type.d

https://github.com/MoonlightSentinel/dmd/blob/72e5188bd66e4bd8b23f12f0979d66e8354c7d2a/test/compilable/dtoh_functions.d
Nov 18 2020
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 18 November 2020 at 23:32:00 UTC, Petar Kirov 
[ZombineDev] wrote:


 5. Calypso - using Qt from D is nothing sort of astonishing! 
 Unfortunately all the work is done by single developer. If 
 others would join, hopefully the project could progress much 
 faster.


As far as I recall, this is using libclang built-in into LDC. 
This would limit the usage to LDC.


 [3]: https://github.com/jacob-carlborg


Correction: https://github.com/jacob-carlborg/dstep

--
/Jacob Carlborg
Nov 18 2020
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 19 November 2020 at 07:02:58 UTC, Jacob Carlborg 
wrote:

 On Wednesday, 18 November 2020 at 23:32:00 UTC, Petar Kirov 
 [ZombineDev] wrote:


 5. Calypso - using Qt from D is nothing sort of astonishing! 
 Unfortunately all the work is done by single developer. If 
 others would join, hopefully the project could progress much 
 faster.


 As far as I recall, this is using libclang built-in into LDC. 
 This would limit the usage to LDC.


Yes, this project is completely tied to LDC. However, it is the 
one also offering the tightest C++-interop (except for the 
cling-based approach, for which idk - a dconf presentation would 
be nice). The integration has several noticeable advantages:
1. It is using the C++-based libclang API, which from what I've 
heard offers much more than the C wrapper-based one.
2. I don't know the specifics, but AFAIR, the integration point 
is not actually the parsed AST, but actually more closer to the 
mid-layer, so it has much more semantic information and doesn't 
need to use D's extern (C++) support (though that certainly 
helps), as it can do things on the Clang/LLVM-level.
3. It deeply integrates also with DMDFE in order to bridge the 
clang AST and the DMD AST and even makes additions to D itself, 
in order to enable things like C++ overload resolution.
(I'm not involved with the project, so I may be misremembering or 
misrepresenting some details, but that's some of the high-level 
stuff, AFAIR.)

Unfortunately, 1. and 2. are also the project's biggest weakness 
(given the available man power), as C++ clang, LLVM and DMDFE 
APIs are pretty unstable, so I guess it requires extensive effort 
to rebase to the latest versions.


 [3]: https://github.com/jacob-carlborg


 Correction: https://github.com/jacob-carlborg/dstep


Sorry for the typo.

dpp makes for some cool live demoes, but I've always preferred my 
source code committed to git, rather than just existing in-memory 
(well unless it's well-written CTFE code :D), so dstep has been 
my preferred choice for C HtoD. Huge thank you for it! :)
Nov 18 2020
next sibling parent Jacob Carlborg <doob me.com> writes:
On Thursday, 19 November 2020 at 07:37:52 UTC, Petar Kirov 
[ZombineDev] wrote:


 Sorry for the typo.


No worries :)


 dpp makes for some cool live demoes, but I've always preferred 
 my source code committed to git, rather than just existing 
 in-memory (well unless it's well-written CTFE code :D), so 
 dstep has been my preferred choice for C HtoD. Huge thank you 
 for it! :)


You're welcome. Thanks for those kind words.

--
/Jacob Carlborg
Nov 19 2020
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On Thursday, 19 November 2020 at 07:37:52 UTC, Petar Kirov 
[ZombineDev] wrote:


 Yes, this project is completely tied to LDC. However, it is the 
 one also offering the tightest C++-interop


A side note: looks like Swift is planning for C++ 
interoperability [1]. For the C the interoperability, Swift uses 
the same approach as Calypso. Zig does this as well [2].

[1] 
https://github.com/apple/swift/blob/main/docs/CppInteroperabilityManifesto.md
[2] 
https://ziglang.org/#Integration-with-C-libraries-without-FFIbindings

--
/Jacob Carlborg
Nov 19 2020
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 19 November 2020 at 07:37:52 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Thursday, 19 November 2020 at 07:02:58 UTC, Jacob Carlborg 
 wrote:

 On Wednesday, 18 November 2020 at 23:32:00 UTC, Petar Kirov 
 [ZombineDev] wrote:


 dpp makes for some cool live demoes,


I'm using it in autowrap to translate the Python headers. Still 
some issues with macros though, that I had to translate by hand. 
But, then again, there's no other way to get around this and I'm 
out of ideas wrt to those particular macros.


 but I've always preferred my source code committed to git,


You can! Use `--keep-d-files`. But, like object files, I don't 
know why you'd want to check them in.


 rather than just existing in-memory


They're on the file system, but by default they get deleted after 
the binary is generated.

I should probably change that - since nearly everything has dub 
dependencies, using d++ as a compiler makes little sense, and dub 
needs to see the resulting D files to actually use them.
Nov 19 2020
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 19 November 2020 at 16:36:47 UTC, Atila Neves wrote:

 [snip]

 I'm using it in autowrap to translate the Python headers. Still 
 some issues with macros though, that I had to translate by 
 hand. But, then again, there's no other way to get around this 
 and I'm out of ideas wrt to those particular macros.
 [snip]


Would using some like warp to process the macros first be 
something you have considered?
Nov 19 2020
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 19 November 2020 at 17:29:54 UTC, jmh530 wrote:

 On Thursday, 19 November 2020 at 16:36:47 UTC, Atila Neves 
 wrote:

 [snip]

 I'm using it in autowrap to translate the Python headers. 
 Still some issues with macros though, that I had to translate 
 by hand. But, then again, there's no other way to get around 
 this and I'm out of ideas wrt to those particular macros.
 [snip]


 Would using some like warp to process the macros first be 
 something you have considered?


Yes.

It didn't build. After I edited it to make it build, it didn't 
work.

In any case, I *am* using the C preprocessor on the macros. The 
problem is translating something like this from 
https://github.com/python/cpython/blob/0e62efc51e31c741b61604787aeab4936e6e50e4/Include/object.h#L84:

#define PyObject_HEAD_INIT(type)        \
     { _PyObject_EXTRA_INIT              \
     1, type },

I know how to translate that as a human, but I don't know how to 
write a program that does that for me and that works in the 
general case. Or even most cases.
Nov 21 2020
prev sibling parent Markus <contact markus-lanner.com> writes:
Thx for your great reply!

On Wednesday, 18 November 2020 at 23:32:00 UTC, Petar Kirov 
[ZombineDev] wrote:

 That said, I'm sorry to disappoint, but I don't think this 
 would be a a viable approach (translating D source code to C++ 
 (not just headers) while preserving the high-level structure). 
 D is also a complex language (just the complexity is much less 
 and much better managed compared to C++) and doing a 
 semantically correct translation (while even preserving 
 high-level concepts, such as OOP) would be quite hard. D's 
 template system is more powerful (and more sane) than the one 
 of C++, so you can't translate any D code that use templates 
 basically (which is pretty much the whole standard and runtime 
 library).


I completely agree! D-templates are complex, but unlike in C++ 
manageable. However I'd simply not support them :) It shall not 
be possible to instance a D-template in C++. Keep it simple and 
usable! In the end of the day, that's one thing that makes D so 
awesome. D is not C++ compatible. However I'd like D to be 
compatible with a sane simple subversion of C++. Which is the 
exchange of build-in-types.
So the interop shall ONLY support build-in types, combinations of 
them (`struct`/POD) and non-templated classes. functions and 
methods shall only take and return build-in-types/POD.
In my opinion that's all that's needed for D to work cleanly and 
safe with C++. Everything else would require "magic".
A module shall have a simple interface. That should do for 99% of 
all the cases. This way it would be easily possible to access the 
vast ecosystem of C++. The D-UseCase provides an Interface of 
what it wants, and in C++ you implement that interface to a 
library/framework of your choosing.

So in my opinion, with that subversion of D and C++ a seamless 
interop should be possible and quite helpful. Could I change your 
mind?


 I believe there are 5 approaches / areas worth investing to:
 1. dmd's built-in extern (C/C++) ABI and header generation [1] 
 [6] [7] [8]


I really like the direction this dtoh is going. And thanks for 
the link to the upcoming version, I did not read that one before. 
I tested it, and what I didn't like is that it generates code 
like:

```cpp
virtual void setString(DArray< char > ) = 0;
virtual SomeClass* getSomeClass() = 0;
```

You get raw pointers and custom arrays. I get it why they are 
doing it like that. It's the fastest and most flexible solution.
However, it includes manual memory management, it's unsafe and 
you still have to write an error-prone glue code. In example of 
the string, I'm sure in a real world project you will cast 
`DArray<char>` to `std::string` and vice-versa. Why not do it for 
them, when generating the wrapper. Out of the box awesomness :D 
and more concentration on the problem, not on the language and 
its interfacing.


 2. dpp [2]


Great tool, until C++ gets involved. Epic for C! no doubt.


 3. dstep [3] (I really wish Atilla would collaborate with Jacob 
 on merging dpp and dstep)


Great tool too... for C.


 4. cling [4] - as far as I know Symmetry have build some sort 
 of bridge between D and C++ based on this.
 5. Calypso - using Qt from D is nothing sort of astonishing!


these are C++ to D tools? not D to C++?
Calypso really looks astonishing. will give it a try!

Have fun
Markus
Nov 19 2020
prev sibling parent Markus <contact markus-lanner.com> writes:
On Wednesday, 18 November 2020 at 23:32:00 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Wednesday, 18 November 2020 at 14:04:36 UTC, Markus wrote:

 [...]


 I'm curious to know more on your thoughts on dmd's dtoh.

 [...]


looks like I messed up the "reply" function in this old-school 
forum. My answer to your post is here:
https://forum.dlang.org/post/ymmmhvnmmhfgajhziikd forum.dlang.org
Nov 19 2020