digitalmars.D.announce - (Oh My) Gentool 0.3.0 released
- evilrat (31/31) May 04 2021 (Oh My) Gentool - Yet another C/C++ binding generator.
- user1234 (7/15) May 05 2021 I have a technical question about the tool itself. It is mostly
- Dominikus Dittes Scherkl (3/5) May 05 2021 Oh dear!
- evilrat (16/21) May 05 2021 To answer both:
- user1234 (3/21) May 05 2021 Thanks for the explanations. BTW I had the same question for LDC
- Dominikus Dittes Scherkl (4/10) May 06 2021 Yeah, but unlike LDC, for a tool that translates C++ to D it is a
- Jacob Carlborg (6/8) May 07 2021 If I understand correctly, the Zig compiler is implemented partially in
- evilrat (17/23) May 07 2021 And it quickly becomes insufficient using only the C API as
- Jacob Carlborg (7/11) May 07 2021 Yes, it's possible to use libclang. DStep [1] is using that and it fully...
(Oh My) Gentool - Yet another C/C++ binding generator. It is a tool to convert C/C++ code to D usable form. It takes JSON config, basically all C++ compiler flags and switches, and outputs extern(C++) declarations, (hopefully) in usable form D! It can already process (dear) imgui library (immediate mode GUI popular in game development and various graphics related tools and demos) without manual fixes! It can process recastnavigation (navmesh generation and pathfinding library) with just a few manual edits. Please note that it is still in its early stage and may contain bugs and missing language features, as well as lack of conversion for certain language constructs. It is still hard to use it directly in the build process on a real libraries due to many syntax and semantics issues, however it is already a valuable tool for making thin wrappers on C++ side to quickly bring them to your D code, given that your wrapper headers does not contains complex bodies or templates, or direct inclusions of other libraries headers such as Boost(ok, no STL too). That's it, even if it produces incomplete translation this could reduce bindings making process from hours down to minutes! Who wants to spent 10 hours manually making bindings for entire PhysX when it can be reduced to just 30 minutes? Absolutely no one! Grab one today and stay ahead of your competitors with regular updates! How to start https://github.com/Superbelko/ohmygentool/wiki/QuickStart Source https://github.com/Superbelko/ohmygentool Windows build https://github.com/Superbelko/ohmygentool/releases/tag/v0.3.0
May 04 2021
On Wednesday, 5 May 2021 at 06:50:29 UTC, evilrat wrote:(Oh My) Gentool - Yet another C/C++ binding generator. It is a tool to convert C/C++ code to D usable form. It takes JSON config, basically all C++ compiler flags and switches, and outputs extern(C++) declarations, (hopefully) in usable form D! ... Source https://github.com/Superbelko/ohmygentool ...I have a technical question about the tool itself. It is mostly written in cpp. Is it possible to use libclang and more generally LLVM c++ api [directly in D](https://dlang.org/spec/cpp_interface.html) or the Cpp interface is too limited ? Was this an option, have you tried ?
May 05 2021
On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:I have a technical question about the tool itself. It is mostly written in cpp.Oh dear! Isn't it possible to use it to translate itself into D?
May 05 2021
On Wednesday, 5 May 2021 at 10:35:23 UTC, Dominikus Dittes Scherkl wrote:On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:To answer both: clang has lots of templates, sometimes not so trivial ones, its code base filled with C++ constructs that does not have nice one to one translation(or simply a C++ specific detail like alignment), and the most annoying part - it has various 'tables' generated as a build step that is a mix of external tools and macros. Basically every template will need some care, likely it will be easier to just emit C++ stubs that will force compiler to emit actual code to link with than trying to translate them. So no, not currently possible. Definitely not an unpaid job, well I'll still reject it even if it is paid one. Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this.I have a technical question about the tool itself. It is mostly written in cpp.Oh dear! Isn't it possible to use it to translate itself into D?
May 05 2021
On Wednesday, 5 May 2021 at 11:51:27 UTC, evilrat wrote:On Wednesday, 5 May 2021 at 10:35:23 UTC, Dominikus Dittes Scherkl wrote:Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar.On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:... To answer both: clang has lots of templates, sometimes not so trivial ones, its code base filled with C++ constructs that does not have nice one to one translation(or simply a C++ specific detail like alignment), and the most annoying part - it has various 'tables' generated as a build step that is a mix of external tools and macros. Basically every template will need some care, likely it will be easier to just emit C++ stubs that will force compiler to emit actual code to link with than trying to translate them. So no, not currently possible. Definitely not an unpaid job, well I'll still reject it even if it is paid one. Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this.
May 05 2021
Dominikus Dittes Scherkl <dominikus scherkl.de> On Wednesday, 5 May 2021 at 11:54:51 UTC, user1234 wrote:On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:Yeah, but unlike LDC, for a tool that translates C++ to D it is a goal being able to also translate itself from C++ to D. And reaching this goal would be a huge milestone!Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this.Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar.
May 06 2021
On 2021-05-05 13:54, user1234 wrote:Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar.If I understand correctly, the Zig compiler is implemented partially in Zig. It use the LLVM C API and some wrappers C around the C++ API where the C API is not sufficient. -- /Jacob Carlborg
May 07 2021
On Friday, 7 May 2021 at 18:17:36 UTC, Jacob Carlborg wrote:On 2021-05-05 13:54, user1234 wrote:And it quickly becomes insufficient using only the C API as feature complexity increases. No idea if Zig has to deal with C++ compiler (clang) or all it needs is pure LLVM, because the latter should have more or less feature rich C API, unlike clang that deals with C++ and has ever "unstable" API. Making wrappers for missing parts still will be a PITA, as having to pass around smart pointers definitely does not makes it easier. Anyway like I said, for bootstrap goal it is probably easier to re-purpose the tool to make thin wrappers & stubs on C++ side, pretty much just like SWIG does. Side note: But all this does not compares to what potential D to nextgen-language bindings making process would look like, as D feature set makes it even harder to translate, esp. stuff like templates and CTFE, now add static if's to that and it becomes a real mess.Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar.If I understand correctly, the Zig compiler is implemented partially in Zig. It use the LLVM C API and some wrappers C around the C++ API where the C API is not sufficient.
May 07 2021
On 2021-05-05 12:01, user1234 wrote:Is it possible to use libclang and more generally LLVM c++ api [directly in D](https://dlang.org/spec/cpp_interface.html) or the Cpp interface is too limited ? Was this an option, have you tried ?Yes, it's possible to use libclang. DStep [1] is using that and it fully written in D. Although DStep cannot create bindings for C++ yet so I cannot guarantee that using only libclang will work for C++ code. [1] https://github.com/jacob-carlborg/dstep -- /Jacob Carlborg
May 07 2021
On Friday, 7 May 2021 at 18:15:47 UTC, Jacob Carlborg wrote:On 2021-05-05 12:01, user1234 wrote:I don't think using libclang is a good idea for C++. I think it works very well and is the ideal choice (due to lack of dependencies and infra) for C, but C++ has a nightmarish AST. Have been investigating doing this with LibTooling using the C++ API, however a bit differently than has been done before (will explain in a minute below). The reasoning is that, when I spoke with LLVM developers, and developers who had built successful codegen tooling for C++, every one of them cautioned against using libclang, said that they regretted the choice, and that the API and AST info it exposes is not sufficient.Is it possible to use libclang and more generally LLVM c++ api [directly in D](https://dlang.org/spec/cpp_interface.html) or the Cpp interface is too limited ? Was this an option, have you tried ?Yes, it's possible to use libclang. DStep [1] is using that and it fully written in D. Although DStep cannot create bindings for C++ yet so I cannot guarantee that using only libclang will work for C++ code. [1] https://github.com/jacob-carlborg/dstep_I had a chance to ask Atila this same thing, last Beerconf, and that was also his stance -- roughly that a large part of the hurdles with dpp (beyond C++ being a nightmare to start) were that it was built on libclang._--- I have a large personal interest in building/helping to build a viable C++-header-to-D-`extern (C++)` generator because I think that is what would unlock a vast amount of potential + power for D. Quick, no-hassle direct bindings to any (or even most) C++ libraries and tools. But I've been doing my homework both on how to approach this and what the state of C++ bindgen is in D. I've tried: - D++ - Dstep - Ohmygentool - SWIG, with and without "Directors" feature enabled - CPP2D Of those, I've had the best success C++ using Ohmygentool by a fairly large margin. It can often do several-thousand line projects with non-trivial `#includes` mostly automatically. --- **However, I had an idea which I haven't seen tried yet, and have been prototyping:** - Using `cppyy` in Python (which uses `cling`) for runtime bindings to C++ and ability to write raw C++ code in Python strings and JIT compile it. - Allow users to write "drivers"/"clients" in Python which do the codegen. Since Python isn't compiled, this means you can realtime tweak and visualize your output much faster than manually recompiling a C++ based LibTooling application. I am thinking of some kind of API where you can declare rules using annotations for AST nodes above functions for handling them. Something like: ```py class DCodegen: Clang/LibTooling's AST API rule(lambda t: t.is_pointer() or t.is_reference() and \ t.pointee().is_record_indirection()) def input(cls, t, args): return f"{{interm}} = &{c_util.struct_cast(t, '{inp}')};" rule(lambda t: t.is_pointer() or t.is_reference()) def input(cls, t, args): raise ValueError("unsupported input pointer/reference type {}".format(t)) ``` This would allow people to contribute or tweak the codegen to their liking very rapidly. For distribution, it could be done in an Ubuntu Docker container that comes with LLVM and Python in it, and the scripts, then mapped to local filesystem for read access + also if you want to edit the `DCodegen` script or supply your own Python file as the driver. --- **What do you all think of this idea?** - Would love to hear feedback/opinions. - I unfortunately have neither the D nor C++ expertise to properly write the translation rules, so it would take some help/collaboration from the community. --- Here's an example of loading LibTooling in Python and building an AST from some code. You can see the C++ object in the `print()` output in the top right: -  - 
May 09 2021
On Sunday, 9 May 2021 at 19:35:52 UTC, Gavin Ray wrote:**However, I had an idea which I haven't seen tried yet, and have been prototyping:** - Using `cppyy` in Python (which uses `cling`) for runtime bindings to C++ and ability to write raw C++ code in Python strings and JIT compile it. - Allow users to write "drivers"/"clients" in Python which do the codegen. Since Python isn't compiled, this means you can realtime tweak and visualize your output much faster than manually recompiling a C++ based LibTooling application. I am thinking of some kind of API where you can declare rules using annotations for AST nodes above functions for handling them. Something like: ```py class DCodegen: Clang/LibTooling's AST API rule(lambda t: t.is_pointer() or t.is_reference() and \ t.pointee().is_record_indirection()) def input(cls, t, args): return f"{{interm}} = &{c_util.struct_cast(t, '{inp}')};" rule(lambda t: t.is_pointer() or t.is_reference()) def input(cls, t, args): raise ValueError("unsupported input pointer/reference type {}".format(t)) ``` This would allow people to contribute or tweak the codegen to their liking very rapidly.It's all libclang under the hood though. Never looked at the sources of cppyy or cling, but very likely this works for them because of libs such as pybind11 that wraps C++ stuff in C++, while the other tools trying to rely on C API(very limited) or keep fighting with C++ API, you've already seen that all that tools from that list except gentool is using C API and what capabilities each provides. Can't say I hate that idea, but it has same issues as SWIG, writing any non trivial rule becomes next to impossible as there is practically zero examples and very poor documentation, the whole process becomes trial and error marathon without chance to win, and it is basically write-only code that is as worse as C++ templates. But it is definitely better than SWIG in that regard as you can get type and functions information using dir() and help() and your trusty IDE with debugger. My current plan though is to provide predefined pre-generate and post-generate rules that is applied declaratively in project config, for example `ignoreDecls *::new[]` that will ignore all new operator overloads in any namespace, or one of any other existing rules that deals with specific patterns, and at some point later allow users to write their own rules like you described. After all this is binding/translator tool, not an universal one-for-all code generator.For distribution, it could be done in an Ubuntu Docker container that comes with LLVM and Python in it, and the scripts, then mapped to local filesystem for read access + also if you want to edit the `DCodegen` script or supply your own Python file as the driver.No way, Docker is too heavy and not very user friendly or even CI friendly. It is ok to have builds optionally packed in container, but not as the only way to distribute. It might work for cppyy/cling because they rely on specific dynamic library properties or fork process on *NIX that is not there on Windows.
May 10 2021
On Monday, 10 May 2021 at 07:11:26 UTC, evilrat wrote:Can't say I hate that idea, but it has same issues as SWIG, writing any non trivial rule becomes next to impossible as there is practically zero examples and very poor documentation, the whole process becomes trial and error marathon without chance to win, and it is basically write-only code that is as worse as C++ templates. But it is definitely better than SWIG in that regard as you can get type and functions information using dir() and help() and your trusty IDE with debugger.Yeah, this is true. I actually ran into this issue already -- you have extreme devspeed + flexibility with Python but you're essentially "blind" as to the API and it's a lot of trial-and-error if it's not a very trivial C++ library. I think the end goal of being able to provide some way to easily hook into the codegen pipeline so that users can make tweaks easily & contribute them back, or customize it for personal use is probably more important than the "how". This was just the best "how" I could come up with at the time, hah.My current plan though is to provide predefined pre-generate and post-generate rules that is applied declaratively in project config, for example `ignoreDecls *::new[]` that will ignore all new operator overloads in any namespace, or one of any other existing rules that deals with specific patterns, and at some point later allow users to write their own rules like you described. After all this is binding/translator tool, not an universal one-for-all code generator.What about adding `cling` for interpreted JIT C++ without handicaps? https://blog.llvm.org/posts/2021-03-25-cling-beyond-just-interpreting-cpp/ This would let both you and users rapidly develop/iterate-on the codegen part without needing to recompile the entire solution. The main application hosts the `cling` process, and you don't do anything except bring the codegen bits in as JIT-interpreted files. It's not like performance is critical here, you know?😅 The end result is that you (or anyone else) can modify the actual codegen code, and re-run the compiled binary without needing to do anything else. `cling` will take care of acting as the engine to sort out `#include`'s, template instatiations, global context, etc. I dunno, I don't really know that much about this area at all. So there's also a good chance this idea couldn't/wouldn't work, and is not a good one -- but in my naive head it sounds good!No way, Docker is too heavy and not very user friendly or even CI friendly. It is ok to have builds optionally packed in container, but not as the only way to distribute. It might work for cppyy/cling because they rely on specific dynamic library properties or fork process on *NIX that is not there on Windows.Yeah, so the issue there was that `cppyy` needs dynamic libs to be able to work. LibTooling hilariously enough can't even be compiled as a dynamic lib on Windows, since there's a limit of ~65,000 exported visible symbols and `LINK.exe` will error. So between wanting the user to have a fairly recent, self-compiled version of LLVM, and not working on Windows at all, the easiest thing seemed to be to ask "Hey, just do `docker run` and that'll run an Ubuntu image with LLVM 13 built with LLVM_ENABLE_SHARED_LIBS and Python 3 and all of this set up for you." Not ideal but also I think maybe the only way to make the `cppyy` thing work haha.
May 10 2021