www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - cpp_binder, a not-yet-useful tool for generating C++ bindings

reply Paul O'Neil <redballoon36 gmail.com> writes:
As the title says, cpp_binder is a tool that generates C++ bindings.  It
reads C++ headers and produces a D file filled with "extern(C++)"
declarations.  It can translate a bunch of cool, small examples, but is
not close to being ready for prime-time.  It crashes a lot, especially
in the STL; since the STL is pretty pervasive, I have not successfully
used cpp_binder on an actual C++ library.

I've written more about cpp_binder and my experiences at
http://www.todayman.net/cpp_binder-pre-announcement-and-status.html.

The code is available at https://github.com/todayman/cpp_binder .
cpp_binder still dumps lots of debugging info to stdout and stderr, so
you'll probably want to redirect those somewhere beesides your console.

I hope that this post will spur discussion / decisions / action binding
C++ libraries into D.  I think the language capabilities (e.g.
extern(C++, namespace)) get really far and that the next big push needs
to be on binding real libraries and tools to help.

-- 
Paul O'Neil
Github / IRC: todayman
Sep 20 2015
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 21 September 2015 at 04:22:30 UTC, Paul O'Neil wrote:

 As the title says, cpp_binder is a tool that generates C++ 
 bindings.  It reads C++ headers and produces a D file filled 
 with "extern(C++)" declarations.  It can translate a bunch of 
 cool, small examples, but is not close to being ready for 
 prime-time.  It crashes a lot, especially in the STL; since the 
 STL is pretty pervasive, I have not successfully used 
 cpp_binder on an actual C++ library.

 I've written more about cpp_binder and my experiences at 
 http://www.todayman.net/cpp_binder-pre-announcement-and-status.html.

 The code is available at https://github.com/todayman/cpp_binder 
 . cpp_binder still dumps lots of debugging info to stdout and 
 stderr, so you'll probably want to redirect those somewhere 
 beesides your console.

 I hope that this post will spur discussion / decisions / action 
 binding C++ libraries into D.  I think the language 
 capabilities (e.g. extern(C++, namespace)) get really far and 
 that the next big push needs to be on binding real libraries 
 and tools to help.


This strikes as a most needed project. How come it crashes a lot 
? Glancing quickly at the source, it looks like it is using clang 
as a source of C++ truth.

If you can make it work on the STL, that would be a significant 
step forward for D. I'm very serious.
Sep 20 2015
parent Paul O'Neil <redballoon36 gmail.com> writes:
How come it crashes a lot ?

 Glancing quickly at the source, it looks like it is using clang as a
 source of C++ truth.


It's crashing a lot because I make some incorrect assumptions about
what's in the AST and not everything is handled.  Getting templates
right is hard, and cpp_binder doesn't do it right.  Usually this
manifests as an assertion failure or dereferencing a null pointer.

-- 
Paul O'Neil
Github / IRC: todayman
Sep 21 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-21 06:22, Paul O'Neil wrote:

 As the title says, cpp_binder is a tool that generates C++ bindings.  It
 reads C++ headers and produces a D file filled with "extern(C++)"
 declarations.  It can translate a bunch of cool, small examples, but is
 not close to being ready for prime-time.  It crashes a lot, especially
 in the STL; since the STL is pretty pervasive, I have not successfully
 used cpp_binder on an actual C++ library.

 I've written more about cpp_binder and my experiences at
 http://www.todayman.net/cpp_binder-pre-announcement-and-status.html.


I read your reasons why you're not using DStep. But I still think it's a 
shame that you chose to create a new tool instead of contributing to DStep.

That's the unfortunate issue with the D community. Everyone is creating 
their on projects instead of working together. That's why the progress 
is so slow.

-- 
/Jacob Carlborg
Sep 20 2015
parent reply Paul O'Neil <redballoon36 gmail.com> writes:
 I read your reasons why you're not using DStep. But I still think it's a
 shame that you chose to create a new tool instead of contributing to DStep.
 
 That's the unfortunate issue with the D community. Everyone is creating
 their on projects instead of working together. That's why the progress
 is so slow.
 


How would you address those issues?  My intermediate data structures
have evolved quite a lot and I'm glad that I wasn't dragging dstep along
as I tried to figure it out.

The model now is more similar to dstep than it was at some points.  For
a while, I was writing to multiple files.  This tool has a long way to
go, and it may make sense to move it into dstep in the future.

-- 
Paul O'Neil
Github / IRC: todayman
Sep 21 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-09-22 02:24, Paul O'Neil wrote:


 How would you address those issues?


You mean that no one works together? Or that DStep only has one pass?


 My intermediate data structures
 have evolved quite a lot and I'm glad that I wasn't dragging dstep along
 as I tried to figure it out.
 The model now is more similar to dstep than it was at some points.  For
 a while, I was writing to multiple files.  This tool has a long way to
 go, and it may make sense to move it into dstep in the future.


DStep has only one pass because that's only what I've needed so far. 
Perhaps if I would start adding C++ support I would notice that I need 
multiple passes.

-- 
/Jacob Carlborg
Sep 22 2015
prev sibling next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 21 September 2015 at 04:22:30 UTC, Paul O'Neil wrote:

 I hope that this post will spur discussion / decisions / action 
 binding C++ libraries into D.  I think the language 
 capabilities (e.g. extern(C++, namespace)) get really far and 
 that the next big push needs to be on binding real libraries 
 and tools to help.


How do you deal with overloading of *, &, &&, const *, const& etc?
Sep 21 2015
parent reply ZombineDev <valid_email he.re> writes:
On Monday, 21 September 2015 at 07:45:48 UTC, Ola Fosheim Grøstad 
wrote:

 On Monday, 21 September 2015 at 04:22:30 UTC, Paul O'Neil wrote:

 I hope that this post will spur discussion / decisions / 
 action binding C++ libraries into D.  I think the language 
 capabilities (e.g. extern(C++, namespace)) get really far and 
 that the next big push needs to be on binding real libraries 
 and tools to help.


 How do you deal with overloading of *, &, &&, const *, const& 
 etc?


I guess that the only thing that doesn't map directly to D is &&. 
The others look on the D side like this:
C++      -> D
T*       -> T*
T&       -> ref T
const T* -> const T*
const T& -> const ref T

I think that T&& actually looks like the T& calling convention 
ABI-wise. The difference being that T&& functions are mangled 
differently so that you can overload them. The challenge is to be 
able to select them from the overload set. Perhaps the path of 
least resistance (for now) is to generate a .cpp (not a 
header-only library) containing a wrapper for every function that 
takes T&&. For example:

// C++ wrapper:

// Wrapper for `vector<T>::push_back( T&& value )`

void push_back_from_rvalue(vector<T>& this_, T& to_be_moved)
{
     this_.push_back(std::move(to_be_moved));
}

// D side:

// Let's say we want to use some move-only type
// like std::thread and push it into a std::vector:
auto my_vec = stdcpp.vector.vector!(stdcpp.Thread);

// The case with rvalue temporary:
// my_vec.push_back_from_rvalue(Thread(&theadFunc));
// ^ creates a temporary Thread object
// and moves it into the vector::push_back call stack.
// Implemented like this:
// (invisible to the user:)
Thread __rvalue_tmp = Thread(&theadFunc);
my_vec.push_back_from_rvalue(__tmp);

The case with lvalue whose ownership is transferred:
auto lvalue_thread_obj = Thread(&theadFunc);

// lvalue_thread_obj is passed by mutable reference
my_vec.push_back_from_rvalue(lvalue_thread_obj);
// (invisible to the user:)
lvalue_thread_obj.__cpp_dtor();
// Unsafe to use lvalue_thread_obj from now on...
Sep 21 2015
next sibling parent reply ZombineDev <valid_email he.re> writes:
On Monday, 21 September 2015 at 09:45:01 UTC, ZombineDev wrote:

 [...]


Another option is to introduce a special built-in function like 
std::move that just selects the rvalue overload. This would 
require a language change, but would save us from adding && type 
qualifiers (which is a far more invasive change with little 
benefit) and would not be complicated to implement in the 
compiler. For example:

auto my_vec = stdcpp.vector.vector!(stdcpp.thread.thread);
my_vec.push_back(__cpp_move(stdcpp.thread.thread(&threadFunc)));
//                 ^--- selects the correct && overload
Sep 21 2015
parent ZombineDev <valid_email he.re> writes:
On Monday, 21 September 2015 at 09:59:44 UTC, ZombineDev wrote:

 On Monday, 21 September 2015 at 09:45:01 UTC, ZombineDev wrote:

 [...]


 Another option is to introduce a special built-in function like 
 std::move that just selects the rvalue overload. This would 
 require a language change, but would save us from adding && 
 type qualifiers (which is a far more invasive change with 
 little benefit) and would not be complicated to implement in 
 the compiler. For example:

 auto my_vec = stdcpp.vector.vector!(stdcpp.thread.thread);
 my_vec.push_back(__cpp_move(stdcpp.thread.thread(&threadFunc)));
 //                 ^--- selects the correct && overload


However we would also need a way to mark a declaration as an 
rvalue overload. Perhaps it can be as simple as:

// D:
extern(C++, namespace, rvalue_params(1, 2))
void fun(const ref T, ref T, ref T, T);

// C++
void fun(const T&, T&&, T&&, T);
Sep 21 2015
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 21 September 2015 at 09:45:01 UTC, ZombineDev wrote:

 I guess that the only thing that doesn't map directly to D is 
 &&. The others look on the D side like this:
 C++      -> D
 T*       -> T*
 T&       -> ref T
 const T* -> const T*
 const T& -> const ref T


I don't think D const and C++ const share semantics. In C++ you 
can write to things pointed to by T even if T is const. You also 
have no guarantee that a "const T" isn't mutated by another 
thread, I believe.

A C++ pointer is what D would call a shared pointer, whereas C++ 
does not have the concept of thread local pointers.

etc.
Sep 21 2015
parent reply Paul O'Neil <redballoon36 gmail.com> writes:
On 09/21/2015 12:01 PM, Ola Fosheim Grøstad wrote:

 On Monday, 21 September 2015 at 09:45:01 UTC, ZombineDev wrote:

 I guess that the only thing that doesn't map directly to D is &&. The
 others look on the D side like this:
 C++      -> D
 T*       -> T*
 T&       -> ref T
 const T* -> const T*
 const T& -> const ref T


 
 I don't think D const and C++ const share semantics. In C++ you can
 write to things pointed to by T even if T is const. You also have no
 guarantee that a "const T" isn't mutated by another thread, I believe.
 
 A C++ pointer is what D would call a shared pointer, whereas C++ does
 not have the concept of thread local pointers.
 
 etc.
 


While D and C++ const don't quite share semantics, they're petty close
and they mangle the same way.  I do what ZombieDev has in the table.

I completely bail on && and intentionally don't bind those things.

-- 
Paul O'Neil
Github / IRC: todayman
Sep 21 2015
parent reply Ola Fosheim Grostad <ola.fosheim.grostad+dlang gmail.com> writes:
On Tuesday, 22 September 2015 at 00:31:45 UTC, Paul O'Neil wrote:

 While D and C++ const don't quite share semantics, they're 
 petty close and they mangle the same way.  I do what ZombieDev 
 has in the table.


Going from const to mutable like that breaks the type system...
Sep 21 2015
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 22 September 2015 at 04:07:54 UTC, Ola Fosheim 
Grostad wrote:

 On Tuesday, 22 September 2015 at 00:31:45 UTC, Paul O'Neil 
 wrote:

 While D and C++ const don't quite share semantics, they're 
 petty close and they mangle the same way.  I do what ZombieDev 
 has in the table.


 Going from const to mutable like that breaks the type system...


Can pragma(mangle, ...) be used on types? Then we can define a 
C/C++ compatible `HeadConst` template that simulates C/C++ const 
semantics, while still being mangled correctly:

template HeadConst(T) {
     pragma(mangle, const(T).mangleof)
     struct HeadConst {
         //  disable opAssign();
         // and whatever else is necessary for head-const
     }
}

The constructor would of course need to make sure that it accepts 
only types with non-const indirection.
Sep 22 2015
parent reply Kagamin <spam here.lot> writes:
On Tuesday, 22 September 2015 at 11:04:17 UTC, Marc Schütz wrote:

 Can pragma(mangle, ...) be used on types? Then we can define a 
 C/C++ compatible `HeadConst` template that simulates C/C++ 
 const semantics, while still being mangled correctly


I believe I've seen const declarations in C incorrectly declared 
as head const when transitive const was really wanted, but it 
wasn't done because transitive const in C is so much PITA. So the 
only real concern here is mangling.
Sep 22 2015
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 22 September 2015 at 14:46:24 UTC, Kagamin wrote:

 I believe I've seen const declarations in C incorrectly 
 declared as head const when transitive const was really wanted, 
 but it wasn't done because transitive const in C is so much 
 PITA. So the only real concern here is mangling.


That's a rather orthodox viewpoint... If D const is to have any 
value then the optimizer should be allowed to cache computations 
and elide memory barriers.

The consequences of a C function mutating something reachable 
through const is either a disaster or the D compiler will have to 
forget about those kind of optimizations after calling a C 
function. But then const has very little value...
Sep 22 2015
parent reply Kagamin <spam here.lot> writes:
On Tuesday, 22 September 2015 at 14:53:54 UTC, Ola Fosheim 
Grøstad wrote:

 The consequences of a C function mutating something reachable 
 through const is either a disaster or the D compiler will have 
 to forget about those kind of optimizations after calling a C 
 function. But then const has very little value...


Another C feature you didn't know about: the standard allows to 
create a mutable pointer to const data and mutate it. But most of 
the time when you see const in C, it works as expected, so 
usually there shouldn't be a problem except for mangling.
Sep 22 2015
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 22 September 2015 at 15:11:55 UTC, Kagamin wrote:

 Another C feature you didn't know about: the standard allows to 
 create a mutable pointer to const data and mutate it.


I didn't? Of course I did. In C++ other threads can write to 
const objects. The function receiving a const objects is not 
supposed to write to const. Const cast is supposed to be used for 
calling legacy const interfaces without a const signature.


 But most of the time when you see const in C, it works as 
 expected, so usually there shouldn't be a problem except for 
 mangling.


No it doesn't. C++ even has an override for const object fields 
called "mutable", so that you can have things like reference 
counts in const objects.

Either D const is _always_ ensured or the compiler have to assume 
that it is _always_ violated after FFI.
Sep 22 2015
parent reply Kagamin <spam here.lot> writes:
On Tuesday, 22 September 2015 at 15:33:40 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 22 September 2015 at 15:11:55 UTC, Kagamin wrote:

 Another C feature you didn't know about: the standard allows 
 to create a mutable pointer to const data and mutate it.


 I didn't? Of course I did. In C++ other threads can write to 
 const objects. The function receiving a const objects is not 
 supposed to write to const. Const cast is supposed to be used 
 for calling legacy const interfaces without a const signature.


Of course you didn't. In C you can mutate const object without 
cast. But it's not an issue because it's not what is usually done 
and usually const works as expected.
Sep 22 2015
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 22 September 2015 at 15:53:22 UTC, Kagamin wrote:

 Of course you didn't. In C you can mutate const object without 
 cast. But it's not an issue because it's not what is usually 
 done and usually const works as expected.


No. Const objects are read only. Writing to read only objects 
would be undefined behaviour. I.e. illegal and could make your 
computer crash if it is a hardware register or memory that does 
not allow writing. Const references can exhibit aliasing if they 
aren't marked as "restrict":

http://stackoverflow.com/questions/457441/does-restrict-help-in-c-if-a-pointer-is-already-marked-const
Sep 22 2015
parent Kagamin <spam here.lot> writes:
On Tuesday, 22 September 2015 at 16:24:33 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 22 September 2015 at 15:53:22 UTC, Kagamin wrote:

 Of course you didn't. In C you can mutate const object without 
 cast. But it's not an issue because it's not what is usually 
 done and usually const works as expected.


 No. Const objects are read only. Writing to read only objects 
 would be undefined behaviour.


That's why there's no choice other than trust developer to do the 
expected thing.
Sep 23 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-21 06:22, Paul O'Neil wrote:


 I've written more about cpp_binder and my experiences at
 http://www.todayman.net/cpp_binder-pre-announcement-and-status.html.


About the plugin architecture. I tried modifying DMD to run DStep to 
automatically generate bindings and use them directly. I announced it as 
a proof of concept [1], but the community seemed to prefer having it as 
a separate tool if I recall correctly.

[1] http://forum.dlang.org/thread/ks3kir$1ubq$1 digitalmars.com

-- 
/Jacob Carlborg
Sep 21 2015
parent Paul O'Neil <redballoon36 gmail.com> writes:
On 09/21/2015 05:29 AM, Jacob Carlborg wrote:

 On 2015-09-21 06:22, Paul O'Neil wrote:
 

 I've written more about cpp_binder and my experiences at
 http://www.todayman.net/cpp_binder-pre-announcement-and-status.html.


 
 About the plugin architecture. I tried modifying DMD to run DStep to
 automatically generate bindings and use them directly. I announced it as
 a proof of concept [1], but the community seemed to prefer having it as
 a separate tool if I recall correctly.
 
 [1] http://forum.dlang.org/thread/ks3kir$1ubq$1 digitalmars.com
 


I don't like it either, but it deals with uninstantiated templates...

It also means that you can handle the curiously recurring template
pattern.  If you cant instantiate the template in the D compiler, then
you need to generate C++ code that expresses the D structure and
properly instantiates the C++ template.  The payoff here is bigger.

-- 
Paul O'Neil
Github / IRC: todayman
Sep 21 2015