www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - H1 2015 - C++ integration

reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
I'm working on integration of D with the C++ STL (at least the 
linux gnu one).

* You can have a look at a current draft implementation (1).

* There is a name mangling issue in dmd related to the 
compression of usual stl types when the type is const
eg. dmd will mangle 'std::vector<int>::size() const' as 
'_ZNK3std6vectorIiSaIiEE4sizeEv' where it should be mangled 
'_ZNKSt6vectorIiSaIiEE4sizeEv'. Note 'std' is being compressed 
into 'St'.

I believe this is coming from 
https://github.com/D-Programming-Language/dmd/blob/master/src/cppmangle.c#L453 
which tests only the non const type pattern.
According to (2) this substitution seems illegal "Note that 
substitutable components are the represented symbolic constructs, 
not their associated mangling character strings."
This lead to undefined reference when linking and that's why my 
code overrides the name mangling.

* In the video Walter posted recently (3), he states that one 
should use a class to represent a std::string or std::vector in D 
because most of the time we want to have reference semantic. I 
find this a bit counter intuitive for people coming from C++ 
since they are clearly value semantic. std::string and 
std::vector should behave the same in C++ and D to confirm the 
principle of least astonishment.

I started implementing them as struct (4) but then I can only 
have  disable default constructors.

I would like to gather opinions on struct vs class. Any ideas ?

Guillaume
---
1. https://github.com/gchatelet/dlang_cpp_std
2. 
http://mentorembedded.github.io/cxx-abi/abi.html#mangling-compression
3. https://www.youtube.com/watch?v=IkwaV6k6BmM
4. 
https://github.com/gchatelet/dlang_cpp_std/blob/master/std_string.d
Feb 13 2015
next sibling parent reply "Kagamin" <spam here.lot> writes:
It's not like you're free to choose, because struct and class can 
use different mangling.
Feb 13 2015
parent reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Friday, 13 February 2015 at 13:03:18 UTC, Kagamin wrote:

 It's not like you're free to choose, because struct and class 
 can use different mangling.


I'm not sure I get your point. In C++ classes without vtables are 
exactly like structs. Also I don't see any difference between a 
struct or a class name mangling on Gnu Linux.

class/struct S{};
S foo();

foo gets mangled "_Z3foov" if S is a struct or a class.

What did I miss ? Do you have compelling examples ?
Feb 13 2015
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 13 February 2015 at 14:07:44 UTC, Guillaume Chatelet 
wrote:

 I'm not sure I get your point. In C++ classes without vtables 
 are exactly like structs. Also I don't see any difference 
 between a struct or a class name mangling on Gnu Linux.


Do you want it to be compatible with Gnu Linux only?


 What did I miss ? Do you have compelling examples ?


AFAIK, MSVC++. There are other C++ compilers too.
Feb 13 2015
parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Friday, 13 February 2015 at 14:15:10 UTC, Kagamin wrote:

 On Friday, 13 February 2015 at 14:07:44 UTC, Guillaume Chatelet 
 wrote:

 I'm not sure I get your point. In C++ classes without vtables 
 are exactly like structs. Also I don't see any difference 
 between a struct or a class name mangling on Gnu Linux.


 Do you want it to be compatible with Gnu Linux only?


 What did I miss ? Do you have compelling examples ?


 AFAIK, MSVC++. There are other C++ compilers too.


Thx, according to wikipedia 
(http://en.wikipedia.org/wiki/Visual_C%2B%2B_name_mangling#Data_Type) 
VisualC++ is indeed encoding struct and class with U anv V.
Feb 13 2015
prev sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Guillaume Chatelet"  wrote in message 
news:gvnxmwplwkyfrydwrulq forum.dlang.org...


 I'm not sure I get your point. In C++ classes without vtables are exactly 
 like structs. Also I don't see any difference between a struct or a class 
 name mangling on Gnu Linux.

 class/struct S{};
 S foo();

 foo gets mangled "_Z3foov" if S is a struct or a class.

 What did I miss ? Do you have compelling examples ?


This is not a good example, because the return type is clearly not mangled 
into the function name.  (Because you can't overload on return type.)

void foo(S); should show the problem with some mangling schemes.
Feb 13 2015
parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Friday, 13 February 2015 at 18:47:53 UTC, Daniel Murphy wrote:

 "Guillaume Chatelet"  wrote in message 
 news:gvnxmwplwkyfrydwrulq forum.dlang.org...


 I'm not sure I get your point. In C++ classes without vtables 
 are exactly like structs. Also I don't see any difference 
 between a struct or a class name mangling on Gnu Linux.

 class/struct S{};
 S foo();

 foo gets mangled "_Z3foov" if S is a struct or a class.

 What did I miss ? Do you have compelling examples ?


 This is not a good example, because the return type is clearly 
 not mangled into the function name.  (Because you can't 
 overload on return type.)

 void foo(S); should show the problem with some mangling schemes.


You're absolutely right, my bad.

class/struct S {};
void foo(S s){}

foo gets mangled "_Z3foo1S" if S is a struct or a class.
Feb 14 2015
prev sibling next sibling parent reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
I did a few tests. Using a class doesn't work because of the 
added vptr.
The data would be managed at the same time on the D and the C++ 
side.

Structs work however because we can add something like this :

struct std_string {
   void[8] _ = void; // to match sizeof(std::string) and pad the 
object correctly.
}

The padding will be left untouched on the D side because of void 
initializer and will be managed entirely on the C++ side.

Using a class crashes (C++ and D step on each others toes).

Structs work expect for :
- name mangling on linux (bug reported in my first message)
- name mangling on Windows at least (name would be mangled as a 
struct instead of class)
- disabled default constructor.

Mangling on Linux is fixable as well as default construction : we 
can provide a special function that would create a new instance.

But for the windows name mangling, I don't see a way out in the 
language as it is right now.

Any ideas ?
Feb 14 2015
next sibling parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Saturday, 14 February 2015 at 17:24:51 UTC, Guillaume Chatelet 
wrote:

 I did a few tests. Using a class doesn't work because of the 
 added vptr.
 The data would be managed at the same time on the D and the C++ 
 side.

 Structs work however because we can add something like this :

 struct std_string {
   void[8] _ = void; // to match sizeof(std::string) and pad the 
 object correctly.
 }

 The padding will be left untouched on the D side because of 
 void initializer and will be managed entirely on the C++ side.

 Using a class crashes (C++ and D step on each others toes).

 Structs work expect for :
 - name mangling on linux (bug reported in my first message)
 - name mangling on Windows at least (name would be mangled as a 
 struct instead of class)
 - disabled default constructor.

 Mangling on Linux is fixable as well as default construction : 
 we can provide a special function that would create a new 
 instance.

 But for the windows name mangling, I don't see a way out in the 
 language as it is right now.

 Any ideas ?


I'm also using pragma(mangle) and struct with a fake virtual 
table...

;-(

I think that another problem is that, if you use a class in D 
instead of a struct, it's a mess with C++ functions taking a 
reference:

     D - class vector .....
     C++ - foo(const vector& a) ...
     C++ - foo(const vector* a) ...

How do we specify the right mangling for the reference?
---
/P
Feb 14 2015
prev sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Guillaume Chatelet"  wrote in message 
news:fzxoskcrswitmsdsztso forum.dlang.org...


 I did a few tests. Using a class doesn't work because of the added vptr.


This is a bug, D currently adds a vptr even if there are no members.  It's 
just one that doesn't happen to affect ddmd so I never got around to fixing 
it.


 struct std_string {
    void[8] _ = void; // to match sizeof(std::string) and pad the object 
 correctly.
 }

 The padding will be left untouched on the D side because of void 
 initializer and will be managed entirely on the C++ side.


Just remember to be careful that any C++ structs don't rely on interior 
pointers.


 - name mangling on linux (bug reported in my first message)


Is this in bugzilla?


 - name mangling on Windows at least (name would be mangled as a struct 
 instead of class)


We can probably use pragma(mangle) to fix this or add a new pragma for it. 
eg pragma(cpp_class/cpp_struct);


 - disabled default constructor.


Yeeeeah I've always felt like we're going to have to add default struct 
ctors to D eventually.
Feb 14 2015
parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Saturday, 14 February 2015 at 19:54:53 UTC, Daniel Murphy 
wrote:

 - name mangling on linux (bug reported in my first message)


 Is this in bugzilla?


I just created it
https://issues.dlang.org/show_bug.cgi?id=14178
Feb 14 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/13/15 4:23 AM, Guillaume Chatelet wrote:

 * In the video Walter posted recently (3), he states that one should use
 a class to represent a std::string or std::vector in D because most of
 the time we want to have reference semantic. I find this a bit counter
 intuitive for people coming from C++ since they are clearly value
 semantic. std::string and std::vector should behave the same in C++ and
 D to confirm the principle of least astonishment.


Yah, this is still a bit in the air. The point here is that the simplest 
route to getting std::vector working in D is to avoid the many little 
difference between C++ copy ctors and D's postblit. As such, we say: 
pass std::vector by reference from/to C++, and never construct or copy 
one on the D side.

I think that's a usable policy - most of the time containers are not 
supposed to be copied and people must carefully pass them by reference 
everywhere. The annoying part is having one as a member in a D type.

Clearly we need to think this through carefully.


 I started implementing them as struct (4) but then I can only have
  disable default constructors.


I think that can be made to work, too.


 I would like to gather opinions on struct vs class. Any ideas ?


To also reply to your more recent message:


 I did a few tests. Using a class doesn't work because of the added vptr.
 The data would be managed at the same time on the D and the C++ side.


That should work. You don't need any layout information at all for 
std::vector on the D side; all you do is pass a pointer to std::vector 
around D code, and when you want to mess with it you pass the pointer to 
"this" appropriately. It all works; there's no need for D to know the 
layout, only the correct pointer and method signatures.

I think a gating issue right now is handling C++ exceptions in D code. 
C++ stdlib types are not really usable without exceptions.


Andrei
Feb 14 2015
parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Saturday, 14 February 2015 at 18:04:50 UTC, Andrei 
Alexandrescu wrote:

 On 2/13/15 4:23 AM, Guillaume Chatelet wrote:

 * In the video Walter posted recently (3), he states that one 
 should use
 a class to represent a std::string or std::vector in D because 
 most of
 the time we want to have reference semantic. I find this a bit 
 counter
 intuitive for people coming from C++ since they are clearly 
 value
 semantic. std::string and std::vector should behave the same 
 in C++ and
 D to confirm the principle of least astonishment.


 Yah, this is still a bit in the air. The point here is that the 
 simplest route to getting std::vector working in D is to avoid 
 the many little difference between C++ copy ctors and D's 
 postblit. As such, we say: pass std::vector by reference 
 from/to C++, and never construct or copy one on the D side.


I think we can do with 'never copy' but never create seems a bit 
rough.
If you want to call a C++ function that takes a vector you'd need 
to allocate it somehow. And struct would make allocation 
predictable by default.


 I think that's a usable policy - most of the time containers 
 are not supposed to be copied and people must carefully pass 
 them by reference everywhere. The annoying part is having one 
 as a member in a D type.

 Clearly we need to think this through carefully.


Definitely. I think I'll do two implementations and see how far I 
can go with both (class vs struct).
My understanding is that if we go with struct we can allocate on 
the D side and we have value semantic (for what I tested copy 
does work and does not leak memory).


 I did a few tests. Using a class doesn't work because of the 
 added vptr.
 The data would be managed at the same time on the D and the 
 C++ side.


 That should work. You don't need any layout information at all 
 for std::vector on the D side; all you do is pass a pointer to 
 std::vector around D code, and when you want to mess with it 
 you pass the pointer to "this" appropriately. It all works; 
 there's no need for D to know the layout, only the correct 
 pointer and method signatures.


I agree but I see several potential issues :
- you can't control the object lifetime on the D side (can't 
allocate on D side  without a C++ helper function, can't delete 
the object)
- using scoped!std_string will crash.


 I think a gating issue right now is handling C++ exceptions in 
 D code. C++ stdlib types are not really usable without 
 exceptions.


For string and vector a lot of the functions are nothrow, so 
those would be safe to use at least.
Feb 14 2015
prev sibling next sibling parent reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
I updated https://github.com/gchatelet/dlang_cpp_std to provide a 
struct and class implementation of string.
Also added a README to weigh the pros and cons of each, list bugs 
I encountered  and other things we need to think about.
Feb 15 2015
next sibling parent reply "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Sunday, 15 February 2015 at 14:42:46 UTC, Guillaume Chatelet 
wrote:

 I updated https://github.com/gchatelet/dlang_cpp_std to provide 
 a struct and class implementation of string.
 Also added a README to weigh the pros and cons of each, list 
 bugs I encountered  and other things we need to think about.


That's a great work.
Guillaume, I'll try to ask this question again.

I'm wrapping the C++ API of OpenCV [1], and it works great.
I'm using a D struct for std::string instead of D class, because 
I agree with your findings that's more convenient.

But, in case I would like to use a D class for std::string, how 
to call a C++ function that takes a std::string&? It's feasible 
right now?

I think we need more control over the mangle in extern(C++)...

[1] http://docs.opencv.org/trunk/modules/refman.html
Feb 15 2015
parent reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Sunday, 15 February 2015 at 15:37:33 UTC, Paolo Invernizzi 
wrote:

 That's a great work.


Thx :)


 Guillaume, I'll try to ask this question again.

 I'm wrapping the C++ API of OpenCV [1], and it works great.
 I'm using a D struct for std::string instead of D class, 
 because I agree with your findings that's more convenient.

 But, in case I would like to use a D class for std::string, how 
 to call a C++ function that takes a std::string&? It's feasible 
 right now?

 I think we need more control over the mangle in extern(C++)...

 [1] http://docs.opencv.org/trunk/modules/refman.html


I just pushed a new commit which shows how to do it with classes 
as well:
https://github.com/gchatelet/dlang_cpp_std/commit/892a736386ecd84516b7330fef4ee75f1b4d2ad3

This requires to pervert the type system though so it's pretty 
unsafe.
I added the two following helper functions to reinterpret a D 
reference semantic as a C++ value semantic :
const (basic_string*) c_ptr() const { return cast(const 
std_string*)(this); }
ref const(basic_string) c_ref() const { return *cast(const 
std_string*)(this); }

You can then call the C++ function :
getStringSize(s.c_ref);

This is cumbersome and unsafe but workable.

Because the C++ function contains a const ref to string we have 
to provide the correct mangling manually anyways because of 
mangling bug https://issues.dlang.org/show_bug.cgi?id=14178
Feb 15 2015
parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Sunday, 15 February 2015 at 19:44:39 UTC, Guillaume Chatelet 
wrote:

 On Sunday, 15 February 2015 at 15:37:33 UTC, Paolo Invernizzi 
 wrote:

 That's a great work.


 Thx :)


 Guillaume, I'll try to ask this question again.

 I'm wrapping the C++ API of OpenCV [1], and it works great.
 I'm using a D struct for std::string instead of D class, 
 because I agree with your findings that's more convenient.

 But, in case I would like to use a D class for std::string, 
 how to call a C++ function that takes a std::string&? It's 
 feasible right now?

 I think we need more control over the mangle in extern(C++)...

 [1] http://docs.opencv.org/trunk/modules/refman.html


 I just pushed a new commit which shows how to do it with 
 classes as well:
 https://github.com/gchatelet/dlang_cpp_std/commit/892a736386ecd84516b7330fef4ee75f1b4d2ad3

 This requires to pervert the type system though so it's pretty 
 unsafe.
 I added the two following helper functions to reinterpret a D 
 reference semantic as a C++ value semantic :
 const (basic_string*) c_ptr() const { return cast(const 
 std_string*)(this); }
 ref const(basic_string) c_ref() const { return *cast(const 
 std_string*)(this); }

 You can then call the C++ function :
 getStringSize(s.c_ref);

 This is cumbersome and unsafe but workable.

 Because the C++ function contains a const ref to string we have 
 to provide the correct mangling manually anyways because of 
 mangling bug https://issues.dlang.org/show_bug.cgi?id=14178


Thanks for sharing!
---
/P
Feb 15 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/15/15 6:42 AM, Guillaume Chatelet wrote:

 I updated https://github.com/gchatelet/dlang_cpp_std to provide a struct
 and class implementation of string.
 Also added a README to weigh the pros and cons of each, list bugs I
 encountered  and other things we need to think about.


Awesome work, thanks. After further thinking of this, Walter and I 
concluded we should make std::string, std::vector etc. be structs in D 
code as well lest we confuse the heck out of people.

Please file bugs for any constructor/destructor issues you find (I 
notice you filed one already). Walter is up to the task and of course 
I'm counting on others, too.

Let's aim for passable support for 2.068.


Thanks,

Andrei
Feb 15 2015
parent reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Sunday, 15 February 2015 at 19:49:28 UTC, Andrei Alexandrescu 
wrote:

 Please file bugs for any constructor/destructor issues you find 
 (I notice you filed one already). Walter is up to the task and 
 of course I'm counting on others, too.


ctor/dtor C++ naming : 
https://issues.dlang.org/show_bug.cgi?id=14086
invalid stl manlging : 
https://issues.dlang.org/show_bug.cgi?id=14178

Questions :
+ Shall I add one more to allow default ctor for struct when they 
are extern C++ ? (ie. no  disable)
+ I guess this C++ STL binding will end up in phobos. Any idea of 
where this should be put ?
Feb 17 2015
next sibling parent reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
We'll also have to provide a new pragma to allow correct mangling 
of types on Windows. Maybe something like :

extern(C++, std) {

pragma(mangleAs, "class")
struct basic_string()
{
...
}

}
Feb 17 2015
next sibling parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Tuesday, 17 February 2015 at 11:39:06 UTC, Guillaume Chatelet 
wrote:

 We'll also have to provide a new pragma to allow correct 
 mangling of types on Windows. Maybe something like :

 extern(C++, std) {

 pragma(mangleAs, "class")
 struct basic_string()
 {
 ...
 }

 }


Yes please. I'm running into the same issue right now. Changing 
everything to struct on the c++ side is not always possible.
Feb 17 2015
parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Tuesday, 17 February 2015 at 12:09:11 UTC, Benjamin Thaut 
wrote:

 On Tuesday, 17 February 2015 at 11:39:06 UTC, Guillaume 
 Chatelet wrote:

 We'll also have to provide a new pragma to allow correct 
 mangling of types on Windows. Maybe something like :

 extern(C++, std) {

 pragma(mangleAs, "class")
 struct basic_string()
 {
 ...
 }

 }


 Yes please. I'm running into the same issue right now. Changing 
 everything to struct on the c++ side is not always possible.


Created https://issues.dlang.org/show_bug.cgi?id=14193
Feb 17 2015
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/17/15 3:39 AM, Guillaume Chatelet wrote:

 We'll also have to provide a new pragma to allow correct mangling of
 types on Windows. Maybe something like :

 extern(C++, std) {

 pragma(mangleAs, "class")
 struct basic_string()
 {
 ...
 }

 }


Yah, I don't think we can get away without that. -- Andrei
Feb 17 2015
prev sibling parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Tuesday, 17 February 2015 at 11:39:06 UTC, Guillaume Chatelet 
wrote:

 We'll also have to provide a new pragma to allow correct 
 mangling of types on Windows. Maybe something like :

 extern(C++, std) {

 pragma(mangleAs, "class")
 struct basic_string()
 {
 ...
 }

 }


+1 please!
---
Paolo
Feb 17 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/17/15 3:34 AM, Guillaume Chatelet wrote:

 On Sunday, 15 February 2015 at 19:49:28 UTC, Andrei Alexandrescu wrote:

 Please file bugs for any constructor/destructor issues you find (I
 notice you filed one already). Walter is up to the task and of course
 I'm counting on others, too.


 ctor/dtor C++ naming : https://issues.dlang.org/show_bug.cgi?id=14086
 invalid stl manlging : https://issues.dlang.org/show_bug.cgi?id=14178

 Questions :
 + Shall I add one more to allow default ctor for struct when they are
 extern C++ ? (ie. no  disable)


This is tricky. I'd say for now we shouldn't allow default-constructed 
C++ objects. We can later add a special type e.g. CppDefault to mean 
"invoke the C++ default constructor".


 + I guess this C++ STL binding will end up in phobos. Any idea of where
 this should be put ?


core.stdcpp.


Andrei
Feb 17 2015
prev sibling parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Friday, 13 February 2015 at 12:23:58 UTC, Guillaume Chatelet 
wrote:

 I'm working on integration of D with the C++ STL (at least the 
 linux gnu one).

 [...]


I finally took some time to put up a first draft
https://github.com/D-Programming-Language/druntime/pull/1316

There's a bunch of questions in there as well, please destroy :)
Jul 05 2015