www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - WeakRefs for a CPP->D wrapper

reply "Abdulhaq" <alynch4047 gmail.com> writes:
I'm implementing a wrapper program for wrapping generic C++ 
libraries - it's going very well (subclassing, virtual methods, 
nested classes, enums working) but I've hit an area that I don't 
have enough D experience to answer, and am hoping someone can 
point me in the right direction. My background is lots of 
professional Java, Python etc., but not so much writing of C++ 
(plenty of reading but not writing).

I need to maintain a mapping between C++ void* addresses and D 
wrapper Objects. The naive implementation would be

Object[void*] wrappingRegistry;

but of course that prevents the wrapped objects from being 
garbage collected - I need weak ref semantics.

I had a go at making it e.g.
ulong[ulong] and storing the cast(ulong) address of the D object, 
but it seems that I don't understand what taking the address of 
an obj (&obj) actually is doing - it doesn't seem to point to the 
memory occupied by the object but instead to the address of the 
variable pointing to the object?

I've had a good google around and because my understanding is 
poor in this area (GC innards, shared data, D pointers etc) I 
can't identify which of the implementations around is best for 
2.064/2.065 and moving forwards. My best guess is the WeakRef 
found in this one:

https://github.com/phobos-x/phobosx/blob/master/source/phobosx/signal.d

by Robert in his signals2 implementation. Can anyone expain what 
is going on in WeakRef and InvisibleAddress?

thanks for any help you can give me.
Jan 11 2014
parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
 Object[void*] wrappingRegistry;

 but of course that prevents the wrapped objects from being 
 garbage collected - I need weak ref semantics.

 I had a go at making it e.g.
 ulong[ulong] and storing the cast(ulong) address of the D 
 object, but it seems that I don't understand what taking the 
 address of an obj (&obj) actually is doing - it doesn't seem to 
 point to the memory occupied by the object but instead to the 
 address of the variable pointing to the object?


class X {};
X x;

x is an reference to an instance of X, with other words a pointer 
without arithmetic but with syntax sugar. &x will take the 
address of this pointer/reference. If you want the address of the 
actual instance, you can use cast(void*) for example.
Jan 11 2014
parent reply "Abdulhaq" <alynch4047 gmail.com> writes:
On Saturday, 11 January 2014 at 20:17:14 UTC, Tobias Pankrath 
wrote:

 class X {};
 X x;

 x is an reference to an instance of X, with other words a 
 pointer without arithmetic but with syntax sugar. &x will take 
 the address of this pointer/reference. If you want the address 
 of the actual instance, you can use cast(void*) for example.


Hi Tobias, can casting the address to void* make a difference to 
its value?

Here's an example of what I don't understand:

import std.stdio;
import std.string: format;

class Foo {
	int x;
}

void printAddress(Foo foo) {
	writeln("Address of parameter foo is %x".format(&foo));
	writeln("Address of parameter foo cast to void* is 
%x".format(cast(void*) &foo));
}

void main() {
	auto foo = new Foo();
	writeln("Address of foo is %x".format(&foo));
	writeln("Address of foo cast to void* is %x".format(cast(void*) 
&foo));
	printAddress(foo);
}

When run I get:

Address of foo is 7fff40ac4558
Address of foo cast to void* is 7fff40ac4558
Address of parameter foo is 7fff40ac4538
Address of parameter foo cast to void* is 7fff40ac4538

So why is the address of the parameter foo different to the 
address of main foo, when they both refer to the same object?

thanks
Jan 11 2014
parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
On Saturday, 11 January 2014 at 20:38:33 UTC, Abdulhaq wrote:

 On Saturday, 11 January 2014 at 20:17:14 UTC, Tobias Pankrath 
 wrote:

 class X {};
 X x;

 x is an reference to an instance of X, with other words a 
 pointer without arithmetic but with syntax sugar. &x will take 
 the address of this pointer/reference. If you want the address 
 of the actual instance, you can use cast(void*) for example.


 Hi Tobias, can casting the address to void* make a difference 
 to its value?


No, try this:
import std.stdio;
class X {}
void foo(X x) { writeln(cast(void*) x); }

void main() {
          X x; // null reference by default.
          writeln(cast(void*) x);
          foo(x);

          x = new X;
          writeln(cast(void*) x);
          foo(x);
}
Jan 11 2014
parent reply "Abdulhaq" <alynch4047 gmail.com> writes:
 No, try this:
 import std.stdio;
 class X {}
 void foo(X x) { writeln(cast(void*) x); }

 void main() {
          X x; // null reference by default.
          writeln(cast(void*) x);
          foo(x);

          x = new X;
          writeln(cast(void*) x);
          foo(x);
 }


Thanks Tobias that indeed works, unfortunately storing the 
address as ulong in an AA does not seem to be defeating the 
garbage collector!

I guess I'll have to go for a WeakRef implementation, I'll try 
Robert's implementation in signals, but I'd really like to know 
how long it is likely to last as a working weak ref (in terms of 
the D GC changing etc.) and if I'm taking the right approach.
Jan 12 2014
next sibling parent "Abdulhaq" <alynch4047 gmail.com> writes:
On Sunday, 12 January 2014 at 10:48:15 UTC, Abdulhaq wrote:

 No, try this:
 import std.stdio;
 class X {}
 void foo(X x) { writeln(cast(void*) x); }

 void main() {
         X x; // null reference by default.
         writeln(cast(void*) x);
         foo(x);

         x = new X;
         writeln(cast(void*) x);
         foo(x);
 }


 Thanks Tobias that indeed works, unfortunately storing the 
 address as ulong in an AA does not seem to be defeating the 
 garbage collector!

 I guess I'll have to go for a WeakRef implementation, I'll try 
 Robert's implementation in signals, but I'd really like to know 
 how long it is likely to last as a working weak ref (in terms 
 of the D GC changing etc.) and if I'm taking the right approach.


Sorry to reply to my own message, looking at Robert's 
InvisibleAddress class I think he's hiding the address from the 
GC by e.g. rotating the bits in the ulong. He's also made it all 
thread safe - I certainly don't understand the details but it's 
given me some ideas.
Jan 12 2014
prev sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
On Sunday, 12 January 2014 at 10:48:15 UTC, Abdulhaq wrote:

 No, try this:
 import std.stdio;
 class X {}
 void foo(X x) { writeln(cast(void*) x); }

 void main() {
         X x; // null reference by default.
         writeln(cast(void*) x);
         foo(x);

         x = new X;
         writeln(cast(void*) x);
         foo(x);
 }


 Thanks Tobias that indeed works, unfortunately storing the 
 address as ulong in an AA does not seem to be defeating the 
 garbage collector!

 I guess I'll have to go for a WeakRef implementation, I'll try 
 Robert's implementation in signals, but I'd really like to know 
 how long it is likely to last as a working weak ref (in terms 
 of the D GC changing etc.) and if I'm taking the right approach.


The current GC is imprecise. Everything that looks like a pointer 
at the bit level, is treated like one. I can't help you create 
weak references.
Jan 12 2014
parent reply "Abdulhaq" <alynch4047 gmail.com> writes:
On Sunday, 12 January 2014 at 12:12:53 UTC, Tobias Pankrath wrote:

 On Sunday, 12 January 2014 at 10:48:15 UTC, Abdulhaq wrote:

 No, try this:
 import std.stdio;
 class X {}
 void foo(X x) { writeln(cast(void*) x); }

 void main() {
        X x; // null reference by default.
        writeln(cast(void*) x);
        foo(x);

        x = new X;
        writeln(cast(void*) x);
        foo(x);
 }


 Thanks Tobias that indeed works, unfortunately storing the 
 address as ulong in an AA does not seem to be defeating the 
 garbage collector!

 I guess I'll have to go for a WeakRef implementation, I'll try 
 Robert's implementation in signals, but I'd really like to 
 know how long it is likely to last as a working weak ref (in 
 terms of the D GC changing etc.) and if I'm taking the right 
 approach.


 The current GC is imprecise. Everything that looks like a 
 pointer at the bit level, is treated like one. I can't help you 
 create weak references.


No don't worry you've got me past that point of incomprehension, 
thanks again
Jan 12 2014
parent reply "MGW" <mgw yandex.ru> writes:
Maybe this will be useful in the work:

Compile
     Windows: dmd st1.d
       Linux: dmd st1.d -L-ldl
// ---------------------------------------

// MGW 05.01.14
// Model in D a C++ object QByteArray of Qt.
//--------------------------------------------

import core.runtime;     // Load  DLL for Win
import std.stdio;        // writeln

version(linux) {
     import core.sys.posix.dlfcn;  // define dlopen() и dlsym()

     // On Linux DMD v2.063.2, these functions are not defined in 
core.runtime, so I had to write to.
     extern (C) void* rt_loadLibrary(const char* name) { return 
dlopen(name, RTLD_GLOBAL || RTLD_LAZY);  }
     void* GetProcAddress(void* hLib, string nameFun) {  return 
dlsym(hLib, nameFun.ptr);    }
}
version(Windows) {
	import std.c.windows.windows;  // GetProcAddress for Windows
}

// Warning!!!
// When defining constructors and member functions attribute 
"extern (C)" required!
alias extern (C) void function(void*, char*)       
t_QByteArray_QByteArray;  t_QByteArray_QByteArray  
QByteArray_QByteArray;
alias extern (C) void* function(void*, char, int)  
t_QByteArray_fill;        t_QByteArray_fill        
QByteArray_fill;

//T he structure of the QByteArray from the file qbytearray.h in 
the include directory. Because C++ inline functions missing in DLL
// there is no possibility to directly call a dozen functions.
// If you look in C++ there definition is as follows:
// inline char *QByteArray::data() { detach(); return d->data; } 
where d is the Data*
struct Data {
         void* rref;
         int   alloc;
         int   size;
         char* data;      // That's what we need, a pointer to an 
array of bytes
         char  array[1];
}

// == Experimental class DQByteArray ==
class DQByteArray {
     Data* QtObj;       // this is object: &QtObj -  size 4 byte 
(32 os)
     // ------------------
     // constructor D called of a constructor C++
     this(char* buf) {
         QByteArray_QByteArray(&QtObj, buf);
     }
     ~this() {
         // I can find a destructor, and here his record, but too 
lazy to do it ....
     }
     // As inline function is not stored in a DLL have to model it 
through the structure of the Data
     char* data() {
         return (*QtObj).data;
     }
     // D format: Data** == C++ format: QByteArray
     // so it became clear that such a C++object, looking at it 
from the D
     void* fill(char ch, int resize=-1) {
         return QByteArray_fill(&QtObj, ch, resize);
     }
}

int main(string[] args) {

// These files get QByteArray C++
version(linux)   {    auto nameQtCore = "libQtCore.so";  }
version(Windows) {    auto nameQtCore = "QtCore4.dll";   }

     auto h = Runtime.loadLibrary(nameQtCore); // Loading dll or so

     // Load function constructor QByteArray::QByteArray(char*);
     QByteArray_QByteArray = 
cast(t_QByteArray_QByteArray)GetProcAddress(h, 
"_ZN10QByteArrayC1EPKc");
     // QByteArray::fill(char*, int);
     QByteArray_fill = cast(t_QByteArray_fill)GetProcAddress(h, 
"_ZN10QByteArray4fillEci");
     // QByteArray::~QByteArray()

     // Create our experimental subject and consider its data
     DQByteArray ba = new DQByteArray(cast(char*)"ABC".ptr);
     printf("\n ba.data() = %s", ba.data());

     // Experience the action of the fill() and see the result
     ba.fill('Z', 5);
     printf("\n ba.data() = %s", ba.data());

     return 0;
}
Jan 12 2014
parent reply "Abdulhaq" <alynch4047 gmail.com> writes:
On Sunday, 12 January 2014 at 16:17:23 UTC, MGW wrote:

 Maybe this will be useful in the work:

 Compile
     Windows: dmd st1.d
       Linux: dmd st1.d -L-ldl
 // ---------------------------------------

 // MGW 05.01.14
 // Model in D a C++ object QByteArray of Qt.
 //--------------------------------------------

 import core.runtime;     // Load  DLL for Win
 import std.stdio;        // writeln

 version(linux) {
     import core.sys.posix.dlfcn;  // define dlopen() и dlsym()

     // On Linux DMD v2.063.2, these functions are not defined 
 in core.runtime, so I had to write to.
     extern (C) void* rt_loadLibrary(const char* name) { return 
 dlopen(name, RTLD_GLOBAL || RTLD_LAZY);  }
     void* GetProcAddress(void* hLib, string nameFun) {  return 
 dlsym(hLib, nameFun.ptr);    }
 }
 version(Windows) {
 	import std.c.windows.windows;  // GetProcAddress for Windows
 }

 // Warning!!!
 // When defining constructors and member functions attribute 
 "extern (C)" required!
 alias extern (C) void function(void*, char*)       
 t_QByteArray_QByteArray;  t_QByteArray_QByteArray  
 QByteArray_QByteArray;
 alias extern (C) void* function(void*, char, int)  
 t_QByteArray_fill;        t_QByteArray_fill        
 QByteArray_fill;

 //T he structure of the QByteArray from the file qbytearray.h 
 in the include directory. Because C++ inline functions missing 
 in DLL
 // there is no possibility to directly call a dozen functions.
 // If you look in C++ there definition is as follows:
 // inline char *QByteArray::data() { detach(); return d->data; 
 } where d is the Data*
 struct Data {
         void* rref;
         int   alloc;
         int   size;
         char* data;      // That's what we need, a pointer to 
 an array of bytes
         char  array[1];
 }

 // == Experimental class DQByteArray ==
 class DQByteArray {
     Data* QtObj;       // this is object: &QtObj -  size 4 byte 
 (32 os)
     // ------------------
     // constructor D called of a constructor C++
     this(char* buf) {
         QByteArray_QByteArray(&QtObj, buf);
     }
     ~this() {
         // I can find a destructor, and here his record, but 
 too lazy to do it ....
     }
     // As inline function is not stored in a DLL have to model 
 it through the structure of the Data
     char* data() {
         return (*QtObj).data;
     }
     // D format: Data** == C++ format: QByteArray
     // so it became clear that such a C++object, looking at it 
 from the D
     void* fill(char ch, int resize=-1) {
         return QByteArray_fill(&QtObj, ch, resize);
     }
 }

 int main(string[] args) {

 // These files get QByteArray C++
 version(linux)   {    auto nameQtCore = "libQtCore.so";  }
 version(Windows) {    auto nameQtCore = "QtCore4.dll";   }

     auto h = Runtime.loadLibrary(nameQtCore); // Loading dll or 
 so

     // Load function constructor QByteArray::QByteArray(char*);
     QByteArray_QByteArray = 
 cast(t_QByteArray_QByteArray)GetProcAddress(h, 
 "_ZN10QByteArrayC1EPKc");
     // QByteArray::fill(char*, int);
     QByteArray_fill = cast(t_QByteArray_fill)GetProcAddress(h, 
 "_ZN10QByteArray4fillEci");
     // QByteArray::~QByteArray()

     // Create our experimental subject and consider its data
     DQByteArray ba = new DQByteArray(cast(char*)"ABC".ptr);
     printf("\n ba.data() = %s", ba.data());

     // Experience the action of the fill() and see the result
     ba.fill('Z', 5);
     printf("\n ba.data() = %s", ba.data());

     return 0;
 }


Hi yes I think noticed in another thread that you were wrapping 
Qt with dynamic loading of the qt libs, interesting idea - does 
your code allow subclassing of the Qt classes and overriding the 
virtual methods? I'm taking a much more traditional approach, but 
there is method in my madness :-)
Jan 12 2014
parent "MGW" <mgw yandex.ru> writes:
 Hi yes I think noticed in another thread that you were wrapping 
 Qt with dynamic loading of the qt libs, interesting idea - does 
 your code allow subclassing of the Qt classes and overriding 
 the virtual methods? I'm taking a much more traditional 
 approach, but there is method in my madness :-)


I found a way to bypass the restriction associated with virtual 
methods. In a virtual method C + + code I put a call D code.

More examples with wrapping Qt:
https://github.com/MGWL/QtE-Qt_for_Dlang_and_Forth
Jan 13 2014