www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Opaque handles...

reply Manu <turkeyman gmail.com> writes:
I've seen some discussions before about people dealing with opaque
pointers, or handle types in D.
It's a common C practise, and there seems to be uncertainty about the best
way to implement this on the D side of the fence.

So I'm looking for opinions... hit me?

I have this in C:

  struct Thing;
  Thing* MakeThing();

And in D:

  struct Thing;
  extern (C) Thing* MakeThing();

The question is, does this suck?
D currently can't allocate an array of Thing*'s for some weird reason.

Are there any concerns relating to the GC that I should be aware of when
handling pointers returned to me from C code?


So a further goal, and maybe I'll add a little back-story; my 30th birthday
game jam from a few weeks back (which is still an active thread!) was
interfacing with my engine using basically an extern(C) API precisely
mirroring the C headers.
I'm planning on joining another game jam in a couple of week, using D again
(maybe I can start another thread as long and controversial as the last
one!) ;)
But this time, I want to spend some preparation time building a much more
D-styled API to interact with. Again, I'm struggling with just how I should
wrap the C calls and opaque types up.

What do people feel is conventional, and has proven to work well?

Again, wrt the opaque pointer problem. I don't really feel using an opaque
pointer on the D side is really right.
What I really want to use is something that feels more like a class
reference, and I toyed with the idea of abusing class references to hold my
handles, and methods would internally case 'this' to my handle type before
passing it back into the C code it wraps up.
This seems okay at first, but then you realise there are serious problems
with new (wants to return new memory, I can't hook the new operator?), and
there's no way to free it automatically, since D is garbage collected
rather than ref counted >_<
(Note: I'm still fairly certain that I want a ref-counting GC in D)

So... should I make it a struct with a single void* member?

struct MyThing { void* handle; }

I kinda hate this. the struct keyword totally gives the user the wrong
impression, and people have a natural instinct to avoid passing structs
around by value, which is exactly how these objects should be handled...

So if I can't present it as a class, and a struct is a pretty shit
solution... what?

My objects are opaque handles to engine-managed resources. They are
ref-counted, and there is extern(C) API to add/release references to stuff,
which I would rather have automated in the D code...

One final problem with ref-counting in D. Since it's not a 1st class
feature and requires you to store a ref-count in your object, you can't
make your objects immutable anymore because the ref-count needs to be
bumped about as the object moves around. Anyone approached this problem?
Sep 18 2013
next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
 One final problem with ref-counting in D. Since it's not a 1st 
 class
 feature and requires you to store a ref-count in your object, 
 you can't
 make your objects immutable anymore because the ref-count needs 
 to be
 bumped about as the object moves around. Anyone approached this 
 problem?


I use a shared_ptr (especially if I deal with C memory).
Sep 18 2013
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:

 This seems okay at first, but then you realise there are 
 serious problems with new (wants to return new memory, I can't 
 hook the new operator?),


There is an override for new, but it is deprecated in d2.
http://dlang.org/memory.html#newdelete


 I kinda hate this. the struct keyword totally gives the user 
 the wrong impression, and people have a natural instinct to 
 avoid passing structs around by value


...just don't do that? You don't write "ref Class obj" in D, so 
just pretend the struct is a class and do it the same way.

The only time you'd even need to know it is a struct is if you 
look at the source, and there you can write

struct /* but pretend it is a class! */ whatever {

}



Here's how I might do it. Given this simple C test header:

struct Test;

struct Test* makeTest(int num);
void addTestRef(struct Test* t);
void killTest(struct Test* t); // i should have called that 
releaseref lol
int getNumber(struct Test* t);
void setNumber(struct Test* t, int n);



We might use it in D like this:


// this represents the struct Test in C, our opaque pointer
// (the disabled stuff is to workaround a minor dmd bug)
struct c_Test {
	 disable this();
	 disable this(this);
}

// and this is our D wrapper
// always pass this by value
struct Test {
        // the internal reference
	private c_Test* c_ptr;

	// construction can only be done publicly through the static 
make method
	 disable this();
	private this(c_Test* ptr) {
		c_ptr = ptr;
	}

	public static Test make(int num) {
		return Test(makeTest(num));
	}

	// the alias this lets us call the rest of the binded C 
functions with UFCS without manually writing wrappers
	c_Test* getCPointer() { return c_ptr; } // it does a property so 
you can't assign to the internal pointer
	// you CAN break the refcounting with this, but since you have 
to specifically ask for the uglier c_Test* to trigger
	// this, it is unlikely to happen by accident.
	alias getCPointer this;

	// refcount stuff
	~this() {
		c_ptr.killTest();
	}
	this(this) {
		c_ptr.addTestRef();
	}
}

// you might notice the above is pretty generic, perhaps it could 
all be a single template so you don't rewrite it for too many 
types.

// and the C function prototypes
extern(C) {
	c_Test* makeTest(int num);
	void addTestRef(c_Test* t);
	void killTest(c_Test* t);

	int getNumber(c_Test* t);
	void setNumber(c_Test* t, int n);
}

// test program
import core.stdc.stdio;
void foo(Test t) {
	printf("foo on %d\n", t.getNumber());
	t.setNumber(20);
}

void main() {
	auto t = Test.make(12);
	auto t2 = Test.make(24);

	printf("about to call foo\n");
	foo(t);
	printf("done calling foo\n");

	printf("main with t  == %d\n", t.getNumber());
	printf("main with t2 == %d\n", t2.getNumber());
}




You don't have my c implementation but it isn't special and 
running the program produced the following output:

Test made with number 12
Test made with number 24
about to call foo

the function call
foo on 12

ending
done calling foo

by reference
main with t2 == 24

the ref automatically

dtor refcount now 0 on Test 20
Test killed with number 20




Using UFCS for the C functions might be a little bit weird, but 
saves tediously writing them all out again to forward inside the 
struct.
Sep 18 2013
parent Manu <turkeyman gmail.com> writes:
On 19 September 2013 02:57, Adam D. Ruppe <destructionator gmail.com> wrote:


 On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:


 This seems okay at first, but then you realise there are serious problems
 with new (wants to return new memory, I can't hook the new operator?),


 There is an override for new, but it is deprecated in d2.



  I kinda hate this. the struct keyword totally gives the user the wrong

 impression, and people have a natural instinct to avoid passing structs
 around by value


 ...just don't do that? You don't write "ref Class obj" in D, so just
 pretend the struct is a class and do it the same way.

 The only time you'd even need to know it is a struct is if you look at the
 source, and there you can write

 struct /* but pretend it is a class! */ whatever {
 }


Except those times when you read "struct Thing" in the documentation, or
hover over it in the IDE and it says "struct Thing", or when the syntax
hilighting makes it go the struct colour.
It's still a struct, it's obviously a struct, and that communicates some
amount of intent.
What you describe is a hack, and I don't want to build the foundation of a
performance-oriented library upon a hack. If that's as good as it gets,
then there's a deficiency in D here that we need to think about.

I also don't have faith in many compilers passing
struct's-with-only-a-single-member to functions by that member's type
rather than as a struct.

Here's how I might do it. Given this simple C test header:

 struct Test;

 struct Test* makeTest(int num);
 void addTestRef(struct Test* t);
 void killTest(struct Test* t); // i should have called that releaseref lol
 int getNumber(struct Test* t);
 void setNumber(struct Test* t, int n);



 We might use it in D like this:


 // this represents the struct Test in C, our opaque pointer
 // (the disabled stuff is to workaround a minor dmd bug)
 struct c_Test {
          disable this();
          disable this(this);
 }

 // and this is our D wrapper
 // always pass this by value
 struct Test {
        // the internal reference
         private c_Test* c_ptr;

         // construction can only be done publicly through the static make
 method
          disable this();
         private this(c_Test* ptr) {
                 c_ptr = ptr;
         }

         public static Test make(int num) {
                 return Test(makeTest(num));
         }

         // the alias this lets us call the rest of the binded C functions
 with UFCS without manually writing wrappers
         c_Test* getCPointer() { return c_ptr; } // it does a property so
 you can't assign to the internal pointer
         // you CAN break the refcounting with this, but since you have to
 specifically ask for the uglier c_Test* to trigger
         // this, it is unlikely to happen by accident.
         alias getCPointer this;

         // refcount stuff
         ~this() {
                 c_ptr.killTest();
         }
         this(this) {
                 c_ptr.addTestRef();
         }
 }

 // you might notice the above is pretty generic, perhaps it could all be a
 single template so you don't rewrite it for too many types.

 // and the C function prototypes
 extern(C) {
         c_Test* makeTest(int num);
         void addTestRef(c_Test* t);
         void killTest(c_Test* t);

         int getNumber(c_Test* t);
         void setNumber(c_Test* t, int n);
 }

 // test program
 import core.stdc.stdio;
 void foo(Test t) {
         printf("foo on %d\n", t.getNumber());
         t.setNumber(20);
 }

 void main() {
         auto t = Test.make(12);
         auto t2 = Test.make(24);

         printf("about to call foo\n");
         foo(t);
         printf("done calling foo\n");

         printf("main with t  == %d\n", t.getNumber());
         printf("main with t2 == %d\n", t2.getNumber());
 }




 You don't have my c implementation but it isn't special and running the
 program produced the following output:

 Test made with number 12
 Test made with number 24
 about to call foo

 function call
 foo on 12

 done calling foo

 reference
 main with t2 == 24

 automatically

 dtor refcount now 0 on Test 20
 Test killed with number 20




 Using UFCS for the C functions might be a little bit weird, but saves
 tediously writing them all out again to forward inside the struct.


Well, this is precisely what I've tried before, but I don't like it. It
feels like a hack.
Can you make me some promises about the ABI for passing that struct by
value across every compiler+architecture?

I don't like that it's a struct, it's NOT a struct, it's a handle.

There was some discussion about windows HANDLE types recently. I can't
remember where I saw it, but I felt like that was quite relevant.
It even dealt with the problem of HWND implicitly casting to a HANDLE, and
I also have that problem here.
Sep 18 2013
prev sibling next sibling parent "Tavi Cacina" <octavian.cacina outlook.com> writes:
On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:

 One final problem with ref-counting in D. Since it's not a 1st 
 class
 feature and requires you to store a ref-count in your object, 
 you can't
 make your objects immutable anymore because the ref-count needs 
 to be
 bumped about as the object moves around. Anyone approached this 
 problem?


I'de like to here some guidelines regarding this too. I tried 
once to manage some resources with ref-counted structs. The 
problem was that I could not hold them in classes anymore, so for 
deterministic life-management I had to use just structs, 
everywhere...
Sep 18 2013
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:

 struct MyThing { void* handle; }

 I kinda hate this. the struct keyword totally gives the user 
 the wrong
 impression, and people have a natural instinct to avoid passing 
 structs
 around by value, which is exactly how these objects should be 
 handled...


You've got some wrong people, lol. How do they work with smart 
pointers?
Sep 19 2013
parent reply "growler" <growlercab gmail.com> writes:
On Thursday, 19 September 2013 at 09:17:09 UTC, Kagamin wrote:

 On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:

 struct MyThing { void* handle; }

 I kinda hate this. the struct keyword totally gives the user 
 the wrong
 impression, and people have a natural instinct to avoid 
 passing structs
 around by value, which is exactly how these objects should be 
 handled...


 You've got some wrong people, lol. How do they work with smart 
 pointers?


I had a similar problem not long ago and ended up using RefCount 
from std.typecons. It worked well enough for my needs, maybe it 
would be an option here.

I'm not sure of the performance though for a game situation, 
you'd have to test it.
Sep 19 2013
parent "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 19 September 2013 at 09:53:28 UTC, growler wrote:

 On Thursday, 19 September 2013 at 09:17:09 UTC, Kagamin wrote:

 On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:

 struct MyThing { void* handle; }

 I kinda hate this. the struct keyword totally gives the user 
 the wrong
 impression, and people have a natural instinct to avoid 
 passing structs
 around by value, which is exactly how these objects should be 
 handled...


 You've got some wrong people, lol. How do they work with smart 
 pointers?


 I had a similar problem not long ago and ended up using 
 RefCount from std.typecons. It worked well enough for my needs, 
 maybe it would be an option here.

 I'm not sure of the performance though for a game situation, 
 you'd have to test it.


Yes, that should be enough. But I, for my part, do not trust the 
phobos solution completely. Therefore I use my own.
Sep 19 2013