www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - A question about moving GC

reply Max Samuha <maxter i.com.ua_spamless> writes:
How to ensure D code that interoperates with external libraries will
work when a moving gc is eventually used with D? Will i need to add
lots of pinning/unpinning code?

extern(C) external(Foo* foo); 

struct Foo
{
}

class Bar
{
	Foo foo;
}

main()
{
	Foo* foo = new Foo();
	
	// Should foo be pinned before the call?
	external(foo);

	Bar bar = new Bar();

	// Should bar be pinned before the call?
	external(&bar.foo);
}


btw, ares seems to provides a moving gc and does have a gc.pin method.
Should i consider looking closer at ares? Thanks
Jun 03 2006
next sibling parent Lars Ivar Igesund <larsivar igesund.net> writes:
Max Samuha wrote:


 How to ensure D code that interoperates with external libraries will
 work when a moving gc is eventually used with D? Will i need to add
 lots of pinning/unpinning code?
 
 extern(C) external(Foo* foo);
 
 struct Foo
 {
 }
 
 class Bar
 {
 Foo foo;
 }
 
 main()
 {
 Foo* foo = new Foo();
 
 // Should foo be pinned before the call?
 external(foo);
 
 Bar bar = new Bar();
 
 // Should bar be pinned before the call?
 external(&bar.foo);
 }
 
 
 btw, ares seems to provides a moving gc and does have a gc.pin method.
 Should i consider looking closer at ares? Thanks


The GC in Ares is currently the same as the one in Phobos, but the interface
in Ares should make it possible to easier switch between other types of
GC's.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi
Jun 03 2006
prev sibling next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Typically, garbage collectors run only on memory 
allocation/deallocation/similar.  They won't run in the middle of a 
foreign program... unless it's threaded, then all bets are off.

-[Unknown]



 How to ensure D code that interoperates with external libraries will
 work when a moving gc is eventually used with D? Will i need to add
 lots of pinning/unpinning code?
 
 extern(C) external(Foo* foo); 
 
 struct Foo
 {
 }
 
 class Bar
 {
 	Foo foo;
 }
 
 main()
 {
 	Foo* foo = new Foo();
 	
 	// Should foo be pinned before the call?
 	external(foo);
 
 	Bar bar = new Bar();
 
 	// Should bar be pinned before the call?
 	external(&bar.foo);
 }
 
 
 btw, ares seems to provides a moving gc and does have a gc.pin method.
 Should i consider looking closer at ares? Thanks
Jun 03 2006
parent reply Max Samuha <maxter i.com.ua_spamless> writes:
On Sat, 03 Jun 2006 11:09:56 -0700, "Unknown W. Brackets"
<unknown simplemachines.org> wrote:


Typically, garbage collectors run only on memory 
allocation/deallocation/similar.  They won't run in the middle of a 
foreign program... unless it's threaded, then all bets are off.

-[Unknown]


Thank you for prompt replies. Most programs will run more than one
thread, so it is not guaranteed that garbage collection will not be
triggered when control is in external code. What's the best way to
ensure that the pointers are valid until control is returned to D
code?

version (MovingGC/Ares/whatever)
{
	gc.pin(foo);
}
external(foo);
version (MovingGC/Ares/whatever)
{
	gc.unpin(foo);
}

that's no good. D claims it is compatible with C code but it cannot be
true unless either the specs require that D use only non-moving
garbage collector or there is a standard way to ensure that pointers
to garbage collected memory passed to external functions remain valid
while control is in external code. please correct me if i missed
something. sorry for my poor English
Jun 03 2006
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Well, you are correct that you need to do this for threaded code 
interfacing with C functions via pointers.

That doesn't mean D can't interface with C code properly, just that it 
might not properly do it, necessarily, under this circumstance.

The truth of the matter is, D is still in development, and so is the 
standard library.  This is something that has not yet been provided for, 
you are right.  But, as yet, no D implementation uses a compacting GC.

I would suggest that a compiler that adds a compacting GC in could, 
theoretically, detect the passing of a pointer to a C function, and 
automatically "pin" it before the call.  Unpinning it afterward does not 
seem practical.

Another option would be to declare the variable which holds the pointer 
as extern (C).  This could be a clear indication to the compiler that 
this pointer should not be moved.

However, the above solutions may indeed be impractical.  More thought 
will be needed, and this thought will no doubt come from Walter.  This 
is why, as I said, D is still in an alpha stage.

Even so, I would personally just make a note near the affected area, e.g.:

// !!! Passing a pointer to C.

Or however you prefer to make such notes in your convention; possibly 
noting this specifically as something to check on whenever a new 
compiler is used.

It may very well be that gc.pin() and gc.unpin() functions will be added 
to Phobos, in which case you will need to go through these areas and 
make the changes - just as I had to do with my code when 
on_scope_success was changed to scope (success), and when === was 
changed to is.

It may seem like a pain, but the changes are typically minor, and the 
benefits I see from programming in D vs. C/C++ still far outweigh these 
problems.  I cannot predict if it's the same for you, of course.

I'm American, so I may have to ask you to forgive me my poor English as 
well.  Yours is as good as mine, whether that's a compliment or an insult.

-[Unknown]



 Thank you for prompt replies. Most programs will run more than one
 thread, so it is not guaranteed that garbage collection will not be
 triggered when control is in external code. What's the best way to
 ensure that the pointers are valid until control is returned to D
 code?
 
 version (MovingGC/Ares/whatever)
 {
 	gc.pin(foo);
 }
 external(foo);
 version (MovingGC/Ares/whatever)
 {
 	gc.unpin(foo);
 }
 
 that's no good. D claims it is compatible with C code but it cannot be
 true unless either the specs require that D use only non-moving
 garbage collector or there is a standard way to ensure that pointers
 to garbage collected memory passed to external functions remain valid
 while control is in external code. please correct me if i missed
 something. sorry for my poor English
Jun 03 2006
parent reply Max Samuha <maxter i.com.ua_spamless> writes:
On Sat, 03 Jun 2006 17:06:03 -0700, "Unknown W. Brackets"
<unknown simplemachines.org> wrote:


Well, you are correct that you need to do this for threaded code 
interfacing with C functions via pointers.

That doesn't mean D can't interface with C code properly, just that it 
might not properly do it, necessarily, under this circumstance.

The truth of the matter is, D is still in development, and so is the 
standard library.  This is something that has not yet been provided for, 
you are right.  But, as yet, no D implementation uses a compacting GC.

I would suggest that a compiler that adds a compacting GC in could, 
theoretically, detect the passing of a pointer to a C function, and 
automatically "pin" it before the call.  Unpinning it afterward does not 
seem practical.

Another option would be to declare the variable which holds the pointer 
as extern (C).  This could be a clear indication to the compiler that 
this pointer should not be moved.

However, the above solutions may indeed be impractical.  More thought 
will be needed, and this thought will no doubt come from Walter.  This 
is why, as I said, D is still in an alpha stage.

Even so, I would personally just make a note near the affected area, e.g.:

// !!! Passing a pointer to C.

Or however you prefer to make such notes in your convention; possibly 
noting this specifically as something to check on whenever a new 
compiler is used.

It may very well be that gc.pin() and gc.unpin() functions will be added 
to Phobos, in which case you will need to go through these areas and 
make the changes - just as I had to do with my code when 
on_scope_success was changed to scope (success), and when === was 
changed to is.

It may seem like a pain, but the changes are typically minor, and the 
benefits I see from programming in D vs. C/C++ still far outweigh these 
problems.  I cannot predict if it's the same for you, of course.

I'm American, so I may have to ask you to forgive me my poor English as 
well.  Yours is as good as mine, whether that's a compliment or an insult.

-[Unknown]


Thank you for the extensive explanation!
Marking my code with '// !!! Passing a pointer to C.' is what i've
been doing recently:) and it seems the best solution for now until
ares is mature or phobos has pin/unpin, or pointers passed to C are
pinned implicitly.
Jun 04 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Max Samuha wrote:

 Thank you for the extensive explanation!
 Marking my code with '// !!! Passing a pointer to C.' is what i've
 been doing recently:) and it seems the best solution for now until
 ares is mature or phobos has pin/unpin, or pointers passed to C are
 pinned implicitly.


They will be pinned implicitly, no need to call pin/unpin. The only time 
you're going to have a problem is if the C routine hides the pointer in 
a malloc'd buffer where the gc won't see it. The gc *will* see pointers 
on the C stack.
Jun 07 2006
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Walter Bright wrote:

 Max Samuha wrote:

 Thank you for the extensive explanation!
 Marking my code with '// !!! Passing a pointer to C.' is what i've
 been doing recently:) and it seems the best solution for now until
 ares is mature or phobos has pin/unpin, or pointers passed to C are
 pinned implicitly.


 
 They will be pinned implicitly, no need to call pin/unpin. The only time 
 you're going to have a problem is if the C routine hides the pointer in 
 a malloc'd buffer where the gc won't see it. The gc *will* see pointers 
 on the C stack.


Would it remain pinned indefinitely, until the C function exits, or 
until the program explicitly unpins it by some means?

Stewart.
Jun 07 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Stewart Gordon wrote:

 Walter Bright wrote:

 Max Samuha wrote:

 Thank you for the extensive explanation!
 Marking my code with '// !!! Passing a pointer to C.' is what i've
 been doing recently:) and it seems the best solution for now until
 ares is mature or phobos has pin/unpin, or pointers passed to C are
 pinned implicitly.


 They will be pinned implicitly, no need to call pin/unpin. The only 
 time you're going to have a problem is if the C routine hides the 
 pointer in a malloc'd buffer where the gc won't see it. The gc *will* 
 see pointers on the C stack.


 
 Would it remain pinned indefinitely, until the C function exits, or 
 until the program explicitly unpins it by some means?


It's pinned because it's on the stack, just like pointers on the stack 
for your D functions. The gc doesn't know the difference.
Jun 07 2006
next sibling parent reply Unknown W. Brackets <Unknown_member pathlink.com> writes:
Of course, how foolish of me - so there's no worry about the garbage collector
killing that in a threaded program either.  Duh.

-[Unknown]

In article <e679ot$7e3$1 digitaldaemon.com>, Walter Bright says...

Stewart Gordon wrote: 

 Would it remain pinned indefinitely, until the C function exits, or 
 until the program explicitly unpins it by some means?


It's pinned because it's on the stack, just like pointers on the stack 
for your D functions. The gc doesn't know the difference.
Jun 07 2006
parent Max Samuha <maxter i.com.ua_spamless> writes:
On Wed, 7 Jun 2006 22:11:52 +0000 (UTC), Unknown W. Brackets
<Unknown_member pathlink.com> wrote:


Of course, how foolish of me - so there's no worry about the garbage collector
killing that in a threaded program either.  Duh.

-[Unknown]

In article <e679ot$7e3$1 digitaldaemon.com>, Walter Bright says...

Stewart Gordon wrote: 

 Would it remain pinned indefinitely, until the C function exits, or 
 until the program explicitly unpins it by some means?


It's pinned because it's on the stack, just like pointers on the stack 
for your D functions. The gc doesn't know the difference.




Thanks. It seems I can live with that for now. i can't help saying D
is very addictive. imho, the best C-like language i've had a chance to
try.
Jun 08 2006
prev sibling next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Walter Bright wrote:

 Stewart Gordon wrote:


<snip>

 Would it remain pinned indefinitely, until the C function exits, or 
 until the program explicitly unpins it by some means?


 
 It's pinned because it's on the stack, just like pointers on the stack 
 for your D functions. The gc doesn't know the difference.


But what if the foreign code needs to hold onto the pointer for longer 
than it can remain on the stack?

Stewart.
Jun 08 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Stewart Gordon wrote:

 Walter Bright wrote:

 Stewart Gordon wrote:


 <snip>

 Would it remain pinned indefinitely, until the C function exits, or 
 until the program explicitly unpins it by some means?


 It's pinned because it's on the stack, just like pointers on the stack 
 for your D functions. The gc doesn't know the difference.


 
 But what if the foreign code needs to hold onto the pointer for longer 
 than it can remain on the stack?


Have the D caller code keep a copy, or use addRoots to tell the gc where 
the foreign code is holding it.
Jun 08 2006
next sibling parent reply Unknown W. Brackets <Unknown_member pathlink.com> writes:
But the question is for a "moving" GC, which might not consider a local
reference to mean "don't move it"...?

-[Unknown]

In article <e6abmn$1iaj$1 digitaldaemon.com>, Walter Bright says...

Stewart Gordon wrote:

 But what if the foreign code needs to hold onto the pointer for longer 
 than it can remain on the stack?


Have the D caller code keep a copy, or use addRoots to tell the gc where 
the foreign code is holding it.
Jun 08 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Unknown W. Brackets wrote:

 But the question is for a "moving" GC, which might not consider a local
 reference to mean "don't move it"...?


addroots will work with that.
Jun 09 2006
parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
But what if I don't know where the foreign code is holding it?  The C 
library might not expose this, and then I'm out of luck... unless I can 
modify it, which I probably can't.

But, this is all theoretical anyway.  I can't even think of a library 
that keeps a pointer I give it.

-[Unknown]



 Unknown W. Brackets wrote:

 But the question is for a "moving" GC, which might not consider a local
 reference to mean "don't move it"...?


 
 addroots will work with that.
Jun 09 2006
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Walter Bright wrote:

 Stewart Gordon wrote:


<snip>

 But what if the foreign code needs to hold onto the pointer for longer 
 than it can remain on the stack?


 
 Have the D caller code keep a copy, or use addRoots


What is addRoot_s_?


 to tell the gc where the foreign code is holding it.


How will I know where the foreign code is holding the pointer?  Surely 
it makes more sense to add the pointer itself as a root.

Moreover, the pinning methods'll need to be documented.  So at the 
moment, I'm guessing that the following will implicitly pin a 
GC-allocated block:

- storing a pointer to it in a union
- having a pointer to it on the stack
- adding it as a root

Stewart.
Jun 09 2006
parent Walter Bright <newshound digitalmars.com> writes:
Stewart Gordon wrote:

 Walter Bright wrote:

 Stewart Gordon wrote:


 <snip>

 But what if the foreign code needs to hold onto the pointer for 
 longer than it can remain on the stack?


 Have the D caller code keep a copy, or use addRoots


 
 What is addRoot_s_?


http://www.digitalmars.com/d/phobos/std_gc.html



 to tell the gc where the foreign code is holding it.


 How will I know where the foreign code is holding the pointer?


First of all, the foreign code documentation ought to make some mention 
of what it does with the pointer - after all, it must even if you 
malloc'd it so you know when and if you can ever free it. Have the D 
code caller call addRoot.
Jun 09 2006
prev sibling parent reply Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Walter Bright wrote:

 Stewart Gordon wrote:

 Walter Bright wrote:

 Max Samuha wrote:

 Thank you for the extensive explanation!
 Marking my code with '// !!! Passing a pointer to C.' is what i've
 been doing recently:) and it seems the best solution for now until
 ares is mature or phobos has pin/unpin, or pointers passed to C are
 pinned implicitly.


 They will be pinned implicitly, no need to call pin/unpin. The only 
 time you're going to have a problem is if the C routine hides the 
 pointer in a malloc'd buffer where the gc won't see it. The gc *will* 
 see pointers on the C stack.


 Would it remain pinned indefinitely, until the C function exits, or 
 until the program explicitly unpins it by some means?


 
 It's pinned because it's on the stack, just like pointers on the stack 
 for your D functions. The gc doesn't know the difference.


Whoa, I got a bit confused here. Is any reference that is held in the 
stack automatically pinned (i.e., the referred data doesn't move) ? If 
so, why is that, why wouldn't a moving collector move it?

-- 
Bruno Medeiros - CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jun 09 2006
parent reply Walter Bright <newshound digitalmars.com> writes:
Bruno Medeiros wrote:

 Whoa, I got a bit confused here. Is any reference that is held in the 
 stack automatically pinned (i.e., the referred data doesn't move) ?


Yes.


 If so, why is that, why wouldn't a moving collector move it?


Because then you'd have to track all the register contents instruction 
by instruction.
Jun 09 2006
parent reply Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Walter Bright wrote:

 Bruno Medeiros wrote:

 Whoa, I got a bit confused here. Is any reference that is held in the 
 stack automatically pinned (i.e., the referred data doesn't move) ?


 
 Yes.
 

 If so, why is that, why wouldn't a moving collector move it?


 
 Because then you'd have to track all the register contents instruction 
 by instruction.


:/ I'm lost, I didn't understand that at all. "track all the register 
contents instruction by instruction" ?

-- 
Bruno Medeiros - CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jun 12 2006
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Bruno Medeiros wrote:

 Walter Bright wrote:

 Bruno Medeiros wrote:

 Whoa, I got a bit confused here. Is any reference that is held in the
 stack automatically pinned (i.e., the referred data doesn't move) ?


 Yes.


 If so, why is that, why wouldn't a moving collector move it?


 Because then you'd have to track all the register contents instruction
 by instruction.


 
 :/ I'm lost, I didn't understand that at all. "track all the register
 contents instruction by instruction" ?


DISCLAIMER: /me is not an x86 assembler expert.  Take with the requisite
grain of salt.

Think about it like this: say you want to dereference a pointer.  Let's
assume you've got something like this:




Now, you move this into the ESI pointer to dereference it.



And then HOLD IT!  Moving garbage collector coming through!  Hmm, this
"foo" thing could be moved to be more efficient; let's move it!  Now, go
back to your business.



What happen?  Someone set up us the access violation!  That damned
moving GC preempted us and moved our data before we could get to it!

Even if you're not multi-threaded, this is still a problem: there's
nothing to preclude a pointer being stored in a register, calling new,
causing a moving collect, and then attempting to dereference it.

Isn't memory management fun? :P

	-- Daniel

-- 
Unlike Knuth, I have neither proven or tried the above; it may not even
make sense.

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D
i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/
Jun 12 2006
parent Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Daniel Keep wrote:

 
 Bruno Medeiros wrote:

 Walter Bright wrote:

 Bruno Medeiros wrote:

 Whoa, I got a bit confused here. Is any reference that is held in the
 stack automatically pinned (i.e., the referred data doesn't move) ?


 Yes.


 If so, why is that, why wouldn't a moving collector move it?


 Because then you'd have to track all the register contents instruction
 by instruction.


 :/ I'm lost, I didn't understand that at all. "track all the register
 contents instruction by instruction" ?


 
 DISCLAIMER: /me is not an x86 assembler expert.  Take with the requisite
 grain of salt.
 
 Think about it like this: say you want to dereference a pointer.  Let's
 assume you've got something like this:
 


 
 Now, you move this into the ESI pointer to dereference it.
 

 
 And then HOLD IT!  Moving garbage collector coming through!  Hmm, this
 "foo" thing could be moved to be more efficient; let's move it!  Now, go
 back to your business.
 

 
 What happen?  Someone set up us the access violation!  That damned
 moving GC preempted us and moved our data before we could get to it!
 


The very same thing can happen with references stored in the heap or the 
static segment (i.e. globals). That is, with *any* reference.


-- 
Bruno Medeiros - CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jun 12 2006
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Max Samuha wrote:

 How to ensure D code that interoperates with external libraries will
 work when a moving gc is eventually used with D? Will i need to add
 lots of pinning/unpinning code?


<snip>

Pinning has been requested a few times on these 'groups.  There has also 
been a discussion before on issues that still need to be addressed 
before a GC that moves stuff around will work:

http://www.digitalmars.com/d/archives/26273.html

Stewart.
Jun 06 2006
parent Max Samuha <maxter i.com.ua_spamless> writes:
On Tue, 06 Jun 2006 14:27:22 +0100, Stewart Gordon
<smjg_1998 yahoo.com> wrote:


Pinning has been requested a few times on these 'groups.  There has also 
been a discussion before on issues that still need to be addressed 
before a GC that moves stuff around will work:

http://www.digitalmars.com/d/archives/26273.html

Stewart.


Yes, the thread you mention was posted two years ago. It would be
really nice to hear a word from Walter on this issue.
Jun 06 2006