digitalmars.D.learn - A question about moving GC
- Max Samuha (23/23) Jun 03 2006 How to ensure D code that interoperates with external libraries will
- Lars Ivar Igesund (8/39) Jun 03 2006 The GC in Ares is currently the same as the one in Phobos, but the inter...
- Unknown W. Brackets (4/35) Jun 03 2006 Typically, garbage collectors run only on memory
- Max Samuha (23/27) Jun 03 2006 Thank you for prompt replies. Most programs will run more than one
- Unknown W. Brackets (33/55) Jun 03 2006 Well, you are correct that you need to do this for threaded code
- Max Samuha (7/40) Jun 04 2006 Thank you for the extensive explanation!
- Walter Bright (5/10) Jun 07 2006 They will be pinned implicitly, no need to call pin/unpin. The only time...
- Stewart Gordon (4/15) Jun 07 2006 Would it remain pinned indefinitely, until the C function exits, or
- Walter Bright (3/18) Jun 07 2006 It's pinned because it's on the stack, just like pointers on the stack
- Unknown W. Brackets (4/9) Jun 07 2006 Of course, how foolish of me - so there's no worry about the garbage col...
- Max Samuha (5/15) Jun 08 2006 Thanks. It seems I can live with that for now. i can't help saying D
- Stewart Gordon (5/11) Jun 08 2006 But what if the foreign code needs to hold onto the pointer for longer
- Walter Bright (3/14) Jun 08 2006 Have the D caller code keep a copy, or use addRoots to tell the gc where...
- Unknown W. Brackets (4/9) Jun 08 2006 But the question is for a "moving" GC, which might not consider a local
- Walter Bright (2/4) Jun 09 2006 addroots will work with that.
- Unknown W. Brackets (6/11) Jun 09 2006 But what if I don't know where the foreign code is holding it? The C
- Stewart Gordon (12/18) Jun 09 2006 What is addRoot_s_?
- Walter Bright (6/17) Jun 09 2006 First of all, the foreign code documentation ought to make some mention
- Bruno Medeiros (7/26) Jun 09 2006 Whoa, I got a bit confused here. Is any reference that is held in the
- Walter Bright (4/7) Jun 09 2006 Because then you'd have to track all the register contents instruction
- Bruno Medeiros (6/16) Jun 12 2006 :/ I'm lost, I didn't understand that at all. "track all the register
- Daniel Keep (25/39) Jun 12 2006 DISCLAIMER: /me is not an x86 assembler expert. Take with the requisite
- Bruno Medeiros (6/42) Jun 12 2006 The very same thing can happen with references stored in the heap or the...
-
Stewart Gordon
(7/10)
Jun 06 2006
- Max Samuha (4/9) Jun 06 2006 Yes, the thread you mention was posted two years ago. It would be
How to ensure D code that interoperates with external libraries will work when a moving gc is eventually used with D? Will i need to add lots of pinning/unpinning code? extern(C) external(Foo* foo); struct Foo { } class Bar { Foo foo; } main() { Foo* foo = new Foo(); // Should foo be pinned before the call? external(foo); Bar bar = new Bar(); // Should bar be pinned before the call? external(&bar.foo); } btw, ares seems to provides a moving gc and does have a gc.pin method. Should i consider looking closer at ares? Thanks
Jun 03 2006
Max Samuha wrote:How to ensure D code that interoperates with external libraries will work when a moving gc is eventually used with D? Will i need to add lots of pinning/unpinning code? extern(C) external(Foo* foo); struct Foo { } class Bar { Foo foo; } main() { Foo* foo = new Foo(); // Should foo be pinned before the call? external(foo); Bar bar = new Bar(); // Should bar be pinned before the call? external(&bar.foo); } btw, ares seems to provides a moving gc and does have a gc.pin method. Should i consider looking closer at ares? ThanksThe GC in Ares is currently the same as the one in Phobos, but the interface in Ares should make it possible to easier switch between other types of GC's. -- Lars Ivar Igesund blog at http://larsivi.net DSource & #D: larsivi
Jun 03 2006
Typically, garbage collectors run only on memory allocation/deallocation/similar. They won't run in the middle of a foreign program... unless it's threaded, then all bets are off. -[Unknown]How to ensure D code that interoperates with external libraries will work when a moving gc is eventually used with D? Will i need to add lots of pinning/unpinning code? extern(C) external(Foo* foo); struct Foo { } class Bar { Foo foo; } main() { Foo* foo = new Foo(); // Should foo be pinned before the call? external(foo); Bar bar = new Bar(); // Should bar be pinned before the call? external(&bar.foo); } btw, ares seems to provides a moving gc and does have a gc.pin method. Should i consider looking closer at ares? Thanks
Jun 03 2006
On Sat, 03 Jun 2006 11:09:56 -0700, "Unknown W. Brackets" <unknown simplemachines.org> wrote:Typically, garbage collectors run only on memory allocation/deallocation/similar. They won't run in the middle of a foreign program... unless it's threaded, then all bets are off. -[Unknown]Thank you for prompt replies. Most programs will run more than one thread, so it is not guaranteed that garbage collection will not be triggered when control is in external code. What's the best way to ensure that the pointers are valid until control is returned to D code? version (MovingGC/Ares/whatever) { gc.pin(foo); } external(foo); version (MovingGC/Ares/whatever) { gc.unpin(foo); } that's no good. D claims it is compatible with C code but it cannot be true unless either the specs require that D use only non-moving garbage collector or there is a standard way to ensure that pointers to garbage collected memory passed to external functions remain valid while control is in external code. please correct me if i missed something. sorry for my poor English
Jun 03 2006
Well, you are correct that you need to do this for threaded code interfacing with C functions via pointers. That doesn't mean D can't interface with C code properly, just that it might not properly do it, necessarily, under this circumstance. The truth of the matter is, D is still in development, and so is the standard library. This is something that has not yet been provided for, you are right. But, as yet, no D implementation uses a compacting GC. I would suggest that a compiler that adds a compacting GC in could, theoretically, detect the passing of a pointer to a C function, and automatically "pin" it before the call. Unpinning it afterward does not seem practical. Another option would be to declare the variable which holds the pointer as extern (C). This could be a clear indication to the compiler that this pointer should not be moved. However, the above solutions may indeed be impractical. More thought will be needed, and this thought will no doubt come from Walter. This is why, as I said, D is still in an alpha stage. Even so, I would personally just make a note near the affected area, e.g.: // !!! Passing a pointer to C. Or however you prefer to make such notes in your convention; possibly noting this specifically as something to check on whenever a new compiler is used. It may very well be that gc.pin() and gc.unpin() functions will be added to Phobos, in which case you will need to go through these areas and make the changes - just as I had to do with my code when on_scope_success was changed to scope (success), and when === was changed to is. It may seem like a pain, but the changes are typically minor, and the benefits I see from programming in D vs. C/C++ still far outweigh these problems. I cannot predict if it's the same for you, of course. I'm American, so I may have to ask you to forgive me my poor English as well. Yours is as good as mine, whether that's a compliment or an insult. -[Unknown]Thank you for prompt replies. Most programs will run more than one thread, so it is not guaranteed that garbage collection will not be triggered when control is in external code. What's the best way to ensure that the pointers are valid until control is returned to D code? version (MovingGC/Ares/whatever) { gc.pin(foo); } external(foo); version (MovingGC/Ares/whatever) { gc.unpin(foo); } that's no good. D claims it is compatible with C code but it cannot be true unless either the specs require that D use only non-moving garbage collector or there is a standard way to ensure that pointers to garbage collected memory passed to external functions remain valid while control is in external code. please correct me if i missed something. sorry for my poor English
Jun 03 2006
On Sat, 03 Jun 2006 17:06:03 -0700, "Unknown W. Brackets" <unknown simplemachines.org> wrote:Well, you are correct that you need to do this for threaded code interfacing with C functions via pointers. That doesn't mean D can't interface with C code properly, just that it might not properly do it, necessarily, under this circumstance. The truth of the matter is, D is still in development, and so is the standard library. This is something that has not yet been provided for, you are right. But, as yet, no D implementation uses a compacting GC. I would suggest that a compiler that adds a compacting GC in could, theoretically, detect the passing of a pointer to a C function, and automatically "pin" it before the call. Unpinning it afterward does not seem practical. Another option would be to declare the variable which holds the pointer as extern (C). This could be a clear indication to the compiler that this pointer should not be moved. However, the above solutions may indeed be impractical. More thought will be needed, and this thought will no doubt come from Walter. This is why, as I said, D is still in an alpha stage. Even so, I would personally just make a note near the affected area, e.g.: // !!! Passing a pointer to C. Or however you prefer to make such notes in your convention; possibly noting this specifically as something to check on whenever a new compiler is used. It may very well be that gc.pin() and gc.unpin() functions will be added to Phobos, in which case you will need to go through these areas and make the changes - just as I had to do with my code when on_scope_success was changed to scope (success), and when === was changed to is. It may seem like a pain, but the changes are typically minor, and the benefits I see from programming in D vs. C/C++ still far outweigh these problems. I cannot predict if it's the same for you, of course. I'm American, so I may have to ask you to forgive me my poor English as well. Yours is as good as mine, whether that's a compliment or an insult. -[Unknown]Thank you for the extensive explanation! Marking my code with '// !!! Passing a pointer to C.' is what i've been doing recently:) and it seems the best solution for now until ares is mature or phobos has pin/unpin, or pointers passed to C are pinned implicitly.
Jun 04 2006
Max Samuha wrote:Thank you for the extensive explanation! Marking my code with '// !!! Passing a pointer to C.' is what i've been doing recently:) and it seems the best solution for now until ares is mature or phobos has pin/unpin, or pointers passed to C are pinned implicitly.They will be pinned implicitly, no need to call pin/unpin. The only time you're going to have a problem is if the C routine hides the pointer in a malloc'd buffer where the gc won't see it. The gc *will* see pointers on the C stack.
Jun 07 2006
Walter Bright wrote:Max Samuha wrote:Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means? Stewart.Thank you for the extensive explanation! Marking my code with '// !!! Passing a pointer to C.' is what i've been doing recently:) and it seems the best solution for now until ares is mature or phobos has pin/unpin, or pointers passed to C are pinned implicitly.They will be pinned implicitly, no need to call pin/unpin. The only time you're going to have a problem is if the C routine hides the pointer in a malloc'd buffer where the gc won't see it. The gc *will* see pointers on the C stack.
Jun 07 2006
Stewart Gordon wrote:Walter Bright wrote:It's pinned because it's on the stack, just like pointers on the stack for your D functions. The gc doesn't know the difference.Max Samuha wrote:Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means?Thank you for the extensive explanation! Marking my code with '// !!! Passing a pointer to C.' is what i've been doing recently:) and it seems the best solution for now until ares is mature or phobos has pin/unpin, or pointers passed to C are pinned implicitly.They will be pinned implicitly, no need to call pin/unpin. The only time you're going to have a problem is if the C routine hides the pointer in a malloc'd buffer where the gc won't see it. The gc *will* see pointers on the C stack.
Jun 07 2006
Of course, how foolish of me - so there's no worry about the garbage collector killing that in a threaded program either. Duh. -[Unknown] In article <e679ot$7e3$1 digitaldaemon.com>, Walter Bright says...Stewart Gordon wrote:Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means?It's pinned because it's on the stack, just like pointers on the stack for your D functions. The gc doesn't know the difference.
Jun 07 2006
On Wed, 7 Jun 2006 22:11:52 +0000 (UTC), Unknown W. Brackets <Unknown_member pathlink.com> wrote:Of course, how foolish of me - so there's no worry about the garbage collector killing that in a threaded program either. Duh. -[Unknown] In article <e679ot$7e3$1 digitaldaemon.com>, Walter Bright says...Thanks. It seems I can live with that for now. i can't help saying D is very addictive. imho, the best C-like language i've had a chance to try.Stewart Gordon wrote:Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means?It's pinned because it's on the stack, just like pointers on the stack for your D functions. The gc doesn't know the difference.
Jun 08 2006
Walter Bright wrote:Stewart Gordon wrote:<snip>But what if the foreign code needs to hold onto the pointer for longer than it can remain on the stack? Stewart.Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means?It's pinned because it's on the stack, just like pointers on the stack for your D functions. The gc doesn't know the difference.
Jun 08 2006
Stewart Gordon wrote:Walter Bright wrote:Have the D caller code keep a copy, or use addRoots to tell the gc where the foreign code is holding it.Stewart Gordon wrote:<snip>But what if the foreign code needs to hold onto the pointer for longer than it can remain on the stack?Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means?It's pinned because it's on the stack, just like pointers on the stack for your D functions. The gc doesn't know the difference.
Jun 08 2006
But the question is for a "moving" GC, which might not consider a local reference to mean "don't move it"...? -[Unknown] In article <e6abmn$1iaj$1 digitaldaemon.com>, Walter Bright says...Stewart Gordon wrote:But what if the foreign code needs to hold onto the pointer for longer than it can remain on the stack?Have the D caller code keep a copy, or use addRoots to tell the gc where the foreign code is holding it.
Jun 08 2006
Unknown W. Brackets wrote:But the question is for a "moving" GC, which might not consider a local reference to mean "don't move it"...?addroots will work with that.
Jun 09 2006
But what if I don't know where the foreign code is holding it? The C library might not expose this, and then I'm out of luck... unless I can modify it, which I probably can't. But, this is all theoretical anyway. I can't even think of a library that keeps a pointer I give it. -[Unknown]Unknown W. Brackets wrote:But the question is for a "moving" GC, which might not consider a local reference to mean "don't move it"...?addroots will work with that.
Jun 09 2006
Walter Bright wrote:Stewart Gordon wrote:<snip>What is addRoot_s_?But what if the foreign code needs to hold onto the pointer for longer than it can remain on the stack?Have the D caller code keep a copy, or use addRootsto tell the gc where the foreign code is holding it.How will I know where the foreign code is holding the pointer? Surely it makes more sense to add the pointer itself as a root. Moreover, the pinning methods'll need to be documented. So at the moment, I'm guessing that the following will implicitly pin a GC-allocated block: - storing a pointer to it in a union - having a pointer to it on the stack - adding it as a root Stewart.
Jun 09 2006
Stewart Gordon wrote:Walter Bright wrote:http://www.digitalmars.com/d/phobos/std_gc.htmlStewart Gordon wrote:<snip>What is addRoot_s_?But what if the foreign code needs to hold onto the pointer for longer than it can remain on the stack?Have the D caller code keep a copy, or use addRootsFirst of all, the foreign code documentation ought to make some mention of what it does with the pointer - after all, it must even if you malloc'd it so you know when and if you can ever free it. Have the D code caller call addRoot.to tell the gc where the foreign code is holding it.How will I know where the foreign code is holding the pointer?
Jun 09 2006
Walter Bright wrote:Stewart Gordon wrote:Whoa, I got a bit confused here. Is any reference that is held in the stack automatically pinned (i.e., the referred data doesn't move) ? If so, why is that, why wouldn't a moving collector move it? -- Bruno Medeiros - CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#DWalter Bright wrote:It's pinned because it's on the stack, just like pointers on the stack for your D functions. The gc doesn't know the difference.Max Samuha wrote:Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means?Thank you for the extensive explanation! Marking my code with '// !!! Passing a pointer to C.' is what i've been doing recently:) and it seems the best solution for now until ares is mature or phobos has pin/unpin, or pointers passed to C are pinned implicitly.They will be pinned implicitly, no need to call pin/unpin. The only time you're going to have a problem is if the C routine hides the pointer in a malloc'd buffer where the gc won't see it. The gc *will* see pointers on the C stack.
Jun 09 2006
Bruno Medeiros wrote:Whoa, I got a bit confused here. Is any reference that is held in the stack automatically pinned (i.e., the referred data doesn't move) ?Yes.If so, why is that, why wouldn't a moving collector move it?Because then you'd have to track all the register contents instruction by instruction.
Jun 09 2006
Walter Bright wrote:Bruno Medeiros wrote::/ I'm lost, I didn't understand that at all. "track all the register contents instruction by instruction" ? -- Bruno Medeiros - CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#DWhoa, I got a bit confused here. Is any reference that is held in the stack automatically pinned (i.e., the referred data doesn't move) ?Yes.If so, why is that, why wouldn't a moving collector move it?Because then you'd have to track all the register contents instruction by instruction.
Jun 12 2006
Bruno Medeiros wrote:Walter Bright wrote:DISCLAIMER: /me is not an x86 assembler expert. Take with the requisite grain of salt. Think about it like this: say you want to dereference a pointer. Let's assume you've got something like this: Now, you move this into the ESI pointer to dereference it. And then HOLD IT! Moving garbage collector coming through! Hmm, this "foo" thing could be moved to be more efficient; let's move it! Now, go back to your business. What happen? Someone set up us the access violation! That damned moving GC preempted us and moved our data before we could get to it! Even if you're not multi-threaded, this is still a problem: there's nothing to preclude a pointer being stored in a register, calling new, causing a moving collect, and then attempting to dereference it. Isn't memory management fun? :P -- Daniel -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/Bruno Medeiros wrote::/ I'm lost, I didn't understand that at all. "track all the register contents instruction by instruction" ?Whoa, I got a bit confused here. Is any reference that is held in the stack automatically pinned (i.e., the referred data doesn't move) ?Yes.If so, why is that, why wouldn't a moving collector move it?Because then you'd have to track all the register contents instruction by instruction.
Jun 12 2006
Daniel Keep wrote:Bruno Medeiros wrote:The very same thing can happen with references stored in the heap or the static segment (i.e. globals). That is, with *any* reference. -- Bruno Medeiros - CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#DWalter Bright wrote:DISCLAIMER: /me is not an x86 assembler expert. Take with the requisite grain of salt. Think about it like this: say you want to dereference a pointer. Let's assume you've got something like this: Now, you move this into the ESI pointer to dereference it. And then HOLD IT! Moving garbage collector coming through! Hmm, this "foo" thing could be moved to be more efficient; let's move it! Now, go back to your business. What happen? Someone set up us the access violation! That damned moving GC preempted us and moved our data before we could get to it!Bruno Medeiros wrote::/ I'm lost, I didn't understand that at all. "track all the register contents instruction by instruction" ?Whoa, I got a bit confused here. Is any reference that is held in the stack automatically pinned (i.e., the referred data doesn't move) ?Yes.If so, why is that, why wouldn't a moving collector move it?Because then you'd have to track all the register contents instruction by instruction.
Jun 12 2006
Max Samuha wrote:How to ensure D code that interoperates with external libraries will work when a moving gc is eventually used with D? Will i need to add lots of pinning/unpinning code?<snip> Pinning has been requested a few times on these 'groups. There has also been a discussion before on issues that still need to be addressed before a GC that moves stuff around will work: http://www.digitalmars.com/d/archives/26273.html Stewart.
Jun 06 2006
On Tue, 06 Jun 2006 14:27:22 +0100, Stewart Gordon <smjg_1998 yahoo.com> wrote:Pinning has been requested a few times on these 'groups. There has also been a discussion before on issues that still need to be addressed before a GC that moves stuff around will work: http://www.digitalmars.com/d/archives/26273.html Stewart.Yes, the thread you mention was posted two years ago. It would be really nice to hear a word from Walter on this issue.
Jun 06 2006