www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Weak references.

reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
I've been looking at some of my old nonagon code, and as I'm sure you've all 
experienced looking at your old code, I'm just _cringing_ at some of the 
things I did in it ;)

One thing that I've noticed would be nice to have, however, would be weak 
references to some objects.  For example, I have "brush" objects which 
encapsulate a bunch of visual settings (textures, material, blending modes 
etc.).  These are referenced by objects that can be rendered.  Some of these 
brushes need to be updated often (i.e. every frame to update the environment 
mapping matrices).  So I keep a list (table) of all instances of this brush 
class so I can easily loop through them and update them all.

However when their owners disappear, I end up with uncollectable brushes, 
since they are still being referenced by that table.

So what would be nice to have happen would be to have that table be a table 
of _weak references_ to brush objects.  Then, when no more real references 
existed, they could be collected, and I wouldn't be keeping them around for 
no reason anymore.

I've looked into the scrapple weakref implementation, but it doesn't seem to 
do what I want.  Seemingly it only works if the object that it references is 
explicitly deleted.  I want the weakref to become null if the object that it 
references is collected in any way.

Is this possible at all?  Am I just not using Bill's weakref properly?
Apr 12 2008
next sibling parent Jason House <jason.james.house gmail.com> writes:
Take another look... Weak references are not viewed as references. This means
the objects will get collected when only weak references exist. It sounds like
that is what you want...

Jarrett Billingsley Wrote:


 I've been looking at some of my old nonagon code, and as I'm sure you've all 
 experienced looking at your old code, I'm just _cringing_ at some of the 
 things I did in it ;)
 
 One thing that I've noticed would be nice to have, however, would be weak 
 references to some objects.  For example, I have "brush" objects which 
 encapsulate a bunch of visual settings (textures, material, blending modes 
 etc.).  These are referenced by objects that can be rendered.  Some of these 
 brushes need to be updated often (i.e. every frame to update the environment 
 mapping matrices).  So I keep a list (table) of all instances of this brush 
 class so I can easily loop through them and update them all.
 
 However when their owners disappear, I end up with uncollectable brushes, 
 since they are still being referenced by that table.
 
 So what would be nice to have happen would be to have that table be a table 
 of _weak references_ to brush objects.  Then, when no more real references 
 existed, they could be collected, and I wouldn't be keeping them around for 
 no reason anymore.
 
 I've looked into the scrapple weakref implementation, but it doesn't seem to 
 do what I want.  Seemingly it only works if the object that it references is 
 explicitly deleted.  I want the weakref to become null if the object that it 
 references is collected in any way.
 
 Is this possible at all?  Am I just not using Bill's weakref properly?
Apr 12 2008
prev sibling next sibling parent Bill Baxter <dnewsgroup billbaxter.com> writes:
Jarrett Billingsley wrote:

 I've been looking at some of my old nonagon code, and as I'm sure you've all 
 experienced looking at your old code, I'm just _cringing_ at some of the 
 things I did in it ;)
 
 One thing that I've noticed would be nice to have, however, would be weak 
 references to some objects.  For example, I have "brush" objects which 
 encapsulate a bunch of visual settings (textures, material, blending modes 
 etc.).  These are referenced by objects that can be rendered.  Some of these 
 brushes need to be updated often (i.e. every frame to update the environment 
 mapping matrices).  So I keep a list (table) of all instances of this brush 
 class so I can easily loop through them and update them all.
 
 However when their owners disappear, I end up with uncollectable brushes, 
 since they are still being referenced by that table.
 
 So what would be nice to have happen would be to have that table be a table 
 of _weak references_ to brush objects.  Then, when no more real references 
 existed, they could be collected, and I wouldn't be keeping them around for 
 no reason anymore.
 
 I've looked into the scrapple weakref implementation, but it doesn't seem to 
 do what I want.  Seemingly it only works if the object that it references is 
 explicitly deleted.  I want the weakref to become null if the object that it 
 references is collected in any way.
 
 Is this possible at all?  Am I just not using Bill's weakref properly? 
 
 


It does exactly what you're asking for ... as long as your "Brushes" are 
class instances, which it sounds like they are.
Instead of filling the table with type Brush, fill it with type 
WeakRef!(Brush).  Then when you go to use them from the table you should 
check for null first.  If the ptr value is null, it means the object got 
collected at some point, so you can remove it from the table.

--bb
Apr 12 2008
prev sibling next sibling parent reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Jarrett Billingsley" <kb3ctd2 yahoo.com> wrote in message 
news:ftr9as$12s2$1 digitalmars.com...

 I've been looking at some of my old nonagon code, and as I'm sure you've 
 all experienced looking at your old code, I'm just _cringing_ at some of 
 the things I did in it ;)

 One thing that I've noticed would be nice to have, however, would be weak 
 references to some objects.  For example, I have "brush" objects which 
 encapsulate a bunch of visual settings (textures, material, blending modes 
 etc.).  These are referenced by objects that can be rendered.  Some of 
 these brushes need to be updated often (i.e. every frame to update the 
 environment mapping matrices).  So I keep a list (table) of all instances 
 of this brush class so I can easily loop through them and update them all.

 However when their owners disappear, I end up with uncollectable brushes, 
 since they are still being referenced by that table.

 So what would be nice to have happen would be to have that table be a 
 table of _weak references_ to brush objects.  Then, when no more real 
 references existed, they could be collected, and I wouldn't be keeping 
 them around for no reason anymore.

 I've looked into the scrapple weakref implementation, but it doesn't seem 
 to do what I want.  Seemingly it only works if the object that it 
 references is explicitly deleted.  I want the weakref to become null if 
 the object that it references is collected in any way.

 Is this possible at all?  Am I just not using Bill's weakref properly?


To both: yes, that's what I would have expected, but I can't seem to get the 
object to be collected even if the only reference to it is a weak reference. 
Tried forcing a full collect, allocating bunches of memory, collecting 
again, allocating at the beginning of a long-running program etc.  it never 
seems to get collected, even if I never assign the reference into an actual 
pointer type.

I'm using Tango if it makes any difference.
Apr 12 2008
next sibling parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Jarrett Billingsley wrote:

 To both: yes, that's what I would have expected, but I can't seem to get the 
 object to be collected even if the only reference to it is a weak reference. 
 Tried forcing a full collect, allocating bunches of memory, collecting 
 again, allocating at the beginning of a long-running program etc.  it never 
 seems to get collected, even if I never assign the reference into an actual 
 pointer type.
 
 I'm using Tango if it makes any difference. 


Have you tried looking at the generated assembler code? Perhaps the 
compiler is leaving a reference in some register that's never 
overwritten afterwards or something...
Apr 13 2008
parent "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Frits van Bommel" <fvbommel REMwOVExCAPSs.nl> wrote in message 
news:ftsq01$f05$1 digitalmars.com...

 Jarrett Billingsley wrote:

 To both: yes, that's what I would have expected, but I can't seem to get 
 the object to be collected even if the only reference to it is a weak 
 reference. Tried forcing a full collect, allocating bunches of memory, 
 collecting again, allocating at the beginning of a long-running program 
 etc.  it never seems to get collected, even if I never assign the 
 reference into an actual pointer type.

 I'm using Tango if it makes any difference.


 Have you tried looking at the generated assembler code? Perhaps the 
 compiler is leaving a reference in some register that's never overwritten 
 afterwards or something...


That's what I was thinking, which is why I tried running lots of code after 
the allocation, to clear out the registers.

I've checked the memory block allocated for the WeakRef instance and it's 
marked as "finalize" and "no scan", which are both right.

I absolutely cannot get it to collect the instance that it references.

I really wonder if this is a bug in the Tango GC.
Apr 13 2008
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Jarrett Billingsley" <kb3ctd2 yahoo.com> wrote in message
 news:ftr9as$12s2$1 digitalmars.com...

 I've been looking at some of my old nonagon code, and as I'm sure you've
 all experienced looking at your old code, I'm just _cringing_ at some of
 the things I did in it ;)

 One thing that I've noticed would be nice to have, however, would be weak
 references to some objects.  For example, I have "brush" objects which
 encapsulate a bunch of visual settings (textures, material, blending modes
 etc.).  These are referenced by objects that can be rendered.  Some of
 these brushes need to be updated often (i.e. every frame to update the
 environment mapping matrices).  So I keep a list (table) of all instances
 of this brush class so I can easily loop through them and update them all.

 However when their owners disappear, I end up with uncollectable brushes,
 since they are still being referenced by that table.

 So what would be nice to have happen would be to have that table be a
 table of _weak references_ to brush objects.  Then, when no more real
 references existed, they could be collected, and I wouldn't be keeping
 them around for no reason anymore.

 I've looked into the scrapple weakref implementation, but it doesn't seem
 to do what I want.  Seemingly it only works if the object that it
 references is explicitly deleted.  I want the weakref to become null if
 the object that it references is collected in any way.

 Is this possible at all?  Am I just not using Bill's weakref properly?


 To both: yes, that's what I would have expected, but I can't seem to get the
 object to be collected even if the only reference to it is a weak reference.
 Tried forcing a full collect, allocating bunches of memory, collecting
 again, allocating at the beginning of a long-running program etc.  it never
 seems to get collected, even if I never assign the reference into an actual
 pointer type.
 I'm using Tango if it makes any difference.


Are these tables AAs?  And how do the weak references store their reference?
The type information provided to AAs is a bit weak, and so some non-pointer
data is currently still scanned by the GC.  I'm going to improve this a bit in
the
next Tango release, but it still won't be perfect (because the compiler doesn't
provide enough info for it to be).


Sean
Apr 13 2008
parent reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Sean Kelly" <sean invisibleduck.org> wrote in message 
news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their 
 reference?
 The type information provided to AAs is a bit weak, and so some 
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a 
 bit in the
 next Tango release, but it still won't be perfect (because the compiler 
 doesn't
 provide enough info for it to be).


In my tests I've just been doing

auto weakref = new WeakRef!(A)(new A);

and then checking that after doing god-knows-what to try to get it to 
collect that instance of A, so it's not even that I'm doing anything that 
crazy.
Apr 13 2008
next sibling parent reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Jarrett Billingsley" <kb3ctd2 yahoo.com> wrote in message 
news:fttdej$1nfs$1 digitalmars.com...

 "Sean Kelly" <sean invisibleduck.org> wrote in message 
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their 
 reference?
 The type information provided to AAs is a bit weak, and so some 
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a 
 bit in the
 next Tango release, but it still won't be perfect (because the compiler 
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing

 auto weakref = new WeakRef!(A)(new A);

 and then checking that after doing god-knows-what to try to get it to 
 collect that instance of A, so it's not even that I'm doing anything that 
 crazy.


Testing my code in Linux causes a crash in glibc's free() after the program 
exits.  (And the assertion still fails.)

Something really weird is happening.
Apr 13 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Jarrett Billingsley wrote:


 "Jarrett Billingsley" <kb3ctd2 yahoo.com> wrote in message
 news:fttdej$1nfs$1 digitalmars.com...

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing

 auto weakref = new WeakRef!(A)(new A);

 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


 
 Testing my code in Linux causes a crash in glibc's free() after the
 program
 exits.  (And the assertion still fails.)
 
 Something really weird is happening.


Ok, my latest testing (and testing by others for configurations I don't
have) are collected in the list below.  I don't see any pattern.

tango+linux+dmd = works
tango+linux+gdc = crash in glibc
tango+windows+dmd = does not collect
tango+windows+gdc = ???
phobos+linux+dmd = works
phobos+linux+gdc = does not collect
phobos+windows+dmd = ???
phobos+windows+gdc = ???

These tests are based off the code at http://tango.pastebin.com/m10aa57b6
Note that phobos users must comment out the version=Tango line.  Any
additional tests would be appreciated.
Apr 13 2008
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Jason House wrote:

 Jarrett Billingsley wrote:
 

 "Jarrett Billingsley" <kb3ctd2 yahoo.com> wrote in message
 news:fttdej$1nfs$1 digitalmars.com...

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing

 auto weakref = new WeakRef!(A)(new A);

 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


 Testing my code in Linux causes a crash in glibc's free() after the
 program
 exits.  (And the assertion still fails.)

 Something really weird is happening.


 
 Ok, my latest testing (and testing by others for configurations I don't
 have) are collected in the list below.  I don't see any pattern.
 
 tango+linux+dmd = works
 tango+linux+gdc = crash in glibc
 tango+windows+dmd = does not collect
 tango+windows+gdc = ???
 phobos+linux+dmd = works
 phobos+linux+gdc = does not collect
 phobos+windows+dmd = ???
 phobos+windows+gdc = ???
 
 These tests are based off the code at http://tango.pastebin.com/m10aa57b6
 Note that phobos users must comment out the version=Tango line.  Any
 additional tests would be appreciated.
 


I'd be worried about stack references in the second example.  That inner 
function makeWa may be getting inlined or something with some compiler 
combos.

The first unittest there works ok right?  Maybe explicitly overwriting a 
with null after using it would help.

Anyway, I'm pretty sure Sean said after he added the functions to Tango 
that he thought they should work but they weren't tested very well.  So 
I wouldn't be surprised if there were issues.  And really, Phobos's gc 
functions for weak ref support probably haven't been tested all that 
well either.

--bb
Apr 13 2008
parent Jason House <jason.james.house gmail.com> writes:
Bill Baxter wrote:


 Jason House wrote:

 Jarrett Billingsley wrote:
 

 "Jarrett Billingsley" <kb3ctd2 yahoo.com> wrote in message
 news:fttdej$1nfs$1 digitalmars.com...

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this
 a bit in the
 next Tango release, but it still won't be perfect (because the
 compiler doesn't
 provide enough info for it to be).


 In my tests I've just been doing

 auto weakref = new WeakRef!(A)(new A);

 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything
 that crazy.


 Testing my code in Linux causes a crash in glibc's free() after the
 program
 exits.  (And the assertion still fails.)

 Something really weird is happening.


 
 Ok, my latest testing (and testing by others for configurations I don't
 have) are collected in the list below.  I don't see any pattern.
 
 tango+linux+dmd = works
 tango+linux+gdc = crash in glibc
 tango+windows+dmd = does not collect
 tango+windows+gdc = ???
 phobos+linux+dmd = works
 phobos+linux+gdc = does not collect
 phobos+windows+dmd = ???
 phobos+windows+gdc = ???
 
 These tests are based off the code at http://tango.pastebin.com/m10aa57b6
 Note that phobos users must comment out the version=Tango line.  Any
 additional tests would be appreciated.
 


 
 I'd be worried about stack references in the second example.  That inner
 function makeWa may be getting inlined or something with some compiler
 combos.
 
 The first unittest there works ok right?  Maybe explicitly overwriting a
 with null after using it would help.
 
 Anyway, I'm pretty sure Sean said after he added the functions to Tango
 that he thought they should work but they weren't tested very well.  So
 I wouldn't be surprised if there were issues.  And really, Phobos's gc
 functions for weak ref support probably haven't been tested all that
 well either.
 
 --bb


I'm mostly trying to extract as much info as I can from this example to
better help me file a bug report.  The unit test can't differentiate
between "never cleans up" and "didn't clean up from register values hanging
around"...  Nevertheless, it's a good start for further investigation. 
Certainly the crash inside glibc is a bug in either gdc, tango, or both.
Apr 13 2008
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing
 auto weakref = new WeakRef!(A)(new A);
 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


Just for kicks, try querying the GC to determine whether the GC thinks
weakref contains pointers.


Sean
Apr 13 2008
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
Sean Kelly wrote:


 == Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing
 auto weakref = new WeakRef!(A)(new A);
 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


 
 Just for kicks, try querying the GC to determine whether the GC thinks
 weakref contains pointers.
 
 
 Sean


How would one do that?
Apr 13 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article

 Sean Kelly wrote:

 == Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing
 auto weakref = new WeakRef!(A)(new A);
 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


 Just for kicks, try querying the GC to determine whether the GC thinks
 weakref contains pointers.


 Sean


 How would one do that?


(from memory)

    if( GC.getAttr( weakref ) & GC.BlkAttr.NO_SCAN )
        printf( "The GC will not scan weakref\n" );


Sean
Apr 13 2008
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Sean Kelly wrote:

 == Quote from Jason House (jason.james.house gmail.com)'s article

 Sean Kelly wrote:

 == Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing
 auto weakref = new WeakRef!(A)(new A);
 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


 Just for kicks, try querying the GC to determine whether the GC thinks
 weakref contains pointers.


 Sean


 How would one do that?


 
 (from memory)
 
     if( GC.getAttr( weakref ) & GC.BlkAttr.NO_SCAN )
         printf( "The GC will not scan weakref\n" );


Is that how it's supposed to work?  I mean WeakRef is basically

class WeakRef { size_t hidden_pointer; }

Does that mean Tango shouldn't be trying to scan instances of it?  I 
just ask because I was thinking the GC *would* scan the WeakRef's memory 
just not find anything that looks like a pointer there.


Also (for Jason, in particular) it occurs to me that this code:

   assert(wa.ptr !is null);
   collect();
   assert(wa.ptr is null);

Has to store wa.ptr somewhere (register or stack) when you ask to check 
if it is null or not.  So that act of checking itself will likely make a 
reference that prevents the following collect() from cleaning it up.

--bb
Apr 13 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Bill Baxter Wrote:

     if( GC.getAttr( weakref ) & GC.BlkAttr.NO_SCAN )
         printf( "The GC will not scan weakref\n" );


 
 Is that how it's supposed to work?  I mean WeakRef is basically
 
 class WeakRef { size_t hidden_pointer; }
 
 Does that mean Tango shouldn't be trying to scan instances of it?  I 
 just ask because I was thinking the GC *would* scan the WeakRef's memory 
 just not find anything that looks like a pointer there.


That's the intent.  It keeps a reference without the GC detecting it.  The only
extra wrinkle to the implementation is that it needs to null out its internal
pointer when the object gets collected.



 Also (for Jason, in particular) it occurs to me that this code:
 
    assert(wa.ptr !is null);
    collect();
    assert(wa.ptr is null);
 
 Has to store wa.ptr somewhere (register or stack) when you ask to check 
 if it is null or not.  So that act of checking itself will likely make a 
 reference that prevents the following collect() from cleaning it up.


That's true that that might cause the last assert to fail, oops.  I don't know
what the correct way to clear everything out is (including from the preceding
function call).  Any more tips on making a more correct unit test are
appreciated. 

Jarret was having issues with non-collection (ever) when using weak ref's and I
had written this test to try and exercise it (his problem is windows + tango +
dmd).  An assert failure doesn't mean it won't ever collect but will narrow the
search down and allow closer examination... and the crash on linux+tango+gdc is
definitely not correct behavior.
Apr 14 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article

 Bill Baxter Wrote:

     if( GC.getAttr( weakref ) & GC.BlkAttr.NO_SCAN )
         printf( "The GC will not scan weakref\n" );


 Is that how it's supposed to work?  I mean WeakRef is basically

 class WeakRef { size_t hidden_pointer; }

 Does that mean Tango shouldn't be trying to scan instances of it?  I
 just ask because I was thinking the GC *would* scan the WeakRef's memory
 just not find anything that looks like a pointer there.


 That's the intent.  It keeps a reference without the GC detecting it.  The
only extra wrinkle to the implementation is that it needs to


null out its internal pointer when the object gets collected.

 Also (for Jason, in particular) it occurs to me that this code:

    assert(wa.ptr !is null);
    collect();
    assert(wa.ptr is null);

 Has to store wa.ptr somewhere (register or stack) when you ask to check
 if it is null or not.  So that act of checking itself will likely make a
 reference that prevents the following collect() from cleaning it up.


 That's true that that might cause the last assert to fail, oops.  I don't know
what the correct way to clear everything out is (including


from the preceding function call).  Any more tips on making a more correct unit
test are appreciated.

The easiest thing I can think of would be to do an access of a dummy weakptr
after each access of the real weakptr:

auto dummy = new WeakPtr!(void);
assert(wa.ptr !is null);
assert(dummy.ptr is null);
collect();
assert(wa.ptr is null);

Since the dummy op is identical to the real op, it should hopefully hit the same
registers.  Best way to be sure would be to test with -O turned off.


 Jarret was having issues with non-collection (ever) when using weak ref's and
I had written this test to try and exercise it (his problem


is windows + tango + dmd).  An assert failure doesn't mean it won't ever
collect but will narrow the search down and allow closer
examination... and the crash on linux+tango+gdc is definitely not correct
behavior.

Thanks for any investigation you do for this.  If you can find a bug in Tango,
I'd be
happy to fix it, but I'm too busy to do the research myself at the moment.


Sean
Apr 14 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Sean Kelly Wrote:

 Thanks for any investigation you do for this.  If you can find a bug in Tango,
I'd be
 happy to fix it, but I'm too busy to do the research myself at the moment.


I have a vested interest in it... I want my code to work ;)

Just to be clear, what is enough to proclaim stuff as a bug in Tango?

For the failed assert: prior to collect, show a lack of lingering references
through disassembly?

For a runtime crash: Isn't any crash in core libraries used by Tango a Tango
bug?

For whatever I can prove to be a Tango bug, I'll submit a ticket.
Apr 14 2008
next sibling parent Lars Ivar Igesund <larsivar igesund.net> writes:
Jason House wrote:


 Sean Kelly Wrote:

 Thanks for any investigation you do for this.  If you can find a bug in
 Tango, I'd be happy to fix it, but I'm too busy to do the research myself
 at the moment.


 
 I have a vested interest in it... I want my code to work ;)
 
 Just to be clear, what is enough to proclaim stuff as a bug in Tango?
 
 For the failed assert: prior to collect, show a lack of lingering
 references through disassembly?
 
 For a runtime crash: Isn't any crash in core libraries used by Tango a
 Tango bug?
 
 For whatever I can prove to be a Tango bug, I'll submit a ticket.


You can create a Tango ticket for anything directly related to Tango that
doesn't work. In this particular case, a compiler bug is at least as likely
(due to what is working with what where). We have many tickets for problems
that have been verified to be toolchain problems, as it is easier for us to
track how it affect us that way.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource, #d.tango & #D: larsivi
Dancing the Tango
Apr 14 2008
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article

 Sean Kelly Wrote:

 Thanks for any investigation you do for this.  If you can find a bug in Tango,
I'd be
 happy to fix it, but I'm too busy to do the research myself at the moment.


 I have a vested interest in it... I want my code to work ;)
 Just to be clear, what is enough to proclaim stuff as a bug in Tango?


Ideally, any bug report will contain a simple repro and/or mention of where the
actual
problem is in Tango.  But we see ambiguous bug reports or things that, after
investigation,
turn out to be compiler problems and the like.  But the easier you can make
things on our
end, the faster the problem is likely to be fixed :-)


 For the failed assert: prior to collect, show a lack of lingering references
through disassembly?


What I'd do to diagnose this is rebuild the Tango runtime with debug reporting
turned on for the GC, possibly adding something to print the address of all
blocks being collected (if that doesn't exist already).  We know that weakptr
data isn't being collected on Win32... the question is why.



 For a runtime crash: Isn't any crash in core libraries used by Tango a Tango
bug?


Not necessarily.  We've seen a bunch of crashes that GDB reported were in the
runtime
that actually turned out to be compiler bugs in GDC, or in some cases errors in
the
user program that ended up surfacing as runtime code exploding.  But in theory
you're
right.


 For whatever I can prove to be a Tango bug, I'll submit a ticket.


Thank you :-)


Sean
Apr 14 2008
prev sibling parent reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Sean Kelly" <sean invisibleduck.org> wrote in message 
news:ftubv9$kph$1 digitalmars.com...

 == Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing
 auto weakref = new WeakRef!(A)(new A);
 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


 Just for kicks, try querying the GC to determine whether the GC thinks
 weakref contains pointers.


Already tried it.  The GC sees the weakref instance's memory as NO_SCAN and 
FINALIZE, both correct.
Apr 13 2008
parent Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:ftubv9$kph$1 digitalmars.com...

 == Quote from Jarrett Billingsley (kb3ctd2 yahoo.com)'s article

 "Sean Kelly" <sean invisibleduck.org> wrote in message
 news:fttd02$1mp1$1 digitalmars.com...

 Are these tables AAs?  And how do the weak references store their
 reference?
 The type information provided to AAs is a bit weak, and so some
 non-pointer
 data is currently still scanned by the GC.  I'm going to improve this a
 bit in the
 next Tango release, but it still won't be perfect (because the compiler
 doesn't
 provide enough info for it to be).


 In my tests I've just been doing
 auto weakref = new WeakRef!(A)(new A);
 and then checking that after doing god-knows-what to try to get it to
 collect that instance of A, so it's not even that I'm doing anything that
 crazy.


 Just for kicks, try querying the GC to determine whether the GC thinks
 weakref contains pointers.


 Already tried it.  The GC sees the weakref instance's memory as NO_SCAN and
 FINALIZE, both correct.


Huh.  Afraid I'm at a loss as to what's going on then.  Even stranger that it
works on linux but not Win32, since the code is basically the same for both.
I suppose I could add some profiling code to try and figure out where the
lingering reference is, assuming there is one.  But not sure when I'll have
the time to do this.


Sean
Apr 14 2008
prev sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2008-04-12 17:24:44 -0400, "Jarrett Billingsley" <kb3ctd2 yahoo.com> said:


 Is this possible at all?  Am I just not using Bill's weakref properly?


I've been using an alternate strategy in the D/Objective-C bridge for 
the bridged object lookup table. Basically, I'm casting object 
references to size_t (so they're no longer pointers) and mangling the 
pointers (so the garbage collector can't see them as pointers, I think 
this is no longer be necessary). When the bridged object gets deleted 
on the D side, it removes its mangled pointer from the table.

So if your referenced object knows about its weak reference in the 
table, you could implement it as such and don't have to worry about 
null pointers polluting your table.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Apr 13 2008
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Michel Fortin wrote:

 On 2008-04-12 17:24:44 -0400, "Jarrett Billingsley" <kb3ctd2 yahoo.com> 
 said:
 

 Is this possible at all?  Am I just not using Bill's weakref properly?


 
 I've been using an alternate strategy in the D/Objective-C bridge for 
 the bridged object lookup table. Basically, I'm casting object 
 references to size_t (so they're no longer pointers) 


Yes, that's what the weak ref class does.


 and mangling the 
 pointers (so the garbage collector can't see them as pointers, I think 
 this is no longer be necessary). 


I agree that that's not necessary.


 When the bridged object gets deleted on 
 the D side, it removes its mangled pointer from the table.
 
 So if your referenced object knows about its weak reference in the 
 table, you could implement it as such and don't have to worry about null 
 pointers polluting your table.


Good idea.  I was thinking to suggest something like that too.  A 
WeakRef class is not really an optimal way to keep track of a collection 
of weak refs.  In addition to null pointers polluting the table, there's 
also just the basic overhead of needing to create an extra object for 
each entry.  Ideally there would be WeakArray, WeakSet, WeakHeap... etc. 
  I think I've seen such things in some language.  Python maybe?

--bb
Apr 13 2008
next sibling parent reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Bill Baxter" <dnewsgroup billbaxter.com> wrote in message 
news:fttn2m$2emt$1 digitalmars.com...

 Michel Fortin wrote:

 On 2008-04-12 17:24:44 -0400, "Jarrett Billingsley" <kb3ctd2 yahoo.com> 
 said:


 Is this possible at all?  Am I just not using Bill's weakref properly?


 I've been using an alternate strategy in the D/Objective-C bridge for the 
 bridged object lookup table. Basically, I'm casting object references to 
 size_t (so they're no longer pointers)


 Yes, that's what the weak ref class does.


 and mangling the pointers (so the garbage collector can't see them as 
 pointers, I think this is no longer be necessary).


 I agree that that's not necessary.


 When the bridged object gets deleted on the D side, it removes its 
 mangled pointer from the table.

 So if your referenced object knows about its weak reference in the table, 
 you could implement it as such and don't have to worry about null 
 pointers polluting your table.


 Good idea.  I was thinking to suggest something like that too.  A WeakRef 
 class is not really an optimal way to keep track of a collection of weak 
 refs.  In addition to null pointers polluting the table, there's also just 
 the basic overhead of needing to create an extra object for each entry. 
 Ideally there would be WeakArray, WeakSet, WeakHeap... etc. I think I've 
 seen such things in some language.  Python maybe?


Lua's tables can have weak keys, weak values, or both.
Apr 13 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Jarrett Billingsley wrote:

 Lua's tables can have weak keys, weak values, or both. 


And MiniD's...? ;-P
Apr 13 2008
parent "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Robert Fraser" <fraserofthenight gmail.com> wrote in message 
news:fttt4r$2oic$2 digitalmars.com...

 Jarrett Billingsley wrote:

 Lua's tables can have weak keys, weak values, or both.


 And MiniD's...? ;-P


If I had a weak reference lib for D it could X(
Apr 13 2008
prev sibling parent reply Bjoern <nanali nospam-wanadoo.fr> writes:
Bill Baxter schrieb:

 of weak refs.  In addition to null pointers polluting the table, there's 
 also just the basic overhead of needing to create an extra object for 
 each entry.  Ideally there would be WeakArray, WeakSet, WeakHeap... etc. 
  I think I've seen such things in some language.  Python maybe?
 
 --bb


weakHashMap, weakTreeMap ???

implemented in Java.... ? OpenJDK/ GNU Classpath
Well, Frank is looking for a D weakHashMap implementation. Due to the 
fact that I am actually doing plombering, plastering, electric, 
establish tons of cement concrete  ... I am not able to offer a 
solution. (Beeing allready happy that I am  able to write a msg)

Nevertheless I did like to suggest to use Bill B's weakRef!  as  inner 
adaptor class of WeakHashMap.  (pretty much the same as in GNU 
classpath)  as static array of WeakRef objects. Further the dCollection 
: see dSource / collection classes are <IMO> fitting better than Tango 
Collection.


Bjoern
Apr 23 2008
parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Bjoern wrote:

 Bill Baxter schrieb:

 of weak refs.  In addition to null pointers polluting the table, 
 there's also just the basic overhead of needing to create an extra 
 object for each entry.  Ideally there would be WeakArray, WeakSet, 
 WeakHeap... etc.  I think I've seen such things in some language.  
 Python maybe?

 --bb


 
 weakHashMap, weakTreeMap ???
 
 implemented in Java.... ? OpenJDK/ GNU Classpath
 Well, Frank is looking for a D weakHashMap implementation. Due to the 
 fact that I am actually doing plombering, plastering, electric, 
 establish tons of cement concrete  ... I am not able to offer a 
 solution. (Beeing allready happy that I am  able to write a msg)
 
 Nevertheless I did like to suggest to use Bill B's weakRef!  as  inner 
 adaptor class of WeakHashMap.  (pretty much the same as in GNU 
 classpath)  as static array of WeakRef objects. Further the dCollection 
 : see dSource / collection classes are <IMO> fitting better than Tango 
 Collection.


Using the wrappers might have too much overhead. But perhaps MinTL with 
an allocator using malloc would do the trick?


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Apr 23 2008