www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - weak references

reply Brian Hammond <d brianhammond.com> writes:
Does (or will) D support weak references?

See http://www.python.org/doc/current/lib/module-weakref.html for a
backgrounder.

Thanks!
May 07 2004
next sibling parent reply Ilya Minkov <minkov cs.tum.edu> writes:
Brian Hammond schrieb:

 Does (or will) D support weak references?
 
 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.


At a language level, it is technically not very feasible, since D has to 
be able to work with conservative garbage collectors, as well as other 
automatic memory management systems. In Python, it happens to be well 
implementable because memory is managed by reference- counting garbage 
collector. But for statically compiled languages, reference-counting has 
proven to be a major slowdown, expecially when it has to be thread-safe.

There should be other ways to implement caches and such in D. Normally 
you would have something like a "central repository" for objects of some 
sort, and you "request" them and "put back", basically doing sort-of 
manual reference counting, which is though considerably much less 
overhead than automatic one. There might also be some other ways, e.g. 
by mangling pointers so that garbage collector doesn't find them - but 
then the object must maintain a complete list of its users. A library 
implementation is though definately more tedious to use, would have 
another advantage performance-wise: it may use a smarter clean-up 
strategy than a garbage collector and work asynchronously compared to 
it, thus reducing pauses.

-eye
May 13 2004
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Ilya Minkov wrote:

<snip>

 But for statically compiled languages, reference-counting has 
 proven to be a major slowdown, expecially when it has to be thread-safe.


And a cause of memory leaks, when you try to use circular data structures.


 There should be other ways to implement caches and such in D. Normally 
 you would have something like a "central repository" for objects of some 
 sort, and you "request" them and "put back", basically doing sort-of 
 manual reference counting, which is though considerably much less 
 overhead than automatic one. There might also be some other ways, e.g. 
 by mangling pointers so that garbage collector doesn't find them - but 
 then the object must maintain a complete list of its users.


<snip>

I thought of writing a class that would do something like that.  The 
trouble is that we'd need a means of determining if an object is 
reachable by strong references.  The only way we can really get near 
doing this is to have a common base class for weak-referenceable 
objects, which would clear the references in its destructor.  A 
compromise would be to create an object wrapper enabling something to be 
WR'd, but that might lead to WRs being cleared while the object is still 
active.

But with the base class proviso, a possibility is to assign a serial 
number to each object when it is first WR'd, and have the WR object hold 
this number.  There would be a static mapping table of some sort between
serial numbers and mangled pointers.  When it's time to clear the WRs, 
it would simply set the pointer in the table to null.  The only time 
you're going to run into trouble is if you WR more objects in the course 
of your program than the type of the serial number'll allow.

Stewart.

-- 
My e-mail is valid but not my primary mailbox, aside from its being the 
unfortunate victim of intensive mail-bombing at the moment.  Please keep 
replies on the 'group where everyone may benefit.
May 13 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
In article <c80726$1qq$1 digitaldaemon.com>, Stewart Gordon says...
..

I thought of writing a class that would do something like that.  The 
trouble is that we'd need a means of determining if an object is 
reachable by strong references.  The only way we can really get near 
doing this is to have a common base class for weak-referenceable 
objects, which would clear the references in its destructor.  A 
compromise would be to create an object wrapper enabling something to be 
WR'd, but that might lead to WRs being cleared while the object is still 
active.


The boehm-wheeler gc (and probably most others) has a provision to add
non-scanned areas.  (In D, malloced space is non-scanned, right?)  If you add
one of these areas (needs some language support) and put an array of weak
pointers there, then the only piece missing is a way for the GC to clear
pointers (from that area) for objects that it collects.

So: Weak refs are a tiny 'smart pointer' like object that points at a slot in
the table (or indexes into the unscanned space).  That slot has a non-scanned
pointer to the real object.

When the GC is done reaping the heap, it does one last mark scan: it runs over
the weak ref table:  instead of marking objects as "not garbage", it just zeroes
the weak pointers to them.  If the object is already marked as "not garbage" it
leaves the pointer alone (there must be a strong ref).

- Extra accesses are needed to use weak refs (its a pointer to a pointer).

- This is okay, because weak pointers are rarely used except for this kind
of caching.  Once you fetch the regular pointer, its as fast as any other.

(Side note: The standard application for weak references (caching) seems a bit
"small" to me.  It seems like the number of objects kept in-cache should be
performance tuned, not cleared every time the GC gets hungry, right?  You can
keep a fixed number of regular pointers, plus the weak set; but the weak set
only helps for cache elements that are past the end of this initial size.  The
non-weak portion of the cache is probably as large as you can afford to begin
with, so..)

Kevin
May 13 2004
parent Ben Hinkle <bhinkle4 juno.com> writes:
 (Side note: The standard application for weak references (caching) seems a
 bit
 "small" to me.  It seems like the number of objects kept in-cache should
 be
 performance tuned, not cleared every time the GC gets hungry, right?  You
 can keep a fixed number of regular pointers, plus the weak set; but the
 weak set
 only helps for cache elements that are past the end of this initial size. 
 The non-weak portion of the cache is probably as large as you can afford
 to begin with, so..)


I agree - I've only seen weak references used to manage caches. It would be
neat to have GC callbacks where an object can register a delegate that runs
when the GC is about to full-collect. That way a cache can clear itself out
(or clear out the extra elements of the cache) without needing weak
references. I guess you'd have to be careful not to leak the cache since
the GC would then have a reference to it.

-Ben
May 14 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...

 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.


Weak references always seemed to me to be a massive and brittle kludge to
deal with memory management issues ill-suited to garbage collection. Since D
allows explicit memory management as an alternative for gc, instead of weak
references for gc, use explicit memory management for those references.

So, weak references are not planned for D. I think they are only required
for languages that offer no alternative to gc.
May 14 2004
parent reply Brian Hammond <d at brianhammond dot com> <Brian_member pathlink.com> writes:
Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...

"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...

 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.


Weak references always seemed to me to be a massive and brittle kludge to
deal with memory management issues ill-suited to garbage collection. Since D
allows explicit memory management as an alternative for gc, instead of weak
references for gc, use explicit memory management for those references.

So, weak references are not planned for D. I think they are only required
for languages that offer no alternative to gc.
May 16 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
I would think you can get essentially the same effect in D via destructors:

class Cache {
..
delete(int key) {...}
};

class GCwatch {
int key;
Cache cache;

this(int k, Cache c)
{
key = k;
cache = c;
}

~this()
{
Cache.delete(key);
}
};

add_weak_elem()
{
Cache c = new Cache;
int k = c.get_weak_elem();

new GCwatch(c,k);
..
}

The GCwatch is not held by a strong pointer, so the GC will delete it, causing
Cache cleanups to happen at the same time as GC cleanups.

Kevin

In article <c896ua$quh$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...

Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...

"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...

 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.


Weak references always seemed to me to be a massive and brittle kludge to
deal with memory management issues ill-suited to garbage collection. Since D
allows explicit memory management as an alternative for gc, instead of weak
references for gc, use explicit memory management for those references.

So, weak references are not planned for D. I think they are only required
for languages that offer no alternative to gc.
May 17 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
I should amend this: the cached object needs a pointer to the GCwatch, to
prevent it from being deleted when a strong pointer exists, and the Cache needs
to disguise the pointer to the element.  Also the method should not be called
"delete".

Kevin

In article <c8ambr$2lq$1 digitaldaemon.com>, Kevin Bealer says...

I would think you can get essentially the same effect in D via destructors:

class Cache {
..
delete(int key) {...}
};

class GCwatch {
int key;
Cache cache;

this(int k, Cache c)
{
key = k;
cache = c;
}

~this()
{
Cache.delete(key);
}
};

add_weak_elem()
{
Cache c = new Cache;
int k = c.get_weak_elem();

new GCwatch(c,k);
..
}

The GCwatch is not held by a strong pointer, so the GC will delete it, causing
Cache cleanups to happen at the same time as GC cleanups.

Kevin

In article <c896ua$quh$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...

Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...

"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...

 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.


Weak references always seemed to me to be a massive and brittle kludge to
deal with memory management issues ill-suited to garbage collection. Since D
allows explicit memory management as an alternative for gc, instead of weak
references for gc, use explicit memory management for those references.

So, weak references are not planned for D. I think they are only required
for languages that offer no alternative to gc.
May 17 2004
parent reply Brian Hammond <d at brianhammond dot com> <Brian_member pathlink.com> writes:
Your idea is fine it's just manual.  I was mostly curious if my issue could be
solved using weakrefs in D.  As has been pointed out, weakrefs don't apply well
to D.  Thus, to implement this, the monitored object must notify interested
parties (say a cache) that it's been collected (notification is done in the
destructor). 

In article <c8ampu$3c0$1 digitaldaemon.com>, Kevin Bealer says...

I should amend this: the cached object needs a pointer to the GCwatch, to
prevent it from being deleted when a strong pointer exists, and the Cache needs
to disguise the pointer to the element.  Also the method should not be called
"delete".

Kevin


Keeping an explicit pointer won't keep the object alive if there are no
references to it and the GC runs.  E.g.

import std.gc;

class A
{
this()  { printf("A.this() called.\n");  }
~this() { printf("A.~this() called.\n"); }
}

int main(char[][] args)
{
printf("creating a and pointing b at a\n");
A a = new A();
A* b = &a;
printf("b's target object is %.*s\n", *b ? "not null" : "null");

printf("Nullifying a...\n");
a = null;
printf("b's target object is %.*s\n", *b ? "not null" : "null");

printf("Running gc...\n");
std.gc.fullCollect();
printf("A has %.*s collected\n", !a ? "been" : "not been");
printf("b's target object is %.*s\n", *b ? "not null" : "null");

printf("end.\n");
return 0;
}

output:

creating a and pointing b at a
A.this() called.
b's target object is not null
Nullifying a...
b's target object is null
Running gc...
A.~this() called.
A has been collected
b's target object is null
end.

Thanks! Brian



In article <c8ambr$2lq$1 digitaldaemon.com>, Kevin Bealer says...

I would think you can get essentially the same effect in D via destructors:

class Cache {
..
delete(int key) {...}
};

class GCwatch {
int key;
Cache cache;

this(int k, Cache c)
{
key = k;
cache = c;
}

~this()
{
Cache.delete(key);
}
};

add_weak_elem()
{
Cache c = new Cache;
int k = c.get_weak_elem();

new GCwatch(c,k);
..
}

The GCwatch is not held by a strong pointer, so the GC will delete it, causing
Cache cleanups to happen at the same time as GC cleanups.

Kevin

In article <c896ua$quh$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...

Thanks for the response.  I was thinking of weakrefs as an exploitable feature
for a resource manager I am writing for a game engine in D.  

Weak refs seemed like a good fit for this problem.  The resource manager
maintains a cache of weak references to resource instances.  The manager knows
they exist but doesn't stop the gc of such from happening by holding (strong)
references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1 digitaldaemon.com>, Walter says...

"Brian Hammond" <d brianhammond.com> wrote in message
news:c7grjd$1fq9$1 digitaldaemon.com...

 Does (or will) D support weak references?

 See http://www.python.org/doc/current/lib/module-weakref.html for a
 backgrounder.


Weak references always seemed to me to be a massive and brittle kludge to
deal with memory management issues ill-suited to garbage collection. Since D
allows explicit memory management as an alternative for gc, instead of weak
references for gc, use explicit memory management for those references.

So, weak references are not planned for D. I think they are only required
for languages that offer no alternative to gc.
May 17 2004
parent Kevin Bealer <Kevin_member pathlink.com> writes:
In article <c8b8mj$v82$1 digitaldaemon.com>, Brian Hammond <d at brianhammond
dot com> says...

Your idea is fine it's just manual.  I was mostly curious if my issue could be
solved using weakrefs in D.  As has been pointed out, weakrefs don't apply well
to D.  Thus, to implement this, the monitored object must notify interested
parties (say a cache) that it's been collected (notification is done in the
destructor). 


My original thought was to use this technique to create a generic "weak pointer"
object.  But I have had a realization w.r.t finalization: weak pointers won't
work properly from a "gc-dead" object because the finalizer cannot legally
fiddle with sub-objects: they may also be garbage.

Kevin
May 17 2004