www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Manual Deletion from Destructor

reply dsimcha <dsimcha yahoo.com> writes:
I sometimes run into false pointer issues when using very large data
structures in D.  Often, these data structures are owned by a single class
instance and do not escape.  In these cases, is it safe to do something like:

class Foo {
    // Allocated on GC heap.
    private HugeDataStructure hugeDataStructure;

    ~this() {
        // hugeDataStructure _should_ be GC'd when the Foo instance
        // is GC'd because hugeDataStructure is guaranteed never
        // to escape.  It may not be b/c
        // of false pointer issues.  Delete it manually when instance
        // of Foo that owns it is GC'd.

        delete hugeDataStructure;
    }
}

The point is that the destructor for instances of Foo is called by the GC, not
manually.   The GC may have already realized that hugeDataStructure is not
reachable.  Does this make the delete statement in the d'tor unsafe, or is it
still ok?

Note:  You can assume that hugeDataStructure doesn't have its own destructor,
so delete just frees memory.
Mar 14 2009
next sibling parent Chad J <gamerchad __spam.is.bad__gmail.com> writes:
dsimcha wrote:

 I sometimes run into false pointer issues when using very large data
 structures in D.  Often, these data structures are owned by a single class
 instance and do not escape.  In these cases, is it safe to do something like:
 
 class Foo {
     // Allocated on GC heap.
     private HugeDataStructure hugeDataStructure;
 
     ~this() {
         // hugeDataStructure _should_ be GC'd when the Foo instance
         // is GC'd because hugeDataStructure is guaranteed never
         // to escape.  It may not be b/c
         // of false pointer issues.  Delete it manually when instance
         // of Foo that owns it is GC'd.
 
         delete hugeDataStructure;
     }
 }
 
 The point is that the destructor for instances of Foo is called by the GC, not
 manually.   The GC may have already realized that hugeDataStructure is not
 reachable.  Does this make the delete statement in the d'tor unsafe, or is it
 still ok?
 
 Note:  You can assume that hugeDataStructure doesn't have its own destructor,
 so delete just frees memory.


http://www.digitalmars.com/d/2.0/class.html#Destructor
http://www.digitalmars.com/d/1.0/class.html#Destructor
both say:

"The garbage collector is not guaranteed to run the destructor for all
unreferenced objects. Furthermore, the order in which the garbage
collector calls destructors for unreference objects is not specified.
This means that when the garbage collector calls a destructor for an
object of a class that has members that are references to garbage
collected objects, those references may no longer be valid. This means
that destructors cannot reference sub objects. This rule does not apply
to auto objects or objects deleted with the DeleteExpression, as the
destructor is not being run by the garbage collector, meaning all
references are valid."

I think you are violating that rule.
Mar 14 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Sat, Mar 14, 2009 at 11:53 AM, dsimcha <dsimcha yahoo.com> wrote:


 The point is that the destructor for instances of Foo is called by the GC=


, not

 manually. =A0 The GC may have already realized that hugeDataStructure is =


not

 reachable. =A0Does this make the delete statement in the d'tor unsafe, or=


 is it

 still ok?


Accessing or deleting any GC-managed memory in a destructor is unsafe,
Mar 14 2009
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
dsimcha wrote:

 I sometimes run into false pointer issues when using very large data
 structures in D.  Often, these data structures are owned by a single class
 instance and do not escape.  In these cases, is it safe to do something like:
 
 class Foo {
     // Allocated on GC heap.
     private HugeDataStructure hugeDataStructure;
 
     ~this() {
         // hugeDataStructure _should_ be GC'd when the Foo instance
         // is GC'd because hugeDataStructure is guaranteed never
         // to escape.  It may not be b/c
         // of false pointer issues.  Delete it manually when instance
         // of Foo that owns it is GC'd.
 
         delete hugeDataStructure;
     }
 }
 
 The point is that the destructor for instances of Foo is called by the GC, not
 manually.   The GC may have already realized that hugeDataStructure is not
 reachable.  Does this make the delete statement in the d'tor unsafe, or is it
 still ok?
 
 Note:  You can assume that hugeDataStructure doesn't have its own destructor,
 so delete just frees memory.


You can't call delete against a struct object, so the above wouldn't 
compile. What may solve your problem is calling GC.hasNoPointers against 
the block of memory in which hugeDataStructure lives. But before that... 
isn't the current GC non-conservative for heap-allocated objects? I 
thought it's only conservative for stack objects.

Andrei
Mar 14 2009
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article

 dsimcha wrote:

 I sometimes run into false pointer issues when using very large data
 structures in D.  Often, these data structures are owned by a single class
 instance and do not escape.  In these cases, is it safe to do something like:

 class Foo {
     // Allocated on GC heap.
     private HugeDataStructure hugeDataStructure;

     ~this() {
         // hugeDataStructure _should_ be GC'd when the Foo instance
         // is GC'd because hugeDataStructure is guaranteed never
         // to escape.  It may not be b/c
         // of false pointer issues.  Delete it manually when instance
         // of Foo that owns it is GC'd.

         delete hugeDataStructure;
     }
 }

 The point is that the destructor for instances of Foo is called by the GC, not
 manually.   The GC may have already realized that hugeDataStructure is not
 reachable.  Does this make the delete statement in the d'tor unsafe, or is it
 still ok?

 Note:  You can assume that hugeDataStructure doesn't have its own destructor,
 so delete just frees memory.


 You can't call delete against a struct object, so the above wouldn't
 compile. What may solve your problem is calling GC.hasNoPointers against
 the block of memory in which hugeDataStructure lives. But before that...
 isn't the current GC non-conservative for heap-allocated objects? I
 thought it's only conservative for stack objects.
 Andrei


No, the problem is that there may be things that look like pointers pointing to
internal regions of hugeDataStructure.  For example, let's say hugeDataStructure
is a 100-megabyte array.  It's a pretty big target for false pointers, so even
though the only legitimate reference is from the instance of Foo that owns it,
hugeDataStructure might never get GC'd.  In my specific case, hugeDataStructure
is
a large associative array.  A few nodes don't get freed properly, leading to
heap
fragmentation and massive memory usage.  (I've created my own AA delete
function,
which I know works when not used from a destructor like the above example, see
Bugzilla 2105).
Mar 14 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
dsimcha wrote:

 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article

 You can't call delete against a struct object, so the above wouldn't
 compile. What may solve your problem is calling GC.hasNoPointers against
 the block of memory in which hugeDataStructure lives. But before that...
 isn't the current GC non-conservative for heap-allocated objects? I
 thought it's only conservative for stack objects.
 Andrei


 
 No, the problem is that there may be things that look like pointers pointing to
 internal regions of hugeDataStructure.  For example, let's say
hugeDataStructure
 is a 100-megabyte array.  It's a pretty big target for false pointers, so even
 though the only legitimate reference is from the instance of Foo that owns it,
 hugeDataStructure might never get GC'd.  In my specific case,
hugeDataStructure is
 a large associative array.  A few nodes don't get freed properly, leading to
heap
 fragmentation and massive memory usage.  (I've created my own AA delete
function,
 which I know works when not used from a destructor like the above example, see
 Bugzilla 2105).


Thanks for the info, that's very interesting.

Andrei
Mar 14 2009
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Andrei Alexandrescu wrote:

 
 You can't call delete against a struct object, so the above wouldn't 
 compile. What may solve your problem is calling GC.hasNoPointers against 
 the block of memory in which hugeDataStructure lives. But before that... 
 isn't the current GC non-conservative for heap-allocated objects? I 
 thought it's only conservative for stack objects.


It's conservative for everything, unless you count marking an entire 
block as "has pointers" non-conservative.  The GC currently doesn't know 
where in a block the pointers are, so it treats all such labeled blocks 
conservatively.
Mar 14 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Sean Kelly wrote:

 Andrei Alexandrescu wrote:

 You can't call delete against a struct object, so the above wouldn't 
 compile. What may solve your problem is calling GC.hasNoPointers 
 against the block of memory in which hugeDataStructure lives. But 
 before that... isn't the current GC non-conservative for 
 heap-allocated objects? I thought it's only conservative for stack 
 objects.


 
 It's conservative for everything, unless you count marking an entire 
 block as "has pointers" non-conservative.  The GC currently doesn't know 
 where in a block the pointers are, so it treats all such labeled blocks 
 conservatively.


Ouch. Yet another place where introspection could help lots.

Any statistics about the amount of slack memory due to false pointers 
would be appreciated.


Andrei
Mar 14 2009
parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s

 Any statistics about the amount of slack memory due to false pointers
 would be appreciated.
 Andrei


This is a corner case, but run the following code:

import std.stdio, core.memory;

void main() {
    foreach(count; 1..uint.max) {
        test;
        GC.collect;  // Make it explicit to make absolutely sure GC is running.
        writeln(count);
    }
}

// Marked as pure to emphasize that it's not escaping someArray to anywhere,
// so when test() goes out of scope, someArray is safe to GC.
void test() pure {
    auto someArray = new uint[15_000_000];
}

Disclaimer:  My understanding is that this may work differently on Linux, but if
you run it on Win32, it runs out of memory after about 20 iterations.

I don't know about the average, but the worst cases are occasionally bad enough
to
make me eventually run out of 32-bit address space on stuff that should only be
using ~150 megs of RAM, unless I manually delete a few large data structures.
Associative arrays seem to be the worst problem because:

1.  They generate a lot of false pointers.
2.  They can often be large.
3.  If even a few nodes don't get freed properly, they can fragment the heap
severely.
Mar 14 2009
prev sibling parent reply BCS <none anon.com> writes:
Hello dsimcha,


 I sometimes run into false pointer issues when using very large data
 structures in D.  Often, these data structures are owned by a single
 class instance and do not escape.  In these cases, is it safe to do
 something like:
 


If you can be shure ~this will get called, you might switch to malloc.
Mar 14 2009
parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
BCS wrote:

 Hello dsimcha,
 

 I sometimes run into false pointer issues when using very large data
 structures in D.  Often, these data structures are owned by a single
 class instance and do not escape.  In these cases, is it safe to do
 something like:


 
 If you can be shure ~this will get called, you might switch to malloc.


If you do this, don't forget to register and unregister the memory with 
the GC if it contains any pointers into GC-allocated data. Otherwise, 
the GC won't see those pointers and may delete that data if there aren't 
any other references to it.
Mar 14 2009