www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Don't expect class destructors to be called at all by the GC

reply Mike Franklin <slavo5150 yahoo.com> writes:
"Don't expect class destructors to be called at all by the GC"

I was a bit shocked to read that here: 
https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors

The document tries to clarify with:
"The garbage collector is not guaranteed to run the destructors 
for all unreferenced objects."

Unfortunately, that doesn't really shed much light on this 
oddity.  So, specifically, under what circumstances are 
destructors not called?

Thanks,
Mike
Dec 20 2017
next sibling parent reply user1234 <user1234 12.fr> writes:
On Thursday, 21 December 2017 at 02:57:00 UTC, Mike Franklin 
wrote:
 "Don't expect class destructors to be called at all by the GC"

 I was a bit shocked to read that here: 
 https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors

 The document tries to clarify with:
 "The garbage collector is not guaranteed to run the destructors 
 for all unreferenced objects."

 Unfortunately, that doesn't really shed much light on this 
 oddity.  So, specifically, under what circumstances are 
 destructors not called?

 Thanks,
 Mike
When the GC is unaware of a class instance (an "unreferenced object") it won't call its destructor. This happens when you use alternative memory management strategy, for example using Mallocator and make you get an unreferenced object that you have to manage yourself.
Dec 20 2017
parent reply Mike Parker <aldacron gmail.com> writes:
On Thursday, 21 December 2017 at 04:10:56 UTC, user1234 wrote:
 On Thursday, 21 December 2017 at 02:57:00 UTC, Mike Franklin
 Unfortunately, that doesn't really shed much light on this 
 oddity.  So, specifically, under what circumstances are 
 destructors not called?
 When the GC is unaware of a class instance (an "unreferenced 
 object") it won't call its destructor. This happens when you 
 use alternative memory management strategy, for example using 
 Mallocator and make you get an unreferenced object that you 
 have to manage yourself.
The root of the problem is that in D, class destruction and finalization are conflated. It would be much more accurate to refer to ~this in classes as a finalizer. Then this sort of confusion wouldn't be so widespread. Also, consider the current GC implementation finalizes any objects that haven't yet been finalized when it terminates. It terminates during runtime termination, but *after* static destructors are executed (which is how it should be). We already know that you can't rely on any GC memory references to be valid in a class destructor, but because of this cleanup phase, you also can't rely on any program state still being valid. As an example, each of the Derelict packages used to (pointlessly) unload its shared library in a static destructor, but people repeatedly had segfaults at app exit because their class destructors were calling into the loaded libraries to release resources. The solution there was easy -- stop manually unloading the libraries and let the OS do it at process termination. But anyone who wants to unload any resources in a class destructor needs to be aware of this issue in case a static destructor somewhere is getting to it first. And I still see code using Derelict where people unload the library themselves in a static destructor. I just don't even bother with class destructors. Without a guarantee that they can run and without any sort of deterministic behavior, it's really not appropriate to refer to them as destructors and they're about as useful as Java finalizers, which means not at all. In order to make them less error prone, we need to separate the concept of destruction from finalization and allow both destructors and finalizers. That's what I've taken to doing manually, by implementing a `terminate` function in my classes that I either call directly or via a ref-counted templated struct called Terminator.
Dec 20 2017
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Thursday, 21 December 2017 at 06:50:44 UTC, Mike Parker wrote:
 That's what I've taken to doing manually, by implementing a 
 `terminate` function in my classes that I either call directly 
 or via a ref-counted templated struct called Terminator.
I feel like I'm rambling but.. The problem with that approach is that you can't reuse Unique, RefCounted, scoped!T because they rely on .destroy
Dec 21 2017
parent Mike Parker <aldacron gmail.com> writes:
On Thursday, 21 December 2017 at 14:26:55 UTC, Guillaume Piolat 
wrote:
 On Thursday, 21 December 2017 at 06:50:44 UTC, Mike Parker 
 wrote:
 That's what I've taken to doing manually, by implementing a 
 `terminate` function in my classes that I either call directly 
 or via a ref-counted templated struct called Terminator.
I feel like I'm rambling but.. The problem with that approach is that you can't reuse Unique, RefCounted, scoped!T because they rely on .destroy
I'm not proposing it as a general solution. It's easy to implement and it works for my use case, so it's one possible solution.
Dec 21 2017
prev sibling next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Thursday, 21 December 2017 at 06:50:44 UTC, Mike Parker wrote:
 On Thursday, 21 December 2017 at 04:10:56 UTC, user1234 wrote:
 [...]
[...]
 [...]
The root of the problem is that in D, class destruction and finalization are conflated. It would be much more accurate to refer to ~this in classes as a finalizer. Then this sort of confusion wouldn't be so widespread. [...]
Have you considered writing a DIP on this?
Dec 21 2017
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 21, 2017 at 06:50:44AM +0000, Mike Parker via Digitalmars-d-learn
wrote:
[...]
 I just don't even bother with class destructors. Without a guarantee
 that they can run and without any sort of deterministic behavior, it's
 really not appropriate to refer to them as destructors and they're
 about as useful as Java finalizers, which means not at all. In order
 to make them less error prone, we need to separate the concept of
 destruction from finalization and allow both destructors and
 finalizers. That's what I've taken to doing manually, by implementing
 a `terminate` function in my classes that I either call directly or
 via a ref-counted templated struct called Terminator.
I recently ran into this problem while using Adam Ruppe's lightweight SQLite binding (arsd/sqlite.d). Originally, I kept an open database handle (which is a class instance) throughout the lifetime of the program; in this case, I could just use a scoped reference and it would ensure the DB is closed when the handle went out of scope, just what I want. But as my code developed, I began to need to cache multiple DB handles for performance, and scope no longer helps me there. At first I thought, no problem, the GC would handle this for me. Right? Wrong. Even calling GC.collect directly did not guarantee the DB handle was closed at the right time. This may have been a bug in my code that left dangling references to it, or perhaps the array of Database handles was still scanned through by the GC even though the only remaining array slice has a shorter length. Whatever the reason was, it left me with the very unpleasant prospect of silently accumulating file descriptor leaks. I ended up calling .destroy on the class instance explicitly just so the destructor would run at the right time, right before nulling the reference so that the GC would collect the memory. This makes using classes in D an even dimmer prospect than it already generally is (nowadays, and I don't seem to be the only one, I prefer to just use structs and templates instead of runtime polymorphism, where possible). When the scoped destruction of structs isn't an option, RefCounted!T seems to be a less evil alternative than an unreliable class dtor. :-/ T -- MACINTOSH: Most Applications Crash, If Not, The Operating System Hangs
Dec 21 2017
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 21 December 2017 at 18:20:19 UTC, H. S. Teoh wrote:
 When the scoped destruction of structs isn't an option, 
 RefCounted!T seems to be a less evil alternative than an 
 unreliable class dtor. :-/
Alas, RefCounted doesn't work well with inheritance... Though, what you could do is make the refcounted owners and borrow the actual reference later.
Dec 21 2017
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 21, 2017 at 06:45:27PM +0000, Adam D. Ruppe via Digitalmars-d-learn
wrote:
 On Thursday, 21 December 2017 at 18:20:19 UTC, H. S. Teoh wrote:
 When the scoped destruction of structs isn't an option, RefCounted!T
 seems to be a less evil alternative than an unreliable class dtor.
 :-/
Alas, RefCounted doesn't work well with inheritance...
Oh? What's the issue?
 Though, what you could do is make the refcounted owners and borrow the
 actual reference later.
Yeah, I figured that I pretty much have to use a proxy struct with RefCounted, and have the struct dtor do the actual cleanup of the class reference, something like: class Resource { void cleanup(); // inheritable } struct Proxy { private Resource res; this(Resource _res) { res = _res; } ~this() { res.cleanup(); } } ... auto obj = RefCounted!Proxy(allocateResource()); T -- The problem with the world is that everybody else is stupid.
Dec 21 2017
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 21 December 2017 at 18:48:38 UTC, H. S. Teoh wrote:
 Alas, RefCounted doesn't work well with inheritance...
Oh? What's the issue?
Implicit casts don't work so you can't pass a RefCounted!Class as RefCounted!Interface except in simple cases using alias this tricks.
Dec 21 2017
prev sibling next sibling parent reply Mengu <mengukagan gmail.com> writes:
On Thursday, 21 December 2017 at 18:45:27 UTC, Adam D. Ruppe 
wrote:
 On Thursday, 21 December 2017 at 18:20:19 UTC, H. S. Teoh wrote:
 When the scoped destruction of structs isn't an option, 
 RefCounted!T seems to be a less evil alternative than an 
 unreliable class dtor. :-/
Alas, RefCounted doesn't work well with inheritance... Though, what you could do is make the refcounted owners and borrow the actual reference later.
i really wonder how Objective-C and Swift is pulling this off.
Dec 22 2017
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 22 December 2017 at 23:34:55 UTC, Mengu wrote:
 i really wonder how Objective-C and Swift is pulling this off.
It isn't a fundamental problem, D just can't express it in the existing language (heck, even D, as defined, could do it, the implementation just isn't there.)
Dec 22 2017
prev sibling parent reply Bienlein <jeti789 web.de> writes:
On Thursday, 21 December 2017 at 18:45:27 UTC, Adam D. Ruppe 
wrote:
 On Thursday, 21 December 2017 at 18:20:19 UTC, H. S. Teoh wrote:
 When the scoped destruction of structs isn't an option, 
 RefCounted!T seems to be a less evil alternative than an 
 unreliable class dtor. :-/
Alas, RefCounted doesn't work well with inheritance... Though, what you could do is make the refcounted owners and borrow the actual reference later.
Is there some summary of the things you have to be aware of when using the GC in D and not using the GC? I feel this would be very useful especially for people that are new to D or are not used to that kind of issues (because coming from a GCed language).
Jan 31 2018
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 31, 2018 at 01:38:07PM +0000, Bienlein via Digitalmars-d-learn
wrote:
 On Thursday, 21 December 2017 at 18:45:27 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 December 2017 at 18:20:19 UTC, H. S. Teoh wrote:
 When the scoped destruction of structs isn't an option,
 RefCounted!T seems to be a less evil alternative than an
 unreliable class dtor.  :-/
Alas, RefCounted doesn't work well with inheritance... Though, what you could do is make the refcounted owners and borrow the actual reference later.
Is there some summary of the things you have to be aware of when using the GC in D and not using the GC? I feel this would be very useful especially for people that are new to D or are not used to that kind of issues (because coming from a GCed language).
D was originally designed with a GC in mind, so if you're using the GC and writing typical GC'd code, there's really not much to be aware of, esp. if you're coming from a GC'd language. The only time you have to be actively concerned about the GC is if: 1) You're interacting with C code and passing pointers around. You may need to hold on to a copy of pointers you pass to C, because the GC does not know any pointer roots in C land, so it may wrongly collect something that the C code still holds a reference to. 2) You're writing performance-sensitive code and profiling shows that the GC is becoming a bottleneck. The usual easy solution is to import core.memory and make use of GC.disable, GC.collect, GC.enable at strategic points in your code. 3) You need to do useful work in dtors, and/or need deterministic destruction, which is fundamentally incompatible with the GC. In this case, probably your best bet is RefCounted or some other kind of non-GC memory management scheme. Lately I'm beginning more and more to just avoid class dtors in general, due to their inherent unreliability, and just use scoped destruction (structs with dtors), or RefCounted, or some other memory management scheme (perhaps raw malloc/free if I need tight control over things). T -- VI = Visual Irritation
Feb 01 2018
prev sibling next sibling parent Neia Neutuladh <neia ikeran.org> writes:
On Thursday, 21 December 2017 at 18:20:19 UTC, H. S. Teoh wrote:
 Even calling GC.collect directly did not guarantee the DB 
 handle was closed at the right time.  This may have been a bug 
 in my code that left dangling references to it, or perhaps the 
 array of Database handles was still scanned through by the GC 
 even though the only remaining array slice has a shorter 
 length. Whatever the reason was, it left me with the very 
 unpleasant prospect of silently accumulating file descriptor 
 leaks.
Last I checked, the GC doesn't understand arrays. It only understands "segment of memory that might contain pointers" and "segment of memory that doesn't contain pointers". You might have gotten better results if you had nulled out the reference in the array. Of course, that relies on not having any remaining references on the stack or in registers, neither of which is easy to guarantee.
Dec 21 2017
prev sibling parent reply DanielG <simpletangent gmail.com> writes:
On Thursday, 21 December 2017 at 18:20:19 UTC, H. S. Teoh wrote:
 I ended up calling .destroy on the class instance explicitly 
 just so the destructor would run at the right time, right 
 before nulling the reference so that the GC would collect the 
 memory.
Pardon my probable ignorance (D newbie and all), but why wouldn't a 'delete' work for this? https://dlang.org/spec/expression.html#delete_expressions
Jan 31 2018
parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 31 January 2018 at 10:14:53 UTC, DanielG wrote:

 Pardon my probable ignorance (D newbie and all), but why 
 wouldn't a 'delete' work for this?

 https://dlang.org/spec/expression.html#delete_expressions
delete is deprecated: https://dlang.org/deprecate.html#delete
Jan 31 2018
parent reply DanielG <simpletangent gmail.com> writes:
On Wednesday, 31 January 2018 at 10:34:53 UTC, Mike Parker wrote:
 delete is deprecated:

 https://dlang.org/deprecate.html#delete
Ah, thanks! Actually double-thanks, because my progress through your book is what prompted me to search for threads about class destructors. The existence of .destroy answers my question (namely, "should I just use 'delete', or my own .dispose method, for deterministic resource freeing?")
Jan 31 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, January 31, 2018 10:51:10 DanielG via Digitalmars-d-learn 
wrote:
 On Wednesday, 31 January 2018 at 10:34:53 UTC, Mike Parker wrote:
 delete is deprecated:

 https://dlang.org/deprecate.html#delete
Ah, thanks! Actually double-thanks, because my progress through your book is what prompted me to search for threads about class destructors. The existence of .destroy answers my question (namely, "should I just use 'delete', or my own .dispose method, for deterministic resource freeing?")
The main problem with delete is that it's inherently unsafe. GC-managed memory is supposed to be safe (it's one of the main reasons that D has a GC in the first place), but having the programmer go and delete a GC-managed object rather than waiting for the GC to do it makes it trivial to do wrong stuff like free an object's memory while it's still referenced by something else (the sort of thing that the GC is supposed to avoid). It's far better to either explicitly destroy the object without freeing its memory or to use memory that is not managed by the GC if you want deterministic destruction of an object on the heap. - Jonathan M Davis
Jan 31 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/20/17 9:57 PM, Mike Franklin wrote:
 "Don't expect class destructors to be called at all by the GC"
 
 I was a bit shocked to read that here: 
 https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors
 
 The document tries to clarify with:
 "The garbage collector is not guaranteed to run the destructors for all 
 unreferenced objects."
 
 Unfortunately, that doesn't really shed much light on this oddity.  So, 
 specifically, under what circumstances are destructors not called?
It's implementation defined :) The gist is, you cannot expect that destructors will be run in a timely manner, or at all. They may be called, and most of the time they are. But the language nor the current implementation makes a guarantee that they will be called. For this reason, any classes that use non-memory resources should clean up those resources before becoming garbage. This is why most of the time, such items are managed by structs. Note that the same non-guarantee exists in other GC'd languages, such as -Steve
Dec 21 2017
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 21 December 2017 at 19:43:16 UTC, Steven 
Schveighoffer wrote:

 The gist is, you cannot expect that destructors will be run in 
 a timely manner, or at all.

 They may be called, and most of the time they are. But the 
 language nor the current implementation makes a guarantee that 
 they will be called.
I understand that we can't deterministically predict when a destructor will be called, but if we can't deterministically predict if a destructor will be called, that seems asinine. What condition(s) would cause a destructor for an object that is managed by the GC to potentially not be called? Thanks, Mike
Dec 21 2017
next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Friday, 22 December 2017 at 00:09:31 UTC, Mike Franklin wrote:

 What condition(s) would cause a destructor for an object that 
 is managed by the GC to potentially not be called?
Here: =========== import std.stdio; class Clazz { ~this() { writeln("Class dest"); } } void makeClazz() { auto clazz = new Clazz; } void main() { makeClazz(); } static ~this() { writeln("Static dest"); } ============ This outputs: ============ Static dest Class dest ============ The class destructor is not run during the lifetime of the program. The fact that it's run during runtime termination is an implementation detail. Another implementation might not run a finalization at termination. So the destructors (finalizers) are only run when an object is collected. No collection, no destructor call.
Dec 21 2017
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, December 22, 2017 01:23:26 Mike Parker via Digitalmars-d-learn 
wrote:
 The class destructor is not run during the lifetime of the
 program. The fact that it's run during runtime termination is an
 implementation detail. Another implementation might not run a
 finalization at termination.

 So the destructors (finalizers) are only run when an object is
 collected. No collection, no destructor call.
IIRC, there were reasons why running the destructors/finalizers when unloading dynamically loaded libraries was a problem too and one that we were stuck with, but I could be remembering incorrectly. Regardless, even if it were the case that it were guaranteed that all finalizers were run when the program exited, it would still be terrible practice to rely on it. It's trivial to end up in a situation where no collection is run for quite some time (e.g. just don't do much memory allocation for a while), which would leave any resources that needed to be freed by a finalizer unfreed for quite a while even though they weren't needed anymore. So practically speaking, it doesn't really matter where the finalizers are guaranteed to run. Relying on them to be run rather than forcing them to be run via destroy or using some other helper function is just going to cause problems, so it's just plain bad practice to rely on finalizers to be run to release resources. That's just life with GC's in and why D code should either be using destroy to deal with freeing resources for a class or using structs on the stack for resources that need to be freed. Or alternate memory strategies can be used via std.experimental.allocator. The GC works great for lots of stuff but not for system resources. Honestly, in some ways, we'd be better off if D didn't even have finalizers. - Jonathan M Davis
Dec 24 2017
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Dec 24, 2017 at 02:07:26PM -0700, Jonathan M Davis via
Digitalmars-d-learn wrote:
[...]
 Regardless, even if it were the case that it were guaranteed that all
 finalizers were run when the program exited, it would still be
 terrible practice to rely on it. It's trivial to end up in a situation
 where no collection is run for quite some time (e.g. just don't do
 much memory allocation for a while), which would leave any resources
 that needed to be freed by a finalizer unfreed for quite a while even
 though they weren't needed anymore. So practically speaking, it
 doesn't really matter where the finalizers are guaranteed to run.
 Relying on them to be run rather than forcing them to be run via
 destroy or using some other helper function is just going to cause
 problems, so it's just plain bad practice to rely on finalizers to be
 run to release resources. That's just life with GC's in general, not

 D code should either be using destroy to deal with freeing resources
 for a class or using structs on the stack for resources that need to
 be freed. Or alternate memory strategies can be used via
 std.experimental.allocator. The GC works great for lots of stuff but
 not for system resources. Honestly, in some ways, we'd be better off
 if D didn't even have finalizers.
[...] This makes me wonder if a better approach to memory management is to use refcounting by default, and fallback to the GC to collect cycles. In my current project, I'll probably end up having to use RefCounted!Database just so I can have deterministic releasing of database handles without needing to worry about dangling references that may still be lingering around. (Currently, I'm just calling .destroy directly on the handle when I'm done with it, but there's a slim possibility that there might be dangling references left somewhere. So that needs to be addressed at some point.) T -- "I suspect the best way to deal with procrastination is to put off the procrastination itself until later. I've been meaning to try this, but haven't gotten around to it yet. " -- swr
Dec 28 2017
prev sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Friday, 22 December 2017 at 00:09:31 UTC, Mike Franklin wrote:
 What condition(s) would cause a destructor for an object that 
 is managed by the GC to potentially not be called?
Good question. It's true that barring an Error, they should be called by the GC at runtime termination.
Dec 22 2017
prev sibling parent bauss <jj_1337 live.dk> writes:
On Thursday, 21 December 2017 at 19:43:16 UTC, Steven 
Schveighoffer wrote:
 On 12/20/17 9:57 PM, Mike Franklin wrote:
 [...]
It's implementation defined :) The gist is, you cannot expect that destructors will be run in a timely manner, or at all. They may be called, and most of the time they are. But the language nor the current implementation makes a guarantee that they will be called. For this reason, any classes that use non-memory resources should clean up those resources before becoming garbage. This is why most of the time, such items are managed by structs. Note that the same non-guarantee exists in other GC'd -Steve
can actually be used to prevent this kind of stuff and generally used to clean up non-GC memory.
Dec 21 2017
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, December 28, 2017 10:37:09 H. S. Teoh via Digitalmars-d-learn 
wrote:
 On Sun, Dec 24, 2017 at 02:07:26PM -0700, Jonathan M Davis via
 Digitalmars-d-learn wrote: [...]

 Regardless, even if it were the case that it were guaranteed that all
 finalizers were run when the program exited, it would still be
 terrible practice to rely on it. It's trivial to end up in a situation
 where no collection is run for quite some time (e.g. just don't do
 much memory allocation for a while), which would leave any resources
 that needed to be freed by a finalizer unfreed for quite a while even
 though they weren't needed anymore. So practically speaking, it
 doesn't really matter where the finalizers are guaranteed to run.
 Relying on them to be run rather than forcing them to be run via
 destroy or using some other helper function is just going to cause
 problems, so it's just plain bad practice to rely on finalizers to be
 run to release resources. That's just life with GC's in general, not

 D code should either be using destroy to deal with freeing resources
 for a class or using structs on the stack for resources that need to
 be freed. Or alternate memory strategies can be used via
 std.experimental.allocator. The GC works great for lots of stuff but
 not for system resources. Honestly, in some ways, we'd be better off
 if D didn't even have finalizers.
[...] This makes me wonder if a better approach to memory management is to use refcounting by default, and fallback to the GC to collect cycles. In my current project, I'll probably end up having to use RefCounted!Database just so I can have deterministic releasing of database handles without needing to worry about dangling references that may still be lingering around. (Currently, I'm just calling .destroy directly on the handle when I'm done with it, but there's a slim possibility that there might be dangling references left somewhere. So that needs to be addressed at some point.)
The GC works perfectly well for most things. You just have to be aware of its downsides - and that includes being aware of what's going to happen if you try and use it to manage system resources. For that, ref-counting is almost certainly going to be better (though there are probably cases where it doesn't matter, because it's not really a problem if the resources aren't freed before the program exits). In any case, for a lot of the cases where dispose/IDisposable would be used (either that or scope statements if what's being done is localized and not common enough to create a type to use with RAII). That's probably on the list of things that needs to be written up somewhere as being among best practices for D. There are probably too many things like that that are sitting in the heads of a lot of folks but not necessarily well communicated to others - though this particular issue may have been discussed in the recent GC articles. I don't know. I still need to read them. - Jonathan M Davis
Dec 28 2017