• Roland Hadinger (36/36) Oct 18 2019 These questions probably need some context: I'm working on an
• Jonathan M Davis (64/100) Oct 18 2019 D's GC is a stop-the-world GC. Every thread managed by the GC is stopped

Roland Hadinger <rolandh.dlangforum maildrop.cc> writes:

These questions probably need some context: I'm working on an 
interpreter that will manage memory via reference counted struct 
types. To deal with the problem of strong reference cycles 
retaining memory indefinitely, weak references or recursive 
teardowns have to be used where appropriate.

To help detect memory leaks from within the interpreter, I'd also 
like to employ core.memory.GC in the following fashion:

* keep core.memory.GC off (disabled) by default, but nonetheless 
allocate objects from GC memory
* provide a function that can find (and reclaim) retained 
unreachable object graphs that contain strong reference cycles
* the main purpose of this function is to find and report such 
instances, not to reclaim memory. Retained graphs should be 
reported as warnings on stderr, so that the program can be fixed 
manually, e.g. by weakening some refs in the proper places
* the function will rely on GC.collect to find unreachable objects
* the function will *always* be called implicitly when a program 
terminates
* the function should also be explicitly callable from any point 
within a program.

Now my questions:

Is it safe to assume that a call to GC.collect will be handled 
synchronously (and won't return early)?

Is there a way to ensure that GC.collect will never run unless 
when called explicitly (even in out of memory situations)?

Is it possible and is it OK to print to stderr while the GC is 
collecting (e.g. from  nogc code, using functions from 
core.stdc.stdio)?

Could I implement my function by introducing a shared global flag 
which is set prior to calling GC.collect and reset afterwards, so 
that any destructor can determine whether has been invoked by a 
"flagged" call to GC.collect and act accordingly?

Alternatively: do I need to implement such a flag, or is there 
already a way in which a destructor can determine whether it has 
been invoked by the GC?

Thanks for any help!

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Friday, October 18, 2019 10:54:55 AM MDT Roland Hadinger via Digitalmars-
d-learn wrote:
 These questions probably need some context: I'm working on an
 interpreter that will manage memory via reference counted struct
 types. To deal with the problem of strong reference cycles
 retaining memory indefinitely, weak references or recursive
 teardowns have to be used where appropriate.

 To help detect memory leaks from within the interpreter, I'd also
 like to employ core.memory.GC in the following fashion:

 * keep core.memory.GC off (disabled) by default, but nonetheless
 allocate objects from GC memory
 * provide a function that can find (and reclaim) retained
 unreachable object graphs that contain strong reference cycles
 * the main purpose of this function is to find and report such
 instances, not to reclaim memory. Retained graphs should be
 reported as warnings on stderr, so that the program can be fixed
 manually, e.g. by weakening some refs in the proper places
 * the function will rely on GC.collect to find unreachable objects
 * the function will *always* be called implicitly when a program
 terminates
 * the function should also be explicitly callable from any point
 within a program.

 Now my questions:

 Is it safe to assume that a call to GC.collect will be handled
 synchronously (and won't return early)?

D's GC is a stop-the-world GC. Every thread managed by the GC is stopped
when a thread runs a collection.

 Is there a way to ensure that GC.collect will never run unless
 when called explicitly (even in out of memory situations)?

The GC only runs a collection either when you explicitly tell it to or when
you try to allocate memory using the GC, and it determines that it should
run a collection. Disabling the GC normally prevents a collection from
running, though per the documentation, it sounds like it may still run if
the GC actually runs out of memory. I had thought that it prevented
collections completely, but that's not what the documentation says. I don't
know what the current implementation does.

 Is it possible and is it OK to print to stderr while the GC is
 collecting (e.g. from  nogc code, using functions from
 core.stdc.stdio)?

No code in any thread managed by the GC is run while a collection is running
unless it's code that's triggered by the collection itself (e.g. a finalizer
being called on an object that's being collected - and even that isn't
supposed to access GC-allocated objects, because the GC might have already
destroyed them - e.g. in the case of cycle). If you want code to run at the
same time as a GC collection, it's going to have to be in a thread that is
not attached to the GC, and at that point, you shouldn't be accessing
_anything_ that's managed by the GC unless you have a guarantee that what
you're accessing won't be collected. And even then, you shouldn't be
mutating any of it.

Also,  nogc doesn't say anything about whether the code accesses
GC-allocated objects. It just means that it's not allowed to access most GC
functions, which usually just means that it doesn't allocate anything using
the GC and that it doesn't risk running a collection. So, just because a
function is  nogc doesn't necessarily mean that it's safe to run it from a
thread that isn't managed by the GC while a collection is running.

 Could I implement my function by introducing a shared global flag
 which is set prior to calling GC.collect and reset afterwards, so
 that any destructor can determine whether has been invoked by a
 "flagged" call to GC.collect and act accordingly?

You should be able to do that, but then the destructor can't be pure (though
as I understand it, there's currently a compiler bug with pure destructors
anyway which causes them to not be called), and when a destructor is run as
a finalizer, it shouldn't be accessing any other GC-allocated objects,
because the GC might have actually destroyed them already at that point.
Finalizers really aren't supposed to doh much of anything other than
managing what lives in an object directly or managing non-GC-allocated
resources. Regardless, anything that really should be operating as a
destructor rather than a finalizer has to live on the stack, since
finalizers won't be run until a collection occurs. If you're explicitly
running them yourself via your own reference counting, then you don't have
that problem, but if there's any chance that a destructor is going to be run
as a finalizer by the GC, then you have to write your destructors /
finalizers with the idea that that could happen.

 Alternatively: do I need to implement such a flag, or is there
 already a way in which a destructor can determine whether it has
 been invoked by the GC?

 Thanks for any help!

Honestly, the way things are set up, destructors aren't supposed to know or
care about whether they're being run by the GC as a finalizer. So, the GC
isn't going to provide that kind of functionality. What you're looking to do
is pretty much a giant hack from the perspective of the GC and likely to be
pretty dangerous to attempt. I suspect that what would make a lot more sense
would be to create a custom build of druntime to run which specifically
printed out what wasn't freed when the program shut down rather than trying
to hack around how the GC works. Alternatively, you could just ditch the GC
entirely and then use valgrind to see what didn't get freed to catch cycles
(or other screw-ups that resulted in memory not being freed).

Having the GC take care of cycles for you isn't necessarily a problem, but
having the GC report on what's alive or not is tricky business, particularly
since it's supposed to keep anything that the program still has access to
alive.

Another thing to consider is that some language features outright require
the GC (e.g. closures and anything with dynamic arrays involving
allocation), and if you truly don't want to use the GC for that stuff, it's
probably going to be easier to require that your program not use the GC at
all than to try to have it just manage cycles.

Regardless, if you really want to go forward with something like you're
proposing here, you'll probably need to get answers from one of the few GC
experts around here.

- Jonathan M Davis