www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 3523] New: Fiber is not garbage collected properly

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523

           Summary: Fiber is not garbage collected properly
           Product: D
           Version: 2.032
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: druntime
        AssignedTo: sean invisibleduck.org
        ReportedBy: baryluk smp.if.uj.edu.pl



19:50:33 PST ---
Program below leaks memory.

import core.thread;
import std.stdio;

lass DerivedFiber : Fiber {
  this() { super( &run );  }
 private void run() { Fiber.yield(); }
}

import core.memory : GC;

void main() {
    foreach (i ; 1 .. 1000000) {
        Fiber derived = new DerivedFiber();
        derived.call();
        GC.collect(); // doesn't work
    }
}

Manual destruction of fiber works:
              delete derived;
        }
and then it doesn't leek.

chaning Fiber to "scope" also works (just like delete befor "}"

I know it have something to do with similarity of Fiber to thread, that Fiber
have own pool of root variables and own stack. But if Fiber is not running, and
it is not accesible by any reference from any Thread, then it is imposible to
resume its operation by call(), so also it's root variables and stack is not
avaible, so it can (and should?) be garbage collected.

I have code which creates few fibers, do milions call/yield operations, then
destroy fibers, and recreat new ones, for and essentially repeats. I was hoping
this would not leak memory. Unfortonetly it is.

I this is intended behaviour it should be documented somewhere.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2009
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




19:56:25 PST ---
If one will not call derived.call() (so leaving Fiber in TERM state, and never
running it at all) it will be properly collected.

Adding after derived.call(), a derived.reset() to make it back to TERM state,
doesn't help (still it is not collected).

Adding second derived.call(), after first one, will make collect() to work
correctly.

So i can assume it run() method terminates correctly and underlaying stack is
destroyed, object is properly destructed in colletion phase.

Essentially my problem is because my Fibers doesn't terminate at all they
"yield" infinitly (saving some auxilary data in fields of some objects, so this
data can be used outside of yield, essentiall in thread which called call), but
i want to terminate them automatically (when they are not referenced by any
thread or other Fiber) if needed.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523


Sean Kelly <sean invisibleduck.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED



---
Hm... that's tricky.  The fiber implementation needs to hold a reference to the
fiber on its stack for context switching, and that's the reference that is
keeping the fiber alive.  I'll play with the stack pointers a bit and see if
things work if I exclude that reference from the GC scan.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 19 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




16:05:22 PST ---
I "solved" my problem by changin one of my base clases from:

abstract class AGenerator : Fiber {
protected:
        this(void delegate() dg) {
                super(dg);
        }
}


To:
abstract class AGenerator {
private:
        Fiber x;

protected:
        this(void delegate() dg) {
                x = new Fiber(dg);
        }
        ~this() {
                delete x;
        }
public:
        void call() {
                x.call();
        }
        void yield() {
                x.yield();
        }
        Fiber.State state() {
                return x.state;
        }
}

And it magically started working correctly (Fibers are properly destroyed).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 24 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




05:53:40 PST ---
No, sorry i made mistake. Even after change it is not garbage collected.
Which is normal, because this Fiber is running some method from this object
(namly method named void iter() ), so this object (AGenerator) is still
referenced by Fiber, and it's destructor can't be called. So probably only way
to call destructor will be to separate this into two classes, which is
referenced by delegate from class A, and second which is used only for calling
from outside of fiber.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 25 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




06:11:41 PST ---
Ok, i now i solved it using kind of hack:

/** This class is written because Fiber's are not correctly garbage collected
*/
class GenWrap(T : AGenerator, T2) {
    private T x; /// T derives from AGenerator which derives from Fiber
public:
    this(T x_) { x = x_; }
    ~this() { delete x; }
    T2 getNext() { return x.getNext(); }
    T o() { return x; } // don't assign return value to any variable which can
live longer than this object
}

This is hack, because it can destroy Fibers which are still referenced
somewhere. So All my direct usages of variables of type T, must be changed to
use GenWrap.o(), to be sure that delete x inside destructor is safe.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 25 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523


dawg dawgfoto.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dawg dawgfoto.de



I think this works now, doesn't it?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 14 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




10:20:42 PST ---

 I think this works now, doesn't it?
If you think so, I will test it shortly and report back. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Feb 14 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




20:53:05 PST ---
Still same problem in 2.052. :(

Will check tomorrow 2.057, or 2.058 if it will be released.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 14 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




21:12:50 PST ---

 Still same problem in 2.052. :(
 
 Will check tomorrow 2.057, or 2.058 if it will be released.
Well, it looks 2.058 is already available. I installed it (again 32-bit Linux), and program leaks - it starts at about 14MB of virtual memory usage, 4MB or RAM usage, and both grows about 1MB per second, after minute giving about 100MB, and constantly growing. So problem is still present. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Feb 14 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




Hm... that's tricky.  The fiber implementation needs to hold a reference to the
fiber on its stack for context switching, and that's the reference that is
keeping the fiber alive.  I'll play with the stack pointers a bit and see if
things work if I exclude that reference from the GC scan.
It's simple to fix I think. Don't add the fiber context to the global context list but GCscan all nested contexts instead. Now you only need to push/pop a reference to the Fiber object for call/yield. It's like swapping ownership. When the fiber is halted the Fiber object owns it's stack, when the fiber is being executed the stack owns the Fiber object. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Feb 15 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




08:09:20 PST ---

Hm... that's tricky.  The fiber implementation needs to hold a reference to the
fiber on its stack for context switching, and that's the reference that is
keeping the fiber alive.  I'll play with the stack pointers a bit and see if
things work if I exclude that reference from the GC scan.
It's simple to fix I think. Don't add the fiber context to the global context list but GCscan all nested contexts instead. Now you only need to push/pop a reference to the Fiber object for call/yield.
Well I agree. My reasoning is - if fiber is not currently executed and is not referenced by anything from live set (all live threads, and all executing fibers, and all threads and fibers referenced by them recursively), then there is no way to resume execution of fiber (or terminate it manually, or change it), and it's GC job to terminate and delete it.
 It's like swapping ownership. When the fiber is halted the Fiber object owns
 it's stack, when the fiber is being executed the stack owns the Fiber object.
It will probably introduce small additional overhead to Fiber context switching and execution, but should not be much. I will be happy to test any fix which resolves this problem. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Feb 15 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523


Alex Rønne Petersen <alex lycus.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alex lycus.org



CEST ---
What is the status of this today?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 09 2012
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=3523




When the fiber is halted the Fiber object owns it's stack
This doesn't work out with our current GC mechanisms and the required stack scans. The other solution I can think of is to keep the inner stack free of any references to the enclosing fiber. This would imply that all Fiber bookkeeping must be done using asm or on the outer stack to fully avoid pointer leakage. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Oct 16 2012