digitalmars.D - Advice requested for fixing issue 17914

Brian Schott (6/6) Oct 23 2017 Context: https://issues.dlang.org/show_bug.cgi?id=17914

Kagamin (2/2) Oct 24 2017 Call destroy(fiber) when it completed execution? Who manages
Steven Schveighoffer (4/10) Oct 24 2017 A failing use case would help. Fixing a bug when you can't reproduce is

Brian Schott (3/6) Oct 24 2017 I've attached one to the bug report.

qznc (8/14) Oct 24 2017 Looking at git blame [0], I guess Martin Nowak and Nemanja Boric

Brian Schott (5/7) Oct 24 2017 I've been reading the Fiber code and (so far) that seems seems to

safety0ff (13/18) Oct 24 2017 Just skimming the Fiber code I found the reset(...) API functions

Steven Schveighoffer (32/38) Oct 25 2017 It appears that the limitation applies to mmap calls as well, and mmap

Jonathan M Davis (4/14) Oct 25 2017 Maybe there was a change in the OS(es) being used that affected the limi...

Nemanja Boric (13/31) Oct 25 2017 Yes, the stack is not immediately unmapped because it's very

Nemanja Boric (4/25) Oct 25 2017 I'm not sure that would allow us to mprotect the page guards,

Nemanja Boric (9/36) Oct 25 2017 I think the easiest way to proceed from here is to default the

Steven Schveighoffer (12/42) Oct 25 2017 mmap does return an error. And onOutMemoryError is called when it fails.

Nemanja Boric (21/77) Oct 25 2017 Reading FreeBSD man pages, it looks like at least FreeBSD has the

Steven Schveighoffer (12/41) Oct 25 2017 Hm... the mprotect docs specifically state that calling mprotect on

Nemanja Boric (10/46) Oct 25 2017 I'm sorry I wrote several messages in the row, as the thoughts

Nemanja Boric (3/21) Oct 25 2017 On linux it's controllable by: `sysctl vm.max_map_count`

Nemanja Boric (10/22) Oct 25 2017 Although after the stack overflow protection for fibers number of

Brian Schott <briancschott gmail.com> writes:

Context: https://issues.dlang.org/show_bug.cgi?id=17914

I need to get this issue resolved as soon as possible so that the 
fix makes it into the next compiler release. Because it involves 
cleanup code in a class destructor a design change may be 
necessary. Who should I contact to determine the best way to fix 
this bug?

Oct 23 2017

Kagamin <spam here.lot> writes:

Call destroy(fiber) when it completed execution? Who manages 
fibers?

Oct 24 2017

Steven Schveighoffer <schveiguy yahoo.com> writes:

On 10/23/17 12:56 PM, Brian Schott wrote:
 Context: https://issues.dlang.org/show_bug.cgi?id=17914
 
 I need to get this issue resolved as soon as possible so that the fix 
 makes it into the next compiler release. Because it involves cleanup 
 code in a class destructor a design change may be necessary. Who should 
 I contact to determine the best way to fix this bug?

A failing use case would help. Fixing a bug when you can't reproduce is 
difficult.

-Steve

Oct 24 2017

Brian Schott <briancschott gmail.com> writes:

On Tuesday, 24 October 2017 at 14:28:01 UTC, Steven Schveighoffer 
wrote:
 A failing use case would help. Fixing a bug when you can't 
 reproduce is difficult.

 -Steve

I've attached one to the bug report.

Oct 24 2017

qznc <qznc web.de> writes:

On Monday, 23 October 2017 at 16:56:32 UTC, Brian Schott wrote:
 Context: https://issues.dlang.org/show_bug.cgi?id=17914

 I need to get this issue resolved as soon as possible so that 
 the fix makes it into the next compiler release. Because it 
 involves cleanup code in a class destructor a design change may 
 be necessary. Who should I contact to determine the best way to 
 fix this bug?

Looking at git blame [0], I guess Martin Nowak and Nemanja Boric 
seem to be pretty involved. Not sure how deep Petar Kirov and 
Sean Kelly are into Fibers.

My question wrt to the bug: Why is munmap/freeStack called in the 
destructor? Could be done right after termination?

[0] 
https://github.com/dlang/druntime/blame/ec9a79e15d446863191308fd5e20febce2053546/src/core/thread.d#L4077

Oct 24 2017

Brian Schott <briancschott gmail.com> writes:

On Tuesday, 24 October 2017 at 21:49:10 UTC, qznc wrote:
 My question wrt to the bug: Why is munmap/freeStack called in 
 the destructor? Could be done right after termination?

I've been reading the Fiber code and (so far) that seems seems to 
be reasonable. Can anybody think of a reason that this would be a 
bad idea? I'd rather not create a pull request for a design 
that's not going to work because of a detail I've overlooked.

Oct 24 2017

safety0ff <safety0ff.dev gmail.com> writes:

On Wednesday, 25 October 2017 at 01:26:10 UTC, Brian Schott wrote:
 I've been reading the Fiber code and (so far) that seems seems 
 to be reasonable. Can anybody think of a reason that this would 
 be a bad idea? I'd rather not create a pull request for a 
 design that's not going to work because of a detail I've 
 overlooked.

Just skimming the Fiber code I found the reset(...) API functions 
whose purpose is to re-use Fibers once they've terminated.

Eager stack deallocation would have to coexist with the Fiber 
reuse API.

Perhaps the Fiber reuse API could simply be polished & made easy 
to integrate so that your original use case no longer hits system 
limits.

I.e. Perhaps an optional delegate could be called upon 
termination, making it easier to hook in Fiber recycling.

The reason my thoughts head in that direction is that I've read 
that mmap/unmmap 'ing frequently isn't recommended in performance 
conscious programs.

Oct 24 2017

Steven Schveighoffer <schveiguy yahoo.com> writes:

On 10/23/17 12:56 PM, Brian Schott wrote:
 Context: https://issues.dlang.org/show_bug.cgi?id=17914
 
 I need to get this issue resolved as soon as possible so that the fix 
 makes it into the next compiler release. Because it involves cleanup 
 code in a class destructor a design change may be necessary. Who should 
 I contact to determine the best way to fix this bug?

It appears that the limitation applies to mmap calls as well, and mmap 
call to allocate the stack has been in Fiber since as far as I can tell 
the beginning. How has this not shown up before?

Regardless of the cause, this puts a limitation on the number of 
simultaneous Fibers one can have. In other words, this is not just a 
problem with Fibers not being cleaned up properly, because one may need 
more than 65k fibers actually running simultaneously. We should try to 
prevent that as a limitation.

For example, even the following code I would think is something we 
should support:

void main()
{
	import std.concurrency : Generator, yield;
	import std.stdio : File, writeln;

	auto f = File("/proc/sys/vm/max_map_count", "r");
	ulong n;
	f.readf("%d", &n);
	writeln("/proc/sys/vm/max_map_count = ", n);
	Generator!int[] gens; // retain pointers to all the generators
	foreach (i; 0 .. n + 1000)
	{
		if (i % 1000 == 0)
			writeln("i = ", i);
		gens ~= new Generator!int({ yield(1); });
	}
}

If we *can't* do this, then we should provide a way to manage the limits

I.e. there should be a way to be able to create more than the limit's 
number of fibers, but only allocate stacks when we can (and have a way 
to tell the user what's going on).

-Steve

Oct 25 2017

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Wednesday, October 25, 2017 09:26:26 Steven Schveighoffer via 
Digitalmars-d wrote:
 On 10/23/17 12:56 PM, Brian Schott wrote:
 Context: https://issues.dlang.org/show_bug.cgi?id=17914

 I need to get this issue resolved as soon as possible so that the fix
 makes it into the next compiler release. Because it involves cleanup
 code in a class destructor a design change may be necessary. Who should
 I contact to determine the best way to fix this bug?

 It appears that the limitation applies to mmap calls as well, and mmap
 call to allocate the stack has been in Fiber since as far as I can tell
 the beginning. How has this not shown up before?

Maybe there was a change in the OS(es) being used that affected the limit?

- Jonathan M Davis

Oct 25 2017