www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - nogc, exceptions, generic containers... Issues.

reply "monarch_dodra" <monarchdodra gmail.com> writes:
I'm starting this thread related to two issues I'm encountering 
in regards to avoiding the GC, and the new  nogc attribute.

1) Issue 1)
The first issue is in regards to Throwables. The issue here is 
that they are allocated using the GC, so it is currently almost 
impossible to throw an exception in a  nogc context. This is (I 
think) a serious limitations. Do we have any plans, ideas, on how 
to solve this?

A particularly relevant example of this issue is `RefCounted`: 
This struct uses malloc to ref count an object, give a 
deterministic life cycle, and avoid the GC. Yet, since malloc can 
fail, it does this:
_store = cast(Impl*) enforce(malloc(Impl.sizeof));
Can you see the issue? This object which specifically avoids 
using the GC, end up NOT being  nogc.

Any idea how to approach this problem?

I know there are "workarounds", such as static pre-allocation, 
but that also comes with its own set of problems.

Maybe we could change it to say it's not legal to "hold on" to 
exceptions for longer than they are being thrown? Then, we could 
create the exceptions via allocators, which could 
deterministically delete them at specific points in time (or by 
the GC, if it is still running)? Just a crazy idea...

2) Issue 2)
The second issue is that data which is placed in non-GC *may* 
still need to be scanned, if it holds pointers. You can check for 
this with hasIndirections!T. This is what RefCounted and Array 
currently do. This is usually smart. There's a catch though.
If the object you are storing happens to hold pointers, but NOT 
to GC data, they are still scanned.

A tell-tale example of this problem is Array!int. You'd think 
it's  nogc, right? The issue is that Array has a "Payload" 
object, into which you place malloc'ed memory for your ints. The 
Payload itself is placed in a RefCounted object. See where this 
is going?

Even though we *know* the Payload is malloc'ed, and references 
malloc'ed data, it is still added to the GC's ranges of scanned 
data. Even *if* we solved issue 1, then Array!int would still 
*not* be  nogc, even though it absolutely does not use the GC.

Just the same, an Array!(RefCounted!int) would also be scanned by 
the GC, because RefCounted holds pointers...

A *possible solution* to this problem would be to add an extra 
parameter to these templates called "ScanGC", which would be 
initialized to "hasIndirection!T". EG:
struct Array(T, bool ScanGC = hasIndirections!T)

Does this seem like a good idea? I don't really see any other way 
around this if we want generic code with manual memory 
management, that is "GC friendly" yet still useable in a  nogc 
context.
Sep 08 2014
next sibling parent reply "yazd" <yazan.dabain gmail.com> writes:
An incomplete idea regarding exceptions and  nogc that I have is 
to encapsulate the exception within the returned value. And 
whenever the original value is attempted to be read, a check for 
the exception is done. In other words, the exception is not 
thrown from the place where it is constructed, but from the place 
where the return value is accessed.

This provides two advantages. Firstly, a choice can be made by 
the user of the function on whether to throw or not. If the user 
decided to check for the existence of an exception prior to 
accessing the value, the exception (with all of its data, msg, 
file/line info) can be accessed without (the costly) throwing. 
And if the user did not check for the exception, whenever the 
returned value is attempted to be accessed, throwing will occur 
(this gets us ease of use and flexibility).

Secondly, the returned type can contain the exception. This will 
remove the need for the GC and the need for allocating the 
exception on the heap.

Some preliminary code demonstrating the idea:
http://dpaste.dzfl.pl/3d37d524e4c6

The code currently is not  nogc, because emplace is not.

A disadvantage to this is that it cannot be used with functions 
that have purposes other than returning a value. For example, 
writing to a file.
Sep 08 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 08 Sep 2014 16:46:58 +0000
yazd via Digitalmars-d <digitalmars-d puremagic.com> wrote:

the whole idea behind exceptions is that you can omit such checks.
otherwise we can just use "return codes" and be happy.
Sep 08 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 8 September 2014 at 17:33:35 UTC, ketmar via 
Digitalmars-d wrote:

 On Mon, 08 Sep 2014 16:46:58 +0000
 yazd via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 the whole idea behind exceptions is that you can omit such 
 checks.
 otherwise we can just use "return codes" and be happy.


The type could throw by itself, if it is invalid and is accessed 
(i.e. a method called). Just use `alias getOrThrow() this`. Of 
course, such usage wouldn't be nothrow anymore, even if the 
ultimate methods that are going to be called by `getOrThrow()` 
are.
Sep 08 2014
prev sibling next sibling parent "Brian Schott" <briancschott gmail.com> writes:
On Monday, 8 September 2014 at 15:55:53 UTC, monarch_dodra wrote:

 A *possible solution* to this problem would be to add an extra 
 parameter to these templates called "ScanGC", which would be 
 initialized to "hasIndirection!T". EG:
 struct Array(T, bool ScanGC = hasIndirections!T)

 Does this seem like a good idea? I don't really see any other 
 way around this if we want generic code with manual memory 
 management, that is "GC friendly" yet still useable in a  nogc 
 context.


This is the strategy that the EMSI containers project uses.

This thread is related: 
http://forum.dlang.org/thread/ypbbtgyqsmspvbojghet forum.dlang.org
Sep 08 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 September 2014 at 15:55:53 UTC, monarch_dodra wrote:

 Any idea how to approach this problem?

 I know there are "workarounds", such as static pre-allocation, 
 but that also comes with its own set of problems.

 Maybe we could change it to say it's not legal to "hold on" to 
 exceptions for longer than they are being thrown? Then, we 
 could create the exceptions via allocators, which could 
 deterministically delete them at specific points in time (or by 
 the GC, if it is still running)? Just a crazy idea...


This is pretty much what we do in Sociomantic code base, re-using 
all exception instances and declaring that holding those (or 
making a fiber switch while handling) is illegal by convention.

Only other option I can imagine is to use reference-counted 
exception pools but this can't be implemented in the current 
language state.
Sep 08 2014
prev sibling next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Monday, 8 September 2014 at 15:55:53 UTC, monarch_dodra wrote:

 I'm starting this thread related to two issues I'm encountering 
 in regards to avoiding the GC, and the new  nogc attribute.

 1) Issue 1)
 The first issue is in regards to Throwables. The issue here is 
 that they are allocated using the GC, so it is currently almost 
 impossible to throw an exception in a  nogc context. This is (I 
 think) a serious limitations. Do we have any plans, ideas, on 
 how to solve this?

 A particularly relevant example of this issue is `RefCounted`: 
 This struct uses malloc to ref count an object, give a 
 deterministic life cycle, and avoid the GC. Yet, since malloc 
 can fail, it does this:
 _store = cast(Impl*) enforce(malloc(Impl.sizeof));
 Can you see the issue? This object which specifically avoids 
 using the GC, end up NOT being  nogc.

 Any idea how to approach this problem?

 I know there are "workarounds", such as static pre-allocation, 
 but that also comes with its own set of problems.

 Maybe we could change it to say it's not legal to "hold on" to 
 exceptions for longer than they are being thrown? Then, we 
 could create the exceptions via allocators, which could 
 deterministically delete them at specific points in time (or by 
 the GC, if it is still running)? Just a crazy idea...

 2) Issue 2)
 The second issue is that data which is placed in non-GC *may* 
 still need to be scanned, if it holds pointers. You can check 
 for this with hasIndirections!T. This is what RefCounted and 
 Array currently do. This is usually smart. There's a catch 
 though.
 If the object you are storing happens to hold pointers, but NOT 
 to GC data, they are still scanned.

 A tell-tale example of this problem is Array!int. You'd think 
 it's  nogc, right? The issue is that Array has a "Payload" 
 object, into which you place malloc'ed memory for your ints. 
 The Payload itself is placed in a RefCounted object. See where 
 this is going?

 Even though we *know* the Payload is malloc'ed, and references 
 malloc'ed data, it is still added to the GC's ranges of scanned 
 data. Even *if* we solved issue 1, then Array!int would still 
 *not* be  nogc, even though it absolutely does not use the GC.

 Just the same, an Array!(RefCounted!int) would also be scanned 
 by the GC, because RefCounted holds pointers...

 A *possible solution* to this problem would be to add an extra 
 parameter to these templates called "ScanGC", which would be 
 initialized to "hasIndirection!T". EG:
 struct Array(T, bool ScanGC = hasIndirections!T)

 Does this seem like a good idea? I don't really see any other 
 way around this if we want generic code with manual memory 
 management, that is "GC friendly" yet still useable in a  nogc 
 context.


It's not really a solution, but use of Nullable (or a theoretical 
option type) and a Result type in  nogc code can help remove the 
need for exceptions. An interesting data point is that Rust has a 
try! macro, which wraps an expression and translates into 
something like the following:

let ratio = try!(div(x, y));

becomes

let ratio = match div(x, y)
{
     Ok(val) => val,
     Err(msg) => { return Err; }
}

Maybe we need a similar solution for  nogc.
Sep 08 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Meta:


 let ratio = try!(div(x, y));

 becomes

 let ratio = match div(x, y)
 {
     Ok(val) => val,
     Err(msg) => { return Err; }
 }

 Maybe we need a similar solution for  nogc.


Related:
https://d.puremagic.com/issues/show_bug.cgi?id=6840

Bye,
bearophile
Sep 09 2014
parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 9 September 2014 at 07:05:22 UTC, bearophile wrote:

 Meta:


 let ratio = try!(div(x, y));

 becomes

 let ratio = match div(x, y)
 {
    Ok(val) => val,
    Err(msg) => { return Err; }
 }

 Maybe we need a similar solution for  nogc.


 Related:
 https://d.puremagic.com/issues/show_bug.cgi?id=6840

 Bye,
 bearophile


These are interesting, but they revolve more around avoiding the 
Exception altogether, rather than finding a solution to using 
exceptions in  nogc. In particular, these are pretty imperative 
solutions which (afaik) woulnd't work with 
constructors/destructors.
Sep 09 2014
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 8 September 2014 at 15:55:53 UTC, monarch_dodra wrote:

 A particularly relevant example of this issue is `RefCounted`: 
 This struct uses malloc to ref count an object, give a 
 deterministic life cycle, and avoid the GC. Yet, since malloc 
 can fail, it does this:
 _store = cast(Impl*) enforce(malloc(Impl.sizeof));
 Can you see the issue? This object which specifically avoids 
 using the GC, end up NOT being  nogc.


Although it's tangential to the issue at large, this particular 
code is wrong anyway. It should be using `onOutOfMemoryError` 
(which should be  nogc), otherwise the allocation of the 
exception is likely to fail just as `malloc` did.
Sep 09 2014